Testplan Editor-TestplanListView Editor/en
The test plan editor is used to create, modify and execute test plans. It is shown in the "Testplan" tab when a testplan element is selected in the navigation tree.
Testplans are constructed as a sequence of test cases. Notice that in literature and other test frameworks, these are often called "test steps". Expecco uses the term "test case" for items in a test plan and "test step" for individual action steps within a test case.
To add test cases to a test plan, drag actions (from the left tree) into the test case list. If you want to define your test plan in a top-down fashion (i.e. create the list of test cases first, before any actions are defined), you can also create the test plan's list items with the "Add Test Case" button or menu function, and later specify the concrete action by dragging actions into the "Action" field at the bottom.
Later, when the test plan is executed, these test cases will be executed in sequence one after the other.
The test plan editor consists of two major areas: the top list presents the list of test cases, the bottom presents attributes of the selected item in the list. Attributes specify the behavior of the test plan and individual test cases.
To execute the test plan, click on the green run button in the toolbar.
When a test plan has been executed, the state of its last execution outcome is shown in the top list. Not executed or inconclusive items are shown with a grey color, successful items in green, and failed items in red.
To the right you can see two examples for an opened testplan editor. In the first, the test plan itself is selected; in the second, a test case is shown. Both show a situation after a test run which was only partially successful.
- 1 Buttons
- 2 Tasks
- 2.1 Adding and Arranging Test Cases
- 2.2 Executing a Test
- 2.3 Selective Execution
- 2.4 Repeated Execution (Loop Modes)
- 2.5 Configuring the Testplan
- 2.6 Configuring Testcases
- 3 Context Menu of the Testcase List
- Start the test execution.
- Start the test execution with debug.
- Runs all previously failed test cases again
- Start execution in single step mode or continue single stepping.
- Start execution in single step mode or continue single stepping. Steps into called actions.
- Pause the current test run. To proceed, click on the run button. To proceed single stepping, click on one of the single-step buttons.
- Stop the execution of the current test-case, marking it as aborted (inconclusive). Proceed by starting the next test-case.
- Stop the execution of the current test run. Exception handling and cleanup actions will be performed.
- Hard-Terminate the testrun, without exception handling or cleanup actions. This is useful e.g. for hang-ups or communication failures (i.e. if a cleanup action itself hangs). Be aware, that this may lead to leftover open web-browser windows (in case of web-testing) or leftover open communication channels (sockets, serial line connections). These may have to be closed manually via the "Extra" -> "Debugging" -> "Close Connections" menu, if required.
- Logging drop down menu to load/save a result log
- Generate a report from the log data
- Adds an empty test-case to testplan. The item's action can then be specified in the lower details area. However, a more convenient way to add test-cases is to simply drag&drop an action into the list.
- Remove the selected test-case(s) from test plan
- / Change the execution order of the test-cases, by moving the selected test-case(s) up or down in the list
- Follow execution: autoselect the currently active block in the log (while running).
In recent expecco versions, this button got replaced by a drop-down box, to selecte the number of levels, which should be followed. The button's icon image is slightly different.
- Find the previous failed or erroneous test case
- Find the next failed or erroneous test case
- Find the next inconclusive test case
- Find the next active (currently running) test case
Adding and Arranging Test Cases
There are two ways to create new test case entries in the test plan:
- Create empty items with the toolbar button shown above, then specify the action block in the lower editor's "Action" field.
- Directly drag & drop an action block or another test plan from the navigation tree into the testplan area. For this, it is useful to open a secondary popup or split tree.
Please note that not any block can serve as test case item in this list. Allowed are:
- actions without input pins (or where all pins have default values)
- actions where input pins have only simple types (strings, numbers, filenames)
- other test plans (this creates a hierarchical test plan, with sub-test sequences).
When creating an entry with the toolbar button, the new entry will appear below the selected entry. When a block is dropped to the test plan directly, the new entry is added at the very bottom of the list.
The items' order can be changed via the "Move Item Up" / "Move Item Down" items found on the the popup menu of a selected item. There are also keyboard shortcuts (Ctrl-↑ / Ctrl-↓) to move the selected item(s).
Executing a Test
Start the execution by clicking on the "Start" (or "Run") button (the little green arrow-shaped button). Test cases can be individually skipped or activated by toggling the "Enable" checkbox in the list. There are also popup-menu entries to enable/disable multiple items in one operation (i.e. select multiple items, then use the menu function "Skip Selected Items").
The list keeps the information of the previous run, even if you switch to another editor and come back later. However, only the last execution's result of a test plan is remembered (unless you save the result in a file). Before any run, all test-cases' states are reset to the "Untested" state (which counts as "INCONLUSIVE"), no matter if they are activated for execution or not.
All checked test-cases are executed sequentially. While running, the currently executing test-case is marked with a little clock symbol. The symbols of all other test cases indicate their result state, which is one of the verdicts or the already mentioned "Untested".
After the test run, the list-elements can be expanded (the little [+]/[-] icons) to browse the test-cases' activity logs. You can also follow the execution while running, to keep track on the current execution state. Either click on those activity items, or check the "Follow Activity" toggle, to have expecco automatically follow the currently active action.
Create a report from this test-run by clicking on the "Report" button. Initially, a standard report template (with probably way too much detail) is printed. This can be customized in various ways to suit your needs. For more information see: Report Generation.
You can also execute individual test-cases or a subset of all the test plan.
- Individual Selection:
Individual testcases are excluded/included by toggling their execution toggle in the list.
- Rerun of Failed Tests:
The "Execute Failed" button executes those tests which failed in the previous run. This is useful, if the whole suite would take a long time and individual tests are to be executed again after a fix of the system under test. Or after a change of the test-case's activity diagram.
- Selection by Risk:
You can define individual risk levels to each test-case of the plan. Now toggle the "Risk" check-box at the top of the test plan and set a risk limit. Then, when executing, only those test-cases will be executed which have a greater or equal risk level.
- Selection by Group:
By attaching individual group-tags to each test-case, you can also group related test cases into test groups. Then, toggle the "Testgroup" check-box at the top of the test plan and enter some identifier(s) to specify, which group(s) to execute. The next test-run will only execute test-cases with a matching group tag.
- Dynamic Selection by Pre-Execution Checks
If present, the pre-execution action is run before the actual test case. If it generates a non-successful result, the corresponding test case is skipped (a corresponding post-execution action is also skipped, if present).
- Dynamic Selection by Condition Variables
You can attach a condition variable or a set of condition variables to a testcase. If a name or list of names is specified in the "Check Condition Variable" field, all of them must contain a boolean
truevalue, otherwise, the test case is skipped with an inconclusive outcome. The variables must be defined in the testplan's environment (or the project's top environment) as boolean variables.
In addition, it is possible to update a condition variable (or a set of condition variables) depending on the outcome of a test case. All variables listed in the "set condition variable" field will be set to
true, if the test case passed,
These two allow for a simple conditional execution of individual test cases. Of course, these variables are also visible in the network as regular environment variables. Therefore, more complex checks can be performed using arbitrarily complex logic there.
Finally, by dropping another test-plan into the test-case area, all of its test-cases are embedded and are treated as a single test-case in the outer test-plan. Thus, hierarchies of testplans can be created, which can of course still be executed individually.
Repeated Execution (Loop Modes)
Often it is useful to run the same test scenario multiple times. For example if an error only occurs sporadically, you may want to repeat the test run until an error occurs. Or if your goal is to detect memory leaks or performance degration of the system under test, you may want to run it multiple times to put stress on it. For this, various loop modes can be configured (see "Execution Modes" below):
- repeat the test for a particular number of runs
- repeat the test for a given time duration
- repeat the test until an error occurs
- repeat the test until no error occurs
Of course, all of the above can be easily implemented by creating a compound block and placing the test scenario's action block into it with either an iteration count or by adding appropriate looping blocks around. This is also the preferred mechanism, if different input values or configuration data has to be used for each run. However, in most cases, only a simple repetition of the same test is desired, and this can be done without programming by setting up a loop mode in the execution configuration.
Individual versus Full Report
When executing a test in loop mode, a big amount of trace and report data may accumulate, which may be unhandy to examine and may also reach the available memory limit (in this case, expecco prunes the execution log, by removing older trace information).
For this, it is possible to specify that individual reports are generated and written to a file after each run - either unconditionally or only when an individual run is successful or failed. You can specify a filename pattern for this, so that each file gets a distinguished name (with optional run-number and/or time stamp in the file name). The files will be created relative to the project's attachment folder, unless an absolute pathname is specified.
Alternative Postprocessing Options
In case you want to postprocess the accumulated trace/log information, you can also define a log-processor action and add it to individual testcases as "Log Processor Action". This allows for arbitrary filtering and/or processing of the log information, but requires some knowledge about the structure of the log items and the operation of the log-processing action blocks in the Standard Library and/or the Reflection Library .
Configuring the Testplan
Adding Pre and Post Execution Actions
Testplans can have pre- and post-execution blocks. The pre-execution block is executed before the plan is executed; the post-execution block is executed after it has finished. If the pre-execution block does not finish with success, the testplan is not executed. This can be useful e.g. to allocate and/or release resources. To add a pre- or post-execution action, drag & drop an action block from the navigation tree into the corresponding field. Please note that these blocks cannot receive values via an input pin. See also below for pre- and post actions of individual test cases.
Adding a Background Execution Action
Testplans can have a background-execution block. The background-execution block is executed in parallel to the execution of the testplan and will be terminated after the execution of the testplan has finished. Typically, this will be a block which opens a socket, pipe or other communication channel or to start an external (shell-) process to feed or monitor the system under test. See also below for individual testcase background actions.
Adding an Inventory
If a test case requires a resource (device, lock or operator), an inventory can be specified, which defines the set of available resources and from which the resource will be allocated. Without an inventory, the test will not be able to perform. The inventory is defined in the top-level testSuite's misc tab. If tests are executed from expecco ALM via remote-execution, the inventory is provided by expecco ALM instead. This ensures that measurement devices are correctly acquired and locked for the duration of a tests execution. To add an inventory, drag & drop an existing inventory definition element from the navigation tree.
- Run Time Limit
This field is used to set a time limit for the execution of the test plan. The execution will stop after the specified duration. The entered number can be followed by a time unit, e.g. "ms" / "s" / "m" / "h" / "d". Without unit, "seconds" are assumed. To remove a previously set time limit simply leave the field blank.
- Finish Suite even if Time Limit Reached
This check box determines if either the execution of a test plan is stopped immediately after reaching the time limit or let the execution finish, e.g. to execute the remaining items and any post actions.
- Loop (Stop on Error or Failure / Success)
This check box toggles looping on or off. If on, the execution of the test plan will be repeated until an error or failure (success) is encountered or otherwise the time limit for the execution is reached. The stop on condition feature is useful to catch sporadic error-behavior of a system under test. For example, to run a test unattended over night, but stop when a certain condition arises.
- Loop Count
This field is used to set a maximum loop count for the test plan execution. To loop endless just leave the field blank.
Forces cycles to be executed with this periodic time. For example, if you specify 30m, the testplan will be repeated every 30 minutes. If an individual run takes longer than the period, the next loop iteration will be performed immediately - otherwise, the system waits (idle) for the time difference between period and previous execution time. Use this, if the system under test needs some cleanup or "healing" time after each test run - especially when doing stress tests on a system which needs database or memory management cleanup after each run (e.g. systems which need some pause to perform garbage collection to prevent ever growing memory usage).
- Save Result after each Run
If checked, a report file is written as specified in the filename pattern field after each run - either unconditionally, or for every successful or for every failed run. This is useful if a test only fails/succeeds sporadically, and you need a trace/log for those runs. If you do not specify this option, a single huge report may be generated, which is both hard to examine and which may become larger than the systems memory capacity (in this case, expecco would prune the report and throw away older items, which is usually not the user's intention).
The filename pattern specifies the names of those individual report files, and should contain meta patterns such as:
- "%1" for the current run-number (1..)
- "%2" for the individual run's start timestamp (YYYY-MM-DD-HHMMSS)
- "%3" for the name of the test plan
- "%4" for the name of the test suite.
Thus, "%4-%3-%1-%2.elf" will generate report files named "suite-plan-1-2014-10-04-140005.elf", "suite-plan-2-2014-10-04-140007.elf", etc.
You may specify multiple report file patterns in this field (separated by semicolon ";", to generate reports in different formats. For example, enter "run%1.elf , run%1.pdf" to generate both a full log (which can be reopened with expecco for detail information) and a summary report as pdf document. For all reports, the default report template of the suite (or the user settings) is used.
The individual report files are created either in the project's attachment folder or, if an absolute pathname is given as pattern, in that folder.
The attachment folder is a temporary folder, which is automatically removed when expecco is closed, or another suite is opened. Thus, you should archive those after execution. If expecco ALM is used as a test execution management system, these files will be uploaded and archived automatically after a run.
If expecco is used without expecco ALM, you may want to add a post-execution action, which copies those files to an archive, a database or checks them into a versioning repository. This is also a possible solution, if expecco-tests are to be started via another system, like jenkins or a batch script.
Manual Execution Settings
- Stop between individual Tests
This check box allows you to stop the execution between each testcase. A confirmation dialog will pop up, asking for a confirmation-click to continue. Useful if any manual handling is needed after each test-case.
- Stop between loop cycles
Similar to the above. If checked and one of the loop modes is selected, a confirmation dialog pops up after every loop cycle, asking for a confirmation-click to continue.
- Stop on Error
This check-box determines whether the execution of the test plan is stopped when an error occurs. There are two such check boxes, to specify if it should apply only to test-cases marked as "required" or also to "optional" test cases.
expecco ALM Settings
These settings are only relevant if expecco is used as a "slave test execution engine" for expecco ALM. They are ignored if expecco is running as a stand alone application (e.g. by the test developer) or started manually by a test engineer (see "Command Line Options").
- Visible in expecco ALM
This checkbox determines whether the testplan is visible in expecco ALM. Such externally visible testplans can later be executed by expecco ALM. Invisible testplans are useful for the test developer, to leave partial tests, setup, shutdown or cleanup sequences in the test suite, which are not meant for public use (or use by the automatic test scheduler).
- Operator Needed
This checkbox determines whether an operator is required during execution of the test. Expecco ALM will then ask a human operator to supervise/perform the test.
- Cases Visible in expecco ALM
This checkbox determines whether testcases are individually selectable for execution in expecco ALM. If unchecked, only the whole testplan can be configured for an automatic run. You should only turn this check on, if the testcases are independent from each other, and do not need state from a previous test case or leave the system under test in a state needed by another test case. Otherwise, it is better to define multiple testplans, each with its individual setup/shutdown actions to leave the system under test in a defined state, and let the test manager choose among those plans, instead of individual testcases.
Testcase Description Field
Specifies the test case's descriptive name which is shown in reports, traces and log files. If this field is left blank, the name of the action block which implements the test case is shown. This field does not affect the execution - it only controls the names used in reports and logs.
The action block, which implements the test case. This can be any compound or elementary action block from the left tree. The block must be one without input pins - if required, wrap it into a new compound block and provide input values from any source, such as constant freeze values, database values, CSV values from a file or environment variable values. You can either set the action by dragging a block from the left tree into this field, or by opening a selection dialog via the "..." button at the field's right.
Condition variables provide an easy to use control mechanism over which testcase items are executed during a run. By specifying a list of boolean condition variables in this field, the testcase item will only be executed if all of those variables contain a "true" value (non-existing variables are treated like being "true"). After the execution, the success-state is written to the variables named in the "set variables" field. These variables are managed in the testplan's environment. Therefore, you can also access these variables via regular environment blocks or via the low level code API (environmentAt: / environmentAt:put: calls). Condition variables can be used to disable individual test cases or to guide execution through different paths depending on previous actions. A typical application is to read out the system under tests configuration by a pre-action or first test case step, then setting condition variables, and execute only specific subsets of the suite, depending on the outcome.
More complex conditional execution is possible by wrapping individual test case actions into a compound block and placing condition test actions into that.
Adding Pre and Post Execution
Individual testcase items can have pre- and post-execution actions too. The pre-execution action (if specified) is executed each time the item is executed; the post execution action is executed after the item's execution has finished. This can be useful e.g. to allocate and/or release resources, to check for more complex preconditions, system state or for the test system's configuration, and to leave that information in environment variables to be accessed by later test steps, or be used as condition variables. To add a pre- or post-execution action, simply drag and drop a block from the navigation tree into the according slot. Please note that these block may not have input pins.
The pre-execution action is also a pre condition. If it ends non-successful, the corresponding test case and any post execute action are skipped.
Sometimes, a background server process, monitor or data feeder activity is required to run in parallel to the actual test scenario. This field allows for an action to be defined, which is started in parallel with (actually: right before) the test case action and terminated after the test case's execution. Any compound or elementary block can be specified. Typically, this will be a block which opens a server socket, pipe or other communication channel to feed the system under test, or to start an external program for monitoring, capturing or generating data. Please note that this block may not have input pins.
Log Processor Action
Test execution generates an execution log/trace, which is later used to generate a more or less detailed report. In most cases, the report settings allow for most common reports to be generated (by specifying the amount of data and detail of the report there). However, for very special reports, it may be useful to post process the generated raw data before it is used as input to the report generator. This field allows for an action block to be specified, which gets the raw log as input and should produce a cooked-up version of it as output. Arbitrary processing, filtering, renaming or archival of the raw data is possible in this action. Of course, some knowledge about the structure of that raw data is required, and you should consult the class browser and/or open a data inspector on a generated raw log for this. You will find an example in the "d11_LOG_Processing_Example.ets" suite (in "projects/examples").
Testgroup and Risk Setting
For the selective execution of a test plan it can be useful to set a risk level for certain test cases or to collect related test cases in test groups. The risk level can go from "very low" to "very high" or can be set to "unknown". To add a block to an test group, just enter identifiers for the groups into the according field. Multiple identifiers must be separated by spaces. See "Selective Execution" for more information.
The check box in front of each test case determines whether it is executed or not (in the upper list). Please note that this setting is temporary and will not be saved. To set the default execution setting use the "Default for Execute" check box in the lower attribute area.
Context Menu of the Testcase List
Some of the context menu (right-click) functions of the test-case/activity-log list operate on the selected item or set of items.
- Add Testcase
Adds a new (blank) test-case to the test-plan. You should drag and drop a test action into the "block" field in the lower pane.
- Delete Testcase
Removes the selected test-case from the test-plan.
- Enable selected Testcases
Enables (activates) the selected test-cases for execution.
- Disable selected Testcases
Disables the selected test-cases for execution.
- Make all Enable Flags the Default for Execution
Sets the state of the selection as the default for the test case. This will be the initial enable/disable-state of those test-cases, when the suite is loaded.
- Make selected Testcase Required
Sets the priority of the selected test-cases to "Required".
- Make selected Testcases Optional
Sets the priority of the selected test-cases to "Optional".
- Open Page on selected Item
Opens a new browser page on the selected test-case's action. This function is only available if exactly one test-case is selected.
Update all activity logs and sublogs under the selected item.
- Find Next Error
Find and select the next failed or erroneous test-case in the activity log.
- Find Previous Error
Find and select the previous failed or erroneous test-case in the activity log.
- Remove Result for selected Items
Removes the test results of the selected test-case.
- Compress Result for selected Items
Compresses the log result for the selected items. Compression means that only erroneous log entries are kept; all passed and OK infos are removed.
- Generate Report from here...
Generates a report for the selected test-case (and, if this is a sub-testplan, for all nested test-cases).
- Move Up/Down
To change the execution order of the test-cases.
The full online documentation can be found under: Online Documentation