Notice: starting with release 1.8, this editor has been split into separate editor and execution tabs:
- Definition of the network under test. This is used to define a test or demo for the edited action. As such, it combines features of the network editor.
- Shows information of the current or last execution of the network under test.
This should not be used for "official" test-cases; instead, use it for the "private" execution of a single block (i.e. unit tests for the test-suite itself) or as a place for demonstration or "executable documentation".
To the right, you see a test editor after the successful execution of a test-block.
- 1 Buttons
- 2 Logging Menu
- 3 Views
- 4 State Colors
- 5 Tasks
- "Run" - Start execution. All auto started steps will be triggered. Unless enabled in the execution settings, no debugger is opened when an error occurs; instead, the execution is stopped with a failed status (shown in red)
- "Debug Run" - Start the test execution in debug mode causing the debugger to open on exceptions (regardless of what is specified in the execution settings).
- "Step" - Start the test execution in single step mode or continue single stepping (when at a breakpoint).
- "Step Into" - Like the above, but steps into other compound actions.
- "Pause" - Pause the current test run. To proceed or single step, click on the according button.
- "Stop" - Stop the execution of the current test run. Exception handling & cleanup (post actions) will be performed.
- "Interrupt" (new in 2.12) - Interrupt the execution of the current test run and open a debugger. Useful to debug endless loops or to inspect the execution state in a long running elementary action. In the debugger, the execution can be proceeded, single stepped or aborted.
- "Pause on Error" - This toggle controls how the executor behaves when an error is encountered. The default is to stop the test execution and to mark the test case as "FAILED" or "ERROR". However, especially during the test development process itself, it is often useful to be able to proceed after an error. For example, when the test consists of a web-browser's input field values being checked and validated, you may want to continue after a validation error. With this toggle set, the executor goes into the pause-state whenever an error is encountered, and execution can be continued by pressing the "Run" or "Single Step" button again (i.e. it behaves as if "Pause" was pressed).
- "Hard Stop" - Hard-Terminate the execution of the current test run, without doing any exception/cleanup handling. This is useful e.g. for recursion hang-ups, or if a block's cleanup action leads to another blocking situation.
- Follow execution: auto-select the currently active block in the log (while running).
- In the log, select the next activity which has finished with an error.
- In the log, select the next activity which has finished with an inconclusive state.
- In the log, select the next activity which is currently executing.
The remaining edit-related and breakpoint buttons are described in the network editor.
- Save Log...
- This option allows saving the current logging information into an expecco log file or a XML formatted file. Selecting it brings up a file chooser to determine the target path of the log file.
- Load Log...
- This option allows loading previously saved logging information from an expecco log file or a XML formatted file. Selecting it brings up a file chooser to determine the source path of the log file.
- Remove all Results in System
- This option is used to clear all generated log data in the system. No reports can be generated from any more.
- Remove Result
- This option is used to clear the last generated log data. No reports can be generated from any more.
The activity tree displays a hierarchical view of the activities that are invoked during the test run, each with a single entry in the tree. If an activity spawns sub activities, these sub activities are enlisted in the branch of the spawning activity, and you can expand or collapse the branch with the node browsing item. Only the first level is expanded automatically when the test run is started. Activities that perform elementary actions never have sub activities, while those that perform compound actions always have at least one sub activity (else they fail due to lack of executable steps inside). Activities on the same level are sorted chronologically by invocation time.
The symbol in front of the activity indicates the state of the activity. This can be either in progress or one of the testresults "PASSED", "FAIL", "INCONCLUSIVE" or "ERROR".
A context menu is available in the activity tree view, relating to the Activity that is currently selected:
- Find Next Error
- This option switches the selection in the tree from the currently selected activity to the next, failed activity in order of the tree hierarchy.
- Generate Report from here...
- This option generates a report of the selected activity and all nested activities.
- Open page on selected Item
- This opens a new page on the block corresponding to the selected activity.
This view is shown for compound activities. It displays the activity diagram and highlights steps as they are executed. A step's color reflects its execution state:
About to be executed (i.e. ready, but not yet started)
- light green
finished with error
finished inconclusive (aborted or skipped)
Note that in this view, editing is not possible. You can, however, press <ENTER> after selecting a step, causing a new browser page to open up, showing the selected step's diagram.
This view shows the log created by the activity. The above buttons can be used to show or hide different message types. The checkbox "Sublogs" can be used to show or hide the logs of sub activities.
Pin Data View
This view is available for all types of activities. It shows two lists, the input pin value list to the left, and the output pin value list to the right. In each of the lists, three columns arrange the information. The first field of each row displays the pin name, the second one displays the value, and the third one displays the sender, or the receiver, respectively.
Note that input information is not changing after an activity was spawned, while output values can still be produced or changed during the execution of the activity. In each list, only one row can be selected at a time. According to the selected row, the above icon buttons can be used:
- Jump to the sender of the value. Only available for input pins when a sender object is available. This causes the selection in the activity tree to change to the activity that sent the selected value to this activity.
- Jump to the receiver of the value. Only available for output pins when a receiver object is available. This causes the selection in the activity tree to change to the activity that received the selected value from this activity.
- Inspect the value object.
When a compound action's diagram is executed, a read-only version of the diagram editor is showing the state of the execution. There, steps are colored according to their execution state. The color setting is part of the user settings and can be changed according to your personal taste (or, if you suffer from red-green color blindness, for example). The defaults are:
- grey/original color
the step was not yet executed
the step is about to be executed. This means that it has all of its input values available and is ready to run. However, its code or network is not yet being executed - either because other steps are still running and the maximum execution parallelity has been reached, or because the diagram is single stepped, and waiting for you to press the "Step Next" button.
- bright green
the step is being executed
- normal green
the step has finished execution with success
the step has finished execution with success, but a warning was generated during its execution. Select the step and select the "Log" tab for details.
- normal red
the step finished execution with an error
- dark red
the step finished execution with a failure
- dark grey
the step was skipped or its execution was aborted. Either due to the user initiating an explicit abort (Stop or Hard Stop button), or because another step forced the containing diagram's execution to stop
Running a Trial Test of the Block
To invoke the test wrapper, click on the start button in the editor's tool bar. The network is executed as usual, by starting with steps marked as autostart. If logging is enabled, the lower monitoring panel monitors the execution status. If logging is not enabled, the monitoring panel will be cleared and remains empty. Logging is initially enabled.
The monitoring panel is split vertically, showing the activity tree on the left, and the observation panel to the right, which displays the current state of the selected activity. Select any activity to see the details in the right area. The depth of the activity tree can be limited by setting a log depth limit. See settings.
The "normal" reaction to errors during a testplan-run is to finish the test plan with a FAIL or ERROR status. This behavior is OK for test execution; however, during test development, it is helpful to get a symbolic debugger which gives more detail about what happened, the current state (i.e. variables) and how the program got there (the back-trace). The debugger can be enabled via the global settings dialog. These settings are also used if you press the "Run"-button in this test-block runner. However, the additional "Run with Debug" button starts executing with the debugger enabled, ignoring the global settings.
Modifying the Program while Executing
It is possible to change the running program. This may sound a weird thing to do at first, but is very useful when test cases are created in an explorative manner, or changes have to be made to a long running test, and you do not want (or cannot) restart the test from the beginning. You can change the program at any time, however, any action being already executed continues execution until finished in the old action definition. Only then will any changed definitions become effective. This means:
- if an elementary block is currently being executed, the changed program text will be effective when the elementary block is called the next time
- if a compound action is changed, it will use the new activity diagram, when invoked the next time.
To make code and diagram changes an atomic action, all changes are first done to a temporary edit-copy, which is finally installed as an atomic action when the "Accept" button is pressed. If multiple actions are to be edited, any ongoing execution should be paused first, and proceeded once all changes have been done.
Steps in an Activity Diagram
Either place a breakpoint on a step and proceed by pressing the "Single Step" button, or start the execution right from the start via the "Single Step" button. There is also a "breakpoint"-action, which can be placed into the network. By conditionally enabling these steps, conditional breakpoints can be added. When a breakpoint is reached, you can:
- continue single stepping (by pressing the "Step" button),
- cancel execution (press the "Stop" or "Hard Stop" button), or
- continue with normal execution (press the "Run" button).
Notice that there are now two different single step buttons; one is showing steps as executed in the currently visible network, the other ("Step In") also halts before any step in any subnetwork is about to be executed.
Code Lines in an Elementary Action
Place a line such as:
(in Smalltalk code, write "
somewhere in your elementary block's code. When executed, a debugger will be opened, highlighting the line containing the code-breakpoint.
In the debugger, you press "Next Line" to single step or "Continue" to proceed.
The debugger also shows the calling chain (how did I get there) in the top pane, the activities local variables and temporary variables in the lower pane.
The debugger is described in detail in http://live.exept.de/doc/online/english/tools/debugger/TOP.html.
Premature Stop of a Test
Occasionally, it might be required to stop a test even though it has not completed. For example, if you detect an error or malfunction which is not detected by the test, or the test is caught in an endless loop due to an unexpected situation. In this case, press the "Stop" button. This button's function is to ask the test-executor for a controlled termination of the run. Controlled means, that any cleanup actions (post-execution actions) are to be executed. This is the preferred way to stop test execution, because those cleanup actions are meant to free any acquired resources, remove temporary files or to turn off hardware equipment which was acquired and configured by the test. I.e. to bring the state of your equipment back to a well known (initial) state. However, sometimes, the cleanup actions themselves might encounter trouble. For example, if your test requires a login procedure to a remote machine, and the cleanup tries to perform a clean logout, this may block if for example the connection is lost during the test. In this case, the cleanup action will not be able to perform its logout and may even block forever.
To handle this situation gracefully, expecco will wait for some pre-configured time for all the cleanup actions to complete (this is a configurable settings value found in the "extra"-"execution settings"-dialog). After that, a dialog window will appear, asking if the cleanup should be aborted or if you want to continue waiting for the cleanup.
If you are certain that no cleanup action is required, or if you know in advance, that it will fail or block, use the "Hard Stop" button, which terminates execution without even attempting to perform any cleanup actions. Use this with care, as it may leave allocated resources and devices in a non-determined state.