Test Editor/en: Unterschied zwischen den Versionen

Aus expecco Wiki (Version 2.x)
Zur Navigation springen Zur Suche springen
 
(56 dazwischenliegende Versionen desselben Benutzers werden nicht angezeigt)
Zeile 6: Zeile 6:
:Shows information of the current or last execution of the network under test.
:Shows information of the current or last execution of the network under test.


The separation was made to make better use of the screen (which was too filled for small displays) and also to reuse the execution output view in the [[GUI Browser/en|GUI-Browser]]. This document describes the previous combined editor.
This should not be used for "official" test-runs; instead, use it for the "private" execution of a single block

(i.e. unit tests for the test-suite itself) or as a place for demonstration or "executable documentation".
=== As Part of an Action Block Definition ===
Every action can have an associated "''Test Network''" which is shown in the "''Test/Demo''" tab. The test network typically shows how the action is to be used and/or contains a unit test of the action.
We highly recommend that all of your actions provide an executable example of its use, both as a "unit test" and as additional documentation. It may also be used as a place to provide ready-to-use setups, which can be copied/pasted into other networks.

The "''Test/Demo''" network should not be used for "official" test-runs. Instead, always put real tests into a testplan and only use the test/demo network for the "private" execution of a single block
(i.e. unit tests for the action itself) or as a place for demonstration or "executable documentation".



To the right, you see a test editor after the successful execution of a test-block.
To the right, you see a test editor after the successful execution of a test-block.

=== As Part of the GUI Browser ===

In this context, the test run tab presents the outcome of the last
execution within the GUI browser. This can be either the single action or the complete sequence as recorded so far.

=== As Part of the Testplan Editor ===

Here the outcome of the selected test case is presented.


= Buttons =
= Buttons =


*[[Bild:Icon Run.png]] "Run" <br> Starts execution. All auto started steps will be triggered. Unless enabled in the execution settings, no debugger is opened when an error occurs. Instead, the execution is stopped with a failed/error status (shown in red)<p></p>
*[[Bild:Icon Run.png]] "Run" <br> Starts or continues execution. All auto started steps will be triggered.<br>Unless enabled in the execution settings, no debugger is opened when an error occurs or a breakpoint is hit. Instead, on error or fail, the execution is terminated and the failed/error status remembered in the log (shown in red).<br>Halts or breakpoints are ignored.<p></p>


*[[Bild:Icon Run Debug.png]] "Debug Run" <br>Start the test execution in debug mode causing the debugger to open on exceptions (regardless of what is specified in the execution settings).<p></p>
*[[Bild:Icon Run Debug.png]] "Debug Run" <br>Start or continue the test execution in debug mode causing the debugger to open on exceptions (regardless of what is specified in the execution settings) and also when a breakpoint is hit.<br>In addition, skipInTrace and doNotLog attributes are ignored. Thus, all activities will be logged.<p></p>


*[[Bild:Icon Run Single Step.png]] "Step"<br>Start the test execution in single step mode or continue single stepping (when at a breakpoint).<p></p>
*[[Bild:Icon Run Single Step.png]] "Step"<br>Start the test execution in single step mode or continue single stepping (when at a breakpoint).<p></p>
Zeile 29: Zeile 45:
*[[Bild:Icon PauseOnError.png]] "Pause on Error" <br>This toggle controls how the executor behaves when an error is encountered. The default is to stop the test execution and to mark the test case as "FAILED" or "ERROR". However, especially during the test development process itself, it is often useful to be able to proceed after an error. For example, when the test consists of a web-browser's input field values being checked and validated, you may want to continue after a validation error. With this toggle set, the executor goes into the pause-state whenever an error is encountered, and execution can be continued by pressing the "Run" or "Single Step" button again (i.e. it behaves as if "Pause" was pressed).<p></p>
*[[Bild:Icon PauseOnError.png]] "Pause on Error" <br>This toggle controls how the executor behaves when an error is encountered. The default is to stop the test execution and to mark the test case as "FAILED" or "ERROR". However, especially during the test development process itself, it is often useful to be able to proceed after an error. For example, when the test consists of a web-browser's input field values being checked and validated, you may want to continue after a validation error. With this toggle set, the executor goes into the pause-state whenever an error is encountered, and execution can be continued by pressing the "Run" or "Single Step" button again (i.e. it behaves as if "Pause" was pressed).<p></p>


*[[Bild:Icon Hard Stop.png]] "Hard Stop" <br>Hard-Terminate the execution of the current test run, without doing any exception/cleanup handling. This is useful e.g. for recursion hang-ups, or if a block's cleanup action leads to another blocking situation.
*[[Bild:Icon Hard Stop.png]] "Hard Stop" <br>Hard-Terminate the execution of the current test run, without doing any exception/cleanup handling. This is useful e.g. for recursion hang-ups, or if a block's cleanup action leads to another blocking situation.<p></p>
<!-- no longer; only in testplan runner
<!-- no longer; only in testplan runner
*[[Bild:Icon Logging Menu.png]] Open the logging dropdown menu<p></p>
*[[Bild:Icon Logging Menu.png]] Open the logging dropdown menu<p></p>
Zeile 45: Zeile 61:
*[[Bild:Icon BackToPreviouslyVisited.png]] "Previously Visited"<br>In the log, select the previously visited activity (i.e. back in visited-history).
*[[Bild:Icon BackToPreviouslyVisited.png]] "Previously Visited"<br>In the log, select the previously visited activity (i.e. back in visited-history).


The remaining edit-related and breakpoint buttons are described in the [[CompoundBlock Editor-CompoundWorksheet Editor/en#Buttons|network editor]].
The remaining edit- and breakpoint-related buttons are described in the [[CompoundBlock Editor-CompoundWorksheet Editor/en#Buttons|network editor]].


<!-- only present in the testplan-run-editor !
<!-- only present in the testplan-run-editor !
Zeile 67: Zeile 83:


= Views =
= Views =
After a run, the recorded ''Activity Log'' is presented as a hierarchical tree of executed actions. Select individual actions the tree to see the action's details (inputs, outputs and generated logging messages).

== Activity Tree ==
== Activity Tree ==
The activity tree displays a hierarchical view of the activities that are invoked during the test run, each with a single entry in the tree. If an activity spawns sub activities, these sub activities are enlisted in the branch of the spawning activity, and you can expand or collapse the branch with the node browsing item. Only the first level is expanded automatically when the test run is started. Activities that perform elementary actions never have sub activities, while those that perform compound actions always have at least one sub activity (else they fail due to lack of executable steps inside). Activities on the same level are sorted chronologically by invocation time.
The activity tree displays a hierarchical view of the activities that are invoked during the test run, each with a single entry in the tree. If an activity spawns sub activities, these sub activities are enlisted in the branch of the spawning activity, and you can expand or collapse the branch with the node browsing item. Only the first level is expanded automatically when the test run is started. Activities that perform elementary actions never have sub activities, while those that perform compound actions always have at least one sub activity (else they fail due to lack of executable steps inside). Activities on the same level are sorted chronologically by invocation time.
Zeile 102: Zeile 120:


The toolbar buttons show or hide different message types. The checkbox named "''Sublogs''" shows or hides the logs of subactivities.
The toolbar buttons show or hide different message types. The checkbox named "''Sublogs''" shows or hides the logs of subactivities.

== Timeline View ==
This pane is new with version 23.2. It shows a graphical presentation of the execution times.


== Pin Data View ==
== Pin Data View ==
Zeile 113: Zeile 134:
: Jump to the receiver (the sink) of the value. Only available for output pins when a receiver object is available. This causes the selection in the activity tree to change to the activity that received the selected value from this activity.
: Jump to the receiver (the sink) of the value. Only available for output pins when a receiver object is available. This causes the selection in the activity tree to change to the activity that received the selected value from this activity.
;[[Bild:Icon Inspect Value.png]]
;[[Bild:Icon Inspect Value.png]]
: Inspect the value object.
: Inspect the value object. As a shortcut, double click on the pin value's row.

The popup menu (right click) depends on the selection. If no row is selected, functions operating on all values are provided. Otherwise, functions on the selected item are offered. The selection can be toggled by pressing CTRL with the mouse click.


= State Colors =
= State Colors =
When a compound action's diagram is executed, a read-only version of the diagram editor is showing the state of the execution.
When a compound action's diagram is executed, a read-only version of the diagram editor is showing the state of the execution.
There, steps are colored according to their execution state. The color setting is part of the user settings and
There, steps are colored according to their [[Glossary/en#Verdict|execution state (verdict)]]. The color setting is part of the user settings and
can be changed according to your personal taste (or, if you suffer from red-green color blindness, for example).
can be changed according to your personal taste (or, if you suffer from red-green color blindness, for example).
The defaults are:
<br>The defaults are:
* grey/original color<br>the step was not yet executed
* grey/original color<br>the step was not yet executed


* blue<br>the step is about to be executed. This means that it has all of its input values (and possibly any required resources) are available and it is ready to run. However, its code or network is not yet being executed - either because other steps are still running and the maximum execution parallelity has been reached, or because the diagram is single stepped, and waiting for you to press the "Step Next" button.
* blue<br>the step is about to be executed. This means that it has all of its input values (and possibly any required resources) available and it is ready to run. However, its code or network is not yet being executed - either because other steps are still running and the maximum execution parallelity has been reached, or because the diagram is being single stepped and waiting for you to press the "''Step Next''" button.


* bright green<br>the step is being executed
* bright green (lime)<br>the step is being executed


* normal green<br>the step has finished execution with success
* normal green (darker green)<br>the step has finished execution with success


* yellow<br>the step has finished execution with success, but a warning was generated during its execution. Select the step and select the "Log" tab for details.
* yellow<br>the step has finished execution with success, but a warning was generated during its execution or an exception was caught or ignored. Select the step and the "Log" tab for details.


* normal red<br>the step finished execution with an error
* normal red<br>the step finished execution with an unhanded error


* dark red<br>the step finished execution with a failure
* dark red<br>the step finished execution with a failure


* dark grey<br>the step was skipped or its execution was aborted. Either due to the user initiating an explicit abort ("Stop" or "Hard Stop" button), or because another step forced the containing diagram's execution to stop
* dark grey<br>the step was skipped or its execution was aborted. Either due to the user initiating an explicit abort ("''Stop''" or "''Hard Stop''" button), or because another step forced the containing diagram's execution to stop.


= Tasks =
= Tasks =
Zeile 142: Zeile 165:
The monitoring panel is split vertically, showing the activity tree on the left, and the observation panel to the right, which displays the current state of the selected activity. Select any activity to see the details in the right area. The depth of the activity tree can be limited by setting a log depth limit. See [[Settings/en#Logging Settings|settings]].
The monitoring panel is split vertically, showing the activity tree on the left, and the observation panel to the right, which displays the current state of the selected activity. Select any activity to see the details in the right area. The depth of the activity tree can be limited by setting a log depth limit. See [[Settings/en#Logging Settings|settings]].


If the "follow execution" toggle([[Bild:Icon Follow Execution.png]]) has been checked, the selection in the tree automatically follows the currently executing activity.
If the "follow execution" toggle([[Bild:Icon Follow Execution.gif]]) has been checked, the selection in the tree automatically follows the currently executing activity.


==Debugging==
==Debugging==
The "normal" reaction to errors during a testplan-run is to finish the test plan with a FAIL or ERROR status. This behavior is OK for test execution; however, during test development, it is helpful to get a symbolic debugger which gives more detail about what happened, the current state (i.e. variables) and how the program got there (the back-trace).
The "normal" reaction to errors during a testplan-run is to finish the test plan with a FAIL or ERROR status. This behavior is OK for test execution; however, during test development, it is helpful to get a symbolic debugger which gives more detail about what happened, the current state (i.e. variables) and how the program got there (called "''backtrace''" or "''walkback''" or "''call chain''" in the literature).

The debugger can be enabled via the global settings dialog. These settings are also used if you press the "Run"-button in this test-block runner. However, the additional "Run with Debug" button starts executing with the debugger enabled, ignoring the global settings.
The debugger can be enabled via the global settings dialog. These settings are also used if you press the "''Run''"-button in this test-block runner. However, the additional "''Run with Debug''" button starts executing with the debugger enabled, ignoring the global settings.


==Modifying the Program while Executing==
==Modifying the Program while Executing==
It is possible to change the running program. This may sound a weird thing to do at first, but is very useful when test cases are created in an explorative manner, or changes have to be made to a long running test, and you do not want (or cannot) restart the test from the beginning.
It is possible to change the running program. This may sound a weird thing to do at first, but is very useful when test cases are created in an explorative manner, or changes have to be made to a long running test, and you do not want (or cannot) restart the test from the beginning.


You can change the program at any time (simply edit the action in another tab or window and accept). However, any action being already executed continues execution until finished in the old action definition. Only then (when invoked for the next time) will any changed definitions become effective.
You can change the program at any time (simply edit the action in another tab or window and accept). However, any action being already executed continues execution until finished in the old action definition. Only then (when invoked for the next time) will any changed definition become effective.


This means:
This means:
Zeile 157: Zeile 181:
* if a compound action is changed, it will use the new activity diagram, when invoked the next time.
* if a compound action is changed, it will use the new activity diagram, when invoked the next time.


To make code and diagram changes an atomic action, all changes are first done to a temporary edit-copy, which is finally installed as an atomic action when the "Accept" button is pressed. If multiple actions are to be edited, any ongoing execution should be paused first, and proceeded once all changes have been done.
To make code and diagram changes an atomic action, all changes are first done to a temporary edit-copy, which is finally installed as an atomic action when the "''Accept''" button is pressed. If multiple actions are to be edited, any ongoing execution should be paused first, and proceeded once all changes have been done.


==Single Stepping and Breakpoints==
==Single Stepping and Breakpoints==
===Steps in an Activity Diagram===
===Steps in an Activity Diagram===
Either place a breakpoint on a step and proceed by pressing the "Single Step" button, or start the execution right from the start via the "Single Step" button. There is also a "breakpoint"-action, which can be placed into the network. By conditionally enabling these steps, conditional breakpoints can be added. When a breakpoint is reached, you can:
Either place a breakpoint on a step and proceed by pressing the "''Single Step''" button, or start the execution right from the start via the "''Single Step''" button. There is also a "breakpoint"-action, which can be placed into the network. By conditionally enabling these steps, conditional breakpoints can be added.<br>When a breakpoint is reached, you can:
* continue single stepping (by pressing the "Step" button),
* continue single stepping (by pressing the "Step" button),
* cancel execution (press the "Stop" or "Hard Stop" button), or
* cancel execution (press the "Stop" or "Hard Stop" button), or
* continue with normal execution (press the "Run" button).
* continue with normal execution (press the "Run" button).


Notice that there are now two different single step buttons; one is showing steps as executed in the currently visible network, the other ("Step In") also halts before any step in any subnetwork is about to be executed.
Notice that there are now two different single step buttons; one is showing steps as executed in the currently visible network, the other ("''Step In''") also halts before any step in any subnetwork is about to be executed.


===Code Lines in an Elementary Action===
===Code Lines in an Elementary Action===
Zeile 176: Zeile 200:
somewhere in your elementary block's code. When executed, a debugger will be opened, highlighting the line containing the code-breakpoint.
somewhere in your elementary block's code. When executed, a debugger will be opened, highlighting the line containing the code-breakpoint.


As an alternative, place a line breakpoint by double-clicking beside the code line in the left side-bar area (a red brakpoint sign will be shown then). Notice that line breakpoints can only be placed at expressions starting in that very line, whereas the above described <CODE>halt</CODE> can be placed anywhere, where an expression is allowed.
As an alternative, place a line breakpoint by double-clicking beside the code line in the left side-bar area (a red breakpoint sign will be shown then). Notice that line breakpoints can only be placed at expressions starting in that very line, whereas the above described <CODE>halt</CODE> can be placed anywhere, where an expression is allowed.


In the debugger, press one of
In the debugger, press one of
* "Next Line" to single step to the next code line
* "''Next Line''" to single step to the next code line
* "Step" to single step over the next expression
* "''Step''" to single step over the next expression
* "Send" to step into the next expression
* "''Send''" to step into the next expression
* "Continue" to proceed.
* "''Continue''" to proceed.
* "Abort" to stop the execution
* "''Abort''" to stop the execution


The debugger also shows the calling chain (how did I get there) in the top pane, and the activity's local variables and temporary variables in the lower pane.
The debugger also shows the calling chain (how did I get there) in the top pane, and the activity's local variables and temporary variables in the lower pane.
Zeile 191: Zeile 215:
==Premature Stop of a Test==
==Premature Stop of a Test==
Occasionally, it might be required to stop a test even though it has not completed. For example, if you detect an error or malfunction which is not detected by the test, or the test is caught in an endless loop due to an unexpected situation.
Occasionally, it might be required to stop a test even though it has not completed. For example, if you detect an error or malfunction which is not detected by the test, or the test is caught in an endless loop due to an unexpected situation.
In this case, press the "''Stop''" button.
In this case, press the "Stop" button. This button's function is to ask the test-executor for a controlled termination of the run. Controlled means, that any cleanup actions (post-execution actions) are to be executed.

The "''Stop''" button asks the test-executor for a controlled termination of the run. "''Controlled''" means, that any cleanup actions (post-execution actions) are to be executed.

This is the preferred way to stop test execution, because those cleanup actions are meant to free any acquired resources, remove temporary files or to turn off hardware equipment which was acquired and configured by the test. I.e. to bring the state of your equipment back to a well known (initial) state.
This is the preferred way to stop test execution, because those cleanup actions are meant to free any acquired resources, remove temporary files or to turn off hardware equipment which was acquired and configured by the test. I.e. to bring the state of your equipment back to a well known (initial) state.
However, sometimes, the cleanup actions themselves might encounter trouble. For example, if your test requires a login procedure to a remote machine, and the cleanup tries to perform a clean logout, this may block if for example the connection is lost during the test. In this case, the cleanup action will not be able to perform its logout and may even block forever.


However, sometimes, a cleanup action might itself encounter trouble. For example, if your test requires a login procedure to a remote machine, and the cleanup tries to perform a logout, this may block if the connection was lost during the test. In this case, the cleanup action will not be able to perform its logout and may even block forever.
To handle this situation gracefully, expecco will wait for some pre-configured time for all the cleanup actions to complete (this is a configurable settings value found in the "extra"-"execution settings"-dialog). After that, a dialog window will appear, asking if the cleanup should be aborted or if you want to continue waiting for the cleanup.

To handle this situation gracefully, expecco will wait for some pre-configured time for all the cleanup actions to complete (this is a configurable settings value found in the "''Extras''" &#8594; "''Execution Settings''"-dialog). After that, a dialog window will appear, asking if the cleanup should be aborted or if you want to continue waiting for the cleanup.

If you are certain that no cleanup action is required, or if you know in advance, that it will fail or block, use the "''Hard Stop''" button, which terminates the execution without even attempting to perform any cleanup actions.
Use this with care, as it may leave allocated resources and devices in a non-determined state. It may also lead to file streams or sockets being left open. You can repair this (i.e. manually close any streams) via the "''Extra''" - "''Debugging''" menu, where you fild menu items to find and/or close those streams.
<BR>Notice that unreferenced resources are usually automatically freed after some time via the automatic memory manager's finalization mechanism. However, this would not work if eg. an open file stream is still referenced by being held in an activity log or an expecco environment variable.

= Typical Error Messages when Execution Fails =

=== Empty action "..." (no steps) ===

Obviously, there is no action step at all in your diagram. This makes no sense in a production test suite, and is therefore reported as an error.
However, when developing tests, it may be useful to create dummy actions as placeholders for "to-be-implemented" actions. Then, got to "''Extras''" - "''Settings''" - "''Execution''" - "''Debugging''" and check the "''Allow empty Actions''" toggle.

Solutions:
* add some dummy action to you network. A step which writes a log message such as "to be implemented" is a good idea. This can be written to either the activity log (and will therefore appear in any printed test report) or to the Transcript (console).

* check "''Allow empty Actions''" in the settings.

=== No executable step in action "Test/Demo" ===

No inner action step could be executed inside the diagram.
<br>This happens if:
* an explicit "Start" step is missing,
* and/or no step has the "autostart" attribute set (which is equivalent to a start step)
* and no inner step got a value at one of its required triggering pins (from the compound action).
Solution:
:typically, you forgot to provide an input value to the tested compound action in the "Test/Demo" diagram.

=== No value for input pin "..." of step "..." ===
An inner step was started, but got no input value at one of its required pins.
The inner step was started (probably due to autostart or an explicit trigger, bt not due to an input value arriving).

Solution:
:make sure the inner pin is connected, and also that it will receive a value at the input pin(s). If it has an autostart or explicit trigger, check if that is really what you intended. In most cases, steps should be triggered by incoming data, and not only by control flow triggers (but to have both is perfectly ok).
:
:Be especially careful when steps are in a loop: often an input value is "consumed"in the first iteration, and then no value is present in further loop cycles. Then, make the input a "non-consuming" (parameter) pin, to preserve the value across loop cycles.


= See Also =
If you are certain that no cleanup action is required, or if you know in advance, that it will fail or block, use the "Hard Stop" button, which terminates execution without even attempting to perform any cleanup actions.
API for the builtin and bridged languages: "[[Expecco API]]",
Use this with care, as it may leave allocated resources and devices in a non-determined state. It may also lead to file streams or sockets being left open. You can repair this (i.e. close the streams) via the "Extra" - "Debugging" menu, where you fild menu items to find and/or close those streams.
"[[Expecco_API/en#Pin Functions | Pin-API ]]" and "[[ Expecco_API/en#Current Activity | Activity-API ]]"
<br>[[Expecco_API/en#JavaScript_and_Smalltalk_Elementary_Blocks|API for Smalltalk/JavaScript actions]]
<br>[[Expecco_API/en#Groovy_Elementary_Blocks|API for Java/Groovy actions]]
<br>[[Expecco_API/en#Bridged_Python_Elementary_Blocks|API for Python actions]]
<br>[[Expecco_API/en#Node.js_.28Bridged.29_Elementary_Blocks|API for NodeJS actions]]
<br>[[Expecco_API/en#Bridged_Ruby_Elementary_Blocks|API for Ruby actions]]
<br>[[Expecco_API/en#Bridged C Elementary Blocks|API for C actions]]
<br>Scripted actions: "[[ElementaryBlock_Element/en#Script_Action_Blocks | Script Action Blocks]]"
<br>[[ElementaryBlock_Element/en#Bridge_Action_Blocks_vs._Script_Action_Blocks | Bridged Actions vs. Script Actions]]"
<br>[[Code_Editor/en|Code Editor]]
<br>[[GUI Browser/en|GUI-Browser]]


[[Category:Editors]]
[[Category:Editors]]

Aktuelle Version vom 6. September 2023, 11:33 Uhr

Test Editor after a run

Notice: starting with release 1.8, this editor has been split into separate editor and execution tabs:

Test/Demo
Definition of the network under test. This is used to define a test or demo for the edited action. As such, it combines features of the network editor.
Run
Shows information of the current or last execution of the network under test.

The separation was made to make better use of the screen (which was too filled for small displays) and also to reuse the execution output view in the GUI-Browser. This document describes the previous combined editor.

As Part of an Action Block Definition[Bearbeiten]

Every action can have an associated "Test Network" which is shown in the "Test/Demo" tab. The test network typically shows how the action is to be used and/or contains a unit test of the action. We highly recommend that all of your actions provide an executable example of its use, both as a "unit test" and as additional documentation. It may also be used as a place to provide ready-to-use setups, which can be copied/pasted into other networks.

The "Test/Demo" network should not be used for "official" test-runs. Instead, always put real tests into a testplan and only use the test/demo network for the "private" execution of a single block (i.e. unit tests for the action itself) or as a place for demonstration or "executable documentation".


To the right, you see a test editor after the successful execution of a test-block.

As Part of the GUI Browser[Bearbeiten]

In this context, the test run tab presents the outcome of the last execution within the GUI browser. This can be either the single action or the complete sequence as recorded so far.

As Part of the Testplan Editor[Bearbeiten]

Here the outcome of the selected test case is presented.

Buttons[Bearbeiten]

  • Icon Run.png "Run"
    Starts or continues execution. All auto started steps will be triggered.
    Unless enabled in the execution settings, no debugger is opened when an error occurs or a breakpoint is hit. Instead, on error or fail, the execution is terminated and the failed/error status remembered in the log (shown in red).
    Halts or breakpoints are ignored.

  • Icon Run Debug.png "Debug Run"
    Start or continue the test execution in debug mode causing the debugger to open on exceptions (regardless of what is specified in the execution settings) and also when a breakpoint is hit.
    In addition, skipInTrace and doNotLog attributes are ignored. Thus, all activities will be logged.

  • Icon Run Single Step.png "Step"
    Start the test execution in single step mode or continue single stepping (when at a breakpoint).

  • Icon Run SingleStepInto.png "Step Into"
    Like the above, but steps into other compound actions.

  • Icon Pause.png "Pause"
    Pause the current test run. To proceed or single step, click on the according button.

  • Icon Stop.png "Stop"
    Stop the execution of the current test run. Exception handling & cleanup (post actions) will be performed.

  • Icon InterruptRun.png "Interrupt" (new in 18.1 aka "2.12")
    Interrupt the execution of the current test run and open a debugger. Useful to debug endless loops or to inspect the execution state in a long running elementary action. In the debugger, the execution can be proceeded, single stepped or aborted.

  • Icon PauseOnError.png "Pause on Error"
    This toggle controls how the executor behaves when an error is encountered. The default is to stop the test execution and to mark the test case as "FAILED" or "ERROR". However, especially during the test development process itself, it is often useful to be able to proceed after an error. For example, when the test consists of a web-browser's input field values being checked and validated, you may want to continue after a validation error. With this toggle set, the executor goes into the pause-state whenever an error is encountered, and execution can be continued by pressing the "Run" or "Single Step" button again (i.e. it behaves as if "Pause" was pressed).

  • Icon Hard Stop.png "Hard Stop"
    Hard-Terminate the execution of the current test run, without doing any exception/cleanup handling. This is useful e.g. for recursion hang-ups, or if a block's cleanup action leads to another blocking situation.

  • Icon Print Report.png "Report"
    Generate a report from the log data

  • Icon Follow Execution.gif "Follow execution"
    To enabled/disable auto-select of the currently active block in the log (animation while running).
    This button also provides a drop-down box (right click or press-and-hold), to select the number of levels and a tag filter, to further control which actions should be followed. The button's icon image shows a "minus" when automatic follow is disabled.
    If the tag filter is set, it gives a list of match-patterns (GLOB patterns) separated by semicolon. Automatic follow will then only show action which are both within any level-limit and which have a tag which matches any in the given tag filter pattern

  • Icon FindNextError.png "Find Error"
    In the log, select the next activity which has finished with an error.

  • Icon Next Inconclusive.png "Find Inconclusive"
    In the log, select the next activity which has finished with an inconclusive state.

  • Icon FindNextActive2.png "Find Active"
    In the log, select the next activity which is currently executing.
  • Icon BackToPreviouslyVisited.png "Previously Visited"
    In the log, select the previously visited activity (i.e. back in visited-history).

The remaining edit- and breakpoint-related buttons are described in the network editor.


Views[Bearbeiten]

After a run, the recorded Activity Log is presented as a hierarchical tree of executed actions. Select individual actions the tree to see the action's details (inputs, outputs and generated logging messages).

Activity Tree[Bearbeiten]

The activity tree displays a hierarchical view of the activities that are invoked during the test run, each with a single entry in the tree. If an activity spawns sub activities, these sub activities are enlisted in the branch of the spawning activity, and you can expand or collapse the branch with the node browsing item. Only the first level is expanded automatically when the test run is started. Activities that perform elementary actions never have sub activities, while those that perform compound actions always have at least one sub activity (else they fail due to lack of executable steps inside). Activities on the same level are sorted chronologically by invocation time.

The symbol in front of the activity indicates the state of the activity. This can be either in progress or one of the test results (verdicts): "PASSED", "FAIL", "INCONCLUSIVE" or "ERROR".

Context Menu

A context menu is available in the activity tree view, relating to the Activity that is currently selected:

Find Next Unhandled Error
This option switches the selection in the tree from the currently selected activity to the next, failed activity for which no error handling was performed (and which was not ignored). The search is in the order of the tree hierarchy (i.e. deep-first search, starting at the current selection).
Find Next Error
Similar to the above, but this menu function will find both handled and unhandled errors and failures.
Find Previous Unhandled Error
Similar to the above, but searches backward for the previous unhandled failure or error.
Find Previous Error
Similar to the above, but searches for any error (both handled and unhandled)
Generate Report from here...
This option generates a report of the selected activity and all nested activities.
Open page on selected Item
This opens a new page on the block corresponding to the selected activity.

Network View[Bearbeiten]

This view is shown for compound activities. It displays the activity diagram and highlights steps as they are executed. A step's color reflects its execution state, as described below in "State Colors".

Note that in this view, editing is not possible. You can, however, press Enter after selecting a step, causing a new browser page to open up, showing the selected step's diagram.

Log View[Bearbeiten]

This view shows the log created by the activity. The log consists of the messages (and possibly attached data or screenshots) as generated by the activity (and subactivities) via the "logInfo" / "logWarning" / "logError" and "logFailure" calls (either from elementary code or via the corresponding action steps in a compound action).

The toolbar buttons show or hide different message types. The checkbox named "Sublogs" shows or hides the logs of subactivities.

Timeline View[Bearbeiten]

This pane is new with version 23.2. It shows a graphical presentation of the execution times.

Pin Data View[Bearbeiten]

This view is available for all types of activities. It shows two lists, the input pin value list to the left, and the output pin value list to the right. In each of the lists, three columns arrange the information. The first field of each row displays the pin name, the second one displays the value, and the third one displays the sender, or the receiver, respectively.

Note that input information is not changing after an activity was spawned, while output values can still be produced or changed during the execution of the activity. In each list, only one row can be selected at a time. According to the selected row, the above icon buttons can be used:

Icon Sender.png
Jump to the sender (the source) of the value. Only available for input pins when a sender object is available. This causes the selection in the activity tree to change to the activity that sent the selected value to this activity.
Icon Receiver.png
Jump to the receiver (the sink) of the value. Only available for output pins when a receiver object is available. This causes the selection in the activity tree to change to the activity that received the selected value from this activity.
Icon Inspect Value.png
Inspect the value object. As a shortcut, double click on the pin value's row.

The popup menu (right click) depends on the selection. If no row is selected, functions operating on all values are provided. Otherwise, functions on the selected item are offered. The selection can be toggled by pressing CTRL with the mouse click.

State Colors[Bearbeiten]

When a compound action's diagram is executed, a read-only version of the diagram editor is showing the state of the execution. There, steps are colored according to their execution state (verdict). The color setting is part of the user settings and can be changed according to your personal taste (or, if you suffer from red-green color blindness, for example).
The defaults are:

  • grey/original color
    the step was not yet executed
  • blue
    the step is about to be executed. This means that it has all of its input values (and possibly any required resources) available and it is ready to run. However, its code or network is not yet being executed - either because other steps are still running and the maximum execution parallelity has been reached, or because the diagram is being single stepped and waiting for you to press the "Step Next" button.
  • bright green (lime)
    the step is being executed
  • normal green (darker green)
    the step has finished execution with success
  • yellow
    the step has finished execution with success, but a warning was generated during its execution or an exception was caught or ignored. Select the step and the "Log" tab for details.
  • normal red
    the step finished execution with an unhanded error
  • dark red
    the step finished execution with a failure
  • dark grey
    the step was skipped or its execution was aborted. Either due to the user initiating an explicit abort ("Stop" or "Hard Stop" button), or because another step forced the containing diagram's execution to stop.

Tasks[Bearbeiten]

Running a Trial Test of the Block[Bearbeiten]

To invoke the test wrapper, click on the start button in the editor's tool bar. The network is executed as usual, by starting with steps marked as autostart. If logging is enabled, the lower monitoring panel monitors the execution status. If logging is not enabled, the monitoring panel will be cleared and remains empty. Logging is initially enabled.

The monitoring panel is split vertically, showing the activity tree on the left, and the observation panel to the right, which displays the current state of the selected activity. Select any activity to see the details in the right area. The depth of the activity tree can be limited by setting a log depth limit. See settings.

If the "follow execution" toggle(Icon Follow Execution.gif) has been checked, the selection in the tree automatically follows the currently executing activity.

Debugging[Bearbeiten]

The "normal" reaction to errors during a testplan-run is to finish the test plan with a FAIL or ERROR status. This behavior is OK for test execution; however, during test development, it is helpful to get a symbolic debugger which gives more detail about what happened, the current state (i.e. variables) and how the program got there (called "backtrace" or "walkback" or "call chain" in the literature).

The debugger can be enabled via the global settings dialog. These settings are also used if you press the "Run"-button in this test-block runner. However, the additional "Run with Debug" button starts executing with the debugger enabled, ignoring the global settings.

Modifying the Program while Executing[Bearbeiten]

It is possible to change the running program. This may sound a weird thing to do at first, but is very useful when test cases are created in an explorative manner, or changes have to be made to a long running test, and you do not want (or cannot) restart the test from the beginning.

You can change the program at any time (simply edit the action in another tab or window and accept). However, any action being already executed continues execution until finished in the old action definition. Only then (when invoked for the next time) will any changed definition become effective.

This means:

  • if an elementary block is currently being executed, the changed program text will be effective when the elementary block is called the next time
  • if a compound action is changed, it will use the new activity diagram, when invoked the next time.

To make code and diagram changes an atomic action, all changes are first done to a temporary edit-copy, which is finally installed as an atomic action when the "Accept" button is pressed. If multiple actions are to be edited, any ongoing execution should be paused first, and proceeded once all changes have been done.

Single Stepping and Breakpoints[Bearbeiten]

Steps in an Activity Diagram[Bearbeiten]

Either place a breakpoint on a step and proceed by pressing the "Single Step" button, or start the execution right from the start via the "Single Step" button. There is also a "breakpoint"-action, which can be placed into the network. By conditionally enabling these steps, conditional breakpoints can be added.
When a breakpoint is reached, you can:

  • continue single stepping (by pressing the "Step" button),
  • cancel execution (press the "Stop" or "Hard Stop" button), or
  • continue with normal execution (press the "Run" button).

Notice that there are now two different single step buttons; one is showing steps as executed in the currently visible network, the other ("Step In") also halts before any step in any subnetwork is about to be executed.

Code Lines in an Elementary Action[Bearbeiten]

In a JavaScript action, place a line such as:

this.halt();

or

halt();

(in Smalltalk code, write "self halt") somewhere in your elementary block's code. When executed, a debugger will be opened, highlighting the line containing the code-breakpoint.

As an alternative, place a line breakpoint by double-clicking beside the code line in the left side-bar area (a red breakpoint sign will be shown then). Notice that line breakpoints can only be placed at expressions starting in that very line, whereas the above described halt can be placed anywhere, where an expression is allowed.

In the debugger, press one of

  • "Next Line" to single step to the next code line
  • "Step" to single step over the next expression
  • "Send" to step into the next expression
  • "Continue" to proceed.
  • "Abort" to stop the execution

The debugger also shows the calling chain (how did I get there) in the top pane, and the activity's local variables and temporary variables in the lower pane.

The debugger is described in detail in http://live.exept.de/doc/online/english/tools/debugger/TOP.html.

Premature Stop of a Test[Bearbeiten]

Occasionally, it might be required to stop a test even though it has not completed. For example, if you detect an error or malfunction which is not detected by the test, or the test is caught in an endless loop due to an unexpected situation. In this case, press the "Stop" button.

The "Stop" button asks the test-executor for a controlled termination of the run. "Controlled" means, that any cleanup actions (post-execution actions) are to be executed.

This is the preferred way to stop test execution, because those cleanup actions are meant to free any acquired resources, remove temporary files or to turn off hardware equipment which was acquired and configured by the test. I.e. to bring the state of your equipment back to a well known (initial) state.

However, sometimes, a cleanup action might itself encounter trouble. For example, if your test requires a login procedure to a remote machine, and the cleanup tries to perform a logout, this may block if the connection was lost during the test. In this case, the cleanup action will not be able to perform its logout and may even block forever.

To handle this situation gracefully, expecco will wait for some pre-configured time for all the cleanup actions to complete (this is a configurable settings value found in the "Extras" → "Execution Settings"-dialog). After that, a dialog window will appear, asking if the cleanup should be aborted or if you want to continue waiting for the cleanup.

If you are certain that no cleanup action is required, or if you know in advance, that it will fail or block, use the "Hard Stop" button, which terminates the execution without even attempting to perform any cleanup actions. Use this with care, as it may leave allocated resources and devices in a non-determined state. It may also lead to file streams or sockets being left open. You can repair this (i.e. manually close any streams) via the "Extra" - "Debugging" menu, where you fild menu items to find and/or close those streams.
Notice that unreferenced resources are usually automatically freed after some time via the automatic memory manager's finalization mechanism. However, this would not work if eg. an open file stream is still referenced by being held in an activity log or an expecco environment variable.

Typical Error Messages when Execution Fails[Bearbeiten]

Empty action "..." (no steps)[Bearbeiten]

Obviously, there is no action step at all in your diagram. This makes no sense in a production test suite, and is therefore reported as an error. However, when developing tests, it may be useful to create dummy actions as placeholders for "to-be-implemented" actions. Then, got to "Extras" - "Settings" - "Execution" - "Debugging" and check the "Allow empty Actions" toggle.

Solutions:

  • add some dummy action to you network. A step which writes a log message such as "to be implemented" is a good idea. This can be written to either the activity log (and will therefore appear in any printed test report) or to the Transcript (console).
  • check "Allow empty Actions" in the settings.

No executable step in action "Test/Demo"[Bearbeiten]

No inner action step could be executed inside the diagram.
This happens if:

  • an explicit "Start" step is missing,
  • and/or no step has the "autostart" attribute set (which is equivalent to a start step)
  • and no inner step got a value at one of its required triggering pins (from the compound action).

Solution:

typically, you forgot to provide an input value to the tested compound action in the "Test/Demo" diagram.

No value for input pin "..." of step "..."[Bearbeiten]

An inner step was started, but got no input value at one of its required pins. The inner step was started (probably due to autostart or an explicit trigger, bt not due to an input value arriving).

Solution:

make sure the inner pin is connected, and also that it will receive a value at the input pin(s). If it has an autostart or explicit trigger, check if that is really what you intended. In most cases, steps should be triggered by incoming data, and not only by control flow triggers (but to have both is perfectly ok).
Be especially careful when steps are in a loop: often an input value is "consumed"in the first iteration, and then no value is present in further loop cycles. Then, make the input a "non-consuming" (parameter) pin, to preserve the value across loop cycles.

See Also[Bearbeiten]

API for the builtin and bridged languages: "Expecco API", " Pin-API " and " Activity-API "
API for Smalltalk/JavaScript actions
API for Java/Groovy actions
API for Python actions
API for NodeJS actions
API for Ruby actions
API for C actions
Scripted actions: " Script Action Blocks"
Bridged Actions vs. Script Actions"
Code Editor
GUI-Browser



Copyright © 2014-2024 eXept Software AG