Manual Test Tutorial/en: Unterschied zwischen den Versionen

Aus expecco Wiki (Version 2.x)
Zur Navigation springen Zur Suche springen
Zeile 90: Zeile 90:
Find the setting by selecting the project (the top item in the left tree),
Find the setting by selecting the project (the top item in the left tree),
and switch to the "''Execution''" tab. The check toggle is found in the "'User Interface''" section.
and switch to the "''Execution''" tab. The check toggle is found in the "'User Interface''" section.

== Programmatic Close ==
The imported library contains an action to programmatically close the wizard.


= Manual Test Action Library =
= Manual Test Action Library =

Version vom 21. April 2021, 03:05 Uhr


Note: sorry for the German screenshots. We'll update ASAP (but things should be clear despite this ;-)

Manual Tests[Bearbeiten]

At first sight, it may seem strange to cover manual tests in the context of "automation". However, there are many situations in which manual operations are required - even in an automated test. For example: power on a device, readout of measured values, visual inspection etc.

In addition, expecco helps you to migrate: if you are already performing manual tests, and you can provide machine readable requirements- or test specifications, then expecco can import those and run them as manual tests.

Although still executed manually, this will give you better reports and - probably even more of significance - the guaranty that the steps where executed properly.

You can also migrate incrementally from fully manual tests, via partially automated tests to fully automated tests, by starting with manual test steps and replacing them with automated actions one by one.

Even these partial automated tests will save time and money, as repeated actions (like filling out forms , confirming dialogs or reading measurement values) run much faster when automated and thereby reduce the overall execution time.

If you are impatient, skip right to the "Step-by-Step Tutorial" further down, and return to the details later.

Combining Manual Test Steps with Automated Steps[Bearbeiten]

You can arbitrarily combine manual steps with any other action step inside a diagram or test. For example, a manual step might ask the user to turn on power, to plug a cable, to perform a visual inspection or to perform any other operation involving human interaction.

Also, partial checks which are not (yet) automated can be included this way. If you put these into separate wrapper actions, these can later be replaced by automated ones, without a need to change other parts of your suite.

Especially remember that such actions can be palced into a separate imported expecco action library, so that later, a simple reimport operation is required to switch to the automated version.

The Manual Test Wizard[Bearbeiten]

If any manual test step is executed by expecco, a dialog window (the so called "Manual Test Wizard) is opened, which guides the user and provides hints about the test to be performed.

The tester will be guided in a step-by-step fashion by the wizard, who can add comments, data or screenshot as documentary of the test run, and confirm each step as successful (possibly with additional notes), as failed or as inconclusive. The "'íncionclusive"outcome means that the test step could not be performed (for example because a device was offline), and therefore, no statement about the proper operation of the tests system could be made. Such tests have to be repeated later.

The wizard's user interface consists of 3 main areas:

  1. Toolbar area
  2. general informationen on the current step
  3. Orders on what do do in the current test step


Manual Test GUI

The Toolbar[Bearbeiten]

The toolbar provides functions functions to set the outcome of the current step, to add comments and to attach data files or screenshots to the report.

Of special relevante are the "Next"buttons (triangular right arrows); these confirm the current step and mark it as "Successful" (green), "Failed" (red) or "Inconclusive" (gray). The yellow button also stands for "Success", but asks for a comment to be added as warning to the final report and log file.

The other buttons are used to attach files or screenshots.

Informationen on the current step[Bearbeiten]

This area show the current step's name, the test case, of which it is part of, and the test plan which is currently running. Some of this depends o whether the action is executed as part of a test plan, or as a single-action run (in the "test/demo"tab of the editor):

Manual Test Info Bar

Whether displayed as TEST-CASE or TEST-STEP depends on corresponding tags of the current action. Although this has no semantic meaning w.r.t. the execution of actions it is useful to organize and structure the suite into layers.

Testates are identified by having a tag named TEST-CASE or TEST_CASE. Either of them should be present in tags found in the documentation area of the test case's action definition. If you create such action via the toolbar above the project tree, these tags will be automatically set. But you can cahnge these tags any time later if required.

Manual Test Side Bar

Test steps are marked with TEST-STEP tag. Likewise, this is to be found in the documentation editor at the bottom. This tag is also automatically applied if the step is created via the toolbar.

Manual Test 4.png

Actions marked as TEST-CASE- or TEST-STEP-Tags are shown with special icons in the tree. This is for them to be easier to find and to be somewhat standing out of other actions (but as already mentioned: has no other semantic meaning).

Manual Test 5.png

Orders on What to Do in Test Step[Bearbeiten]

The wizard's lower area present the test step's description.

Depending on how the test step was defined, this can be presented as simple plain text, or in a more appealing HTML presentation. See more on this in the Standard Teststep doc below.

Simple Plaintext presentation:
Manual Test 6.png

HTML Contents:
Manual Test 7.png

Automatic Close when no longer Needed[Bearbeiten]

By default, the wizard remains open when a manual test step finishes, and waits for the next manual test step to be executed.

If the "'Automatic Close UI when no longer needed" checkbox in the project's execution settings is set, the wizard will stay open for a short time, and close itself if no manual test step is executed within that time. However it will reopen itself if such a step is executed any time later.

Find the setting by selecting the project (the top item in the left tree), and switch to the "Execution" tab. The check toggle is found in the "'User Interface" section.

Programmatic Close[Bearbeiten]

The imported library contains an action to programmatically close the wizard.

Manual Test Action Library[Bearbeiten]

Manual test sequences are usually created by placing individual test steps via drag&drop from the Manual Test Library into a diagram.

In this library you'll find the 3 most important actions to execute manual test steps, in addition to support actions eg. to open/close the wizard etc).

Usually you will not need those, as the wizard opens automatically when the first manual test step is executed. However, they are useful to open the wizard at a particular screen position or with a different size, minimized to not cover the screen, or full screen.

In addition, the library contains examples with extended dialogs which can be shown inside the wizard (eg. with multiple Input fields). You can even define your own layouts and use the examples there as a starting point for your design.

Manual Test 8.png

Manual Test Wizard – Start[Bearbeiten]

This action opens the wizard with individual values for position, size and behavior.

If this step is not present, the wizard will be opened with default values when the first manual test step is executed.

InputPin Parameter:

  • origin:
    defines the upper-left corner of of the wizard window in pixel coordinates on the screen. It should be given as a Point instance with x/y coordinates.
    Example: 20@40 means 20 pixel from the left, 40 pixel from top origin of the screen.
  • extend:
    defines the size of the window given as a Point instance with width/height values.
    Example: 600@400 means 600 pixel width and 400 pixel height.
  • wizardWindowMode:
    also defines the size of the window; it can be one of:
    • Normal
      the wizard window opens as specified by the above values (or defaults).
    • Minimize
      the wizard wind gets the specified or default values, but will be minimized initially.
    • fullScreen
      size and position are ignored; the window will take the full screen.
    • fullHeight
      use the specified width but the whole screen's height.
    • fullWidth
      use the specified height but the whole screen's width.
    • fullWidthMinimize
      minimized in full width. 
  • wizardDecorationMode:
    defines the window's decoration:
    • full
      as usual incl. minimize, maximize and close controls.
    • None
      no window controls.
    • Dialog
      with control to close.
  • allowInconclusive:
    to hide the button for "Inconclusive". If off, the tester can only give success/fail results.


Manual Test Wizard – Preset Values[Bearbeiten]

You can preset the layout before the wizard is opened the first time with this action. The meaning of the values is as described in the above “Manual Test Wizard – Start” action; these will be used when the first manual test step is executed.

Manual TestStep (API)[Bearbeiten]

You'll find a actions for different uses in the library. The 3 most heavily used are described below.

be reminded, that the presented texts are not required to be defined as constants. They might of course be generated by another step. For example, they could could be read from a database or a file, or be generated dynamically from text fragments.

Manual TestStep (Text)[Bearbeiten]

This action presents the texts as simple text.

The input pin "testStepDescription" expects the text. This plain text will be shown "as-is" in the step and the same in the wizard.

Manual Test 9.png

Manual TestStep (HTML)[Bearbeiten]

This action presents the text in the wizard as HTML formatted text.

The input "testStepDescription" expects an HTML string. You schuld have some experience with HTML, as the input must contain valid HTML syntax. However, you may also put the text into an attachment or a document element and use the document editor to enter HTML.

Notice: the embedded HTML viewer only supports a subset of the full HTML. For example, the style element is unsupported and tables do not have the full set of features).

The HTML-viewer in the wizard will present the text formatted.

Manual Test 10.png

Manual TestStep (SimpleWikiStyle)[Bearbeiten]

A text format commonly used in Wikis (so called "Wikistyle") can be used to format simple bullet lsits, headlines etc. without a need for HTML. For that, text in Wiki-style can be internally translated to HTML and presented as such.

The input "testStepDescription" expects the Text in simple WikiStyle syntax. The formatting rules are simple:

  • headlines
lines starting with "=" are presented as headline. Subchapters with '==' and '===':
= <h1>
== <h2>
=== <h3>
  • Horizontal Line
lines starting with or containing only a single "-"(minus) will be presneted as a horizontal separating line:

  • Lists
lines starting with * are presented as <li> ("Bullet List").
All other lines are unformatted. In addition, you may embed HTML tags (for example "
").

Here as well, the text is presented in the wizard in an HTML viewer.

Manual Test Example (Step by Step Tutorial)[Bearbeiten]

Step 1: Create a New Test Suite[Bearbeiten]

Open expecco and create a new test suite. You may of course also add manual test steps to any other suite. Then import the Manual Test Library (which is done automatically, as soon as you create manual test step or test case).

Manual Test 11.png

Step 2: Create a Test Case[Bearbeiten]

Next we have to create a test case item in the tree. The easiest way is via the toolbar menu, which creates an action with the TEST-CASE tag already attached, and it appears with its own icon in the tree.

Remark: the TEST-CASE tag is irrelevant, regarding the later execution; it is only a tag to affect its presentation, and makes it easier for you to spot test cases, test steps as oppose to the helper action. Other attributes of the appearance are specific colors or icons; you can define them as to your personal preferences to enhance the "readability" of your suite.

Manual Test 12.png

Note: please always give your test cases (and other items) reasonable and descriptive names. Usually, test cases should include a reference number, which refers to the corresponding items or chapter in your test- or requirements specification. In addition, append a short Description after the number so the humans also see what the individual steps do.

Example: "TC-4711 Login with valid Name".

Note: it has turned out to be a best practice to name test cases as "TC_xxx" and test steps which naht be reused in other scenarios as "TS_xxx". Other helper actions should have their function in the name.

Also please add a short documentation (Documentation-Tab). This helps to find actions and is also useful as these will be shown as tooltip ("fly by help") when you hover the mouse over a tree item or step in a diagram.

Alternative:
You can also mark actions afterwards as test case, by adding the corresponding tag later. Click first on "New Compound Action":

Manual Test 13.png

then navigate to the "Documentation" tab of the action and add the tag "TEST-CASE" to the list of tags at the bottom of the editor. Confirm by pressing "Accept" in the editor pane on the right side.

The icon shown in the project tree changes to "test case".

Manual Test 14.png

Thus, by adding the tag, any action may later be marked as being a test case or test step. You can also use the tree's popup-menu to add this tag to multiple actions at once.

Schritt 3: Create a Test Step for a Test Case[Bearbeiten]

navigate to the Diagramm tab of your test case. Initially, this is empty and has to be filled with individual steps. (sorry, in the picture below this is still named "New Action").

Teure are various options to add a step:

  • by drag & drop from the project tree,
  • by copy-paste from the project tree or from another diagram
  • by searching among existing actions (dialog), directly in the diagram (menu: "New Step...").
  • by creating a new action (dialog), directly in the diagram (menu: "New Action...").

In the following, new steps are added via the diagram's popup menu (right click). The menu item "New Action" opens a dialog which creates a new action and adds it as step into the diagram:

Manual Test 15.png

This opens a little input dialog. Enter the name of the new test step, honor the rules as above.

Note: If you intent to reuse the test step in multiple test cases later, the name should not contain step-numner or test case number. Use names like "TS - Open Application".

After confirming the dialog with ÖK", a cursor appears to let you specify the position of the new step in the diagram. Place the step and confirm the changes with "Accept" to save the editor's changed diagram in the tree.

The project tree now shows the new step below the test case.

Manual Test 16.png

Manual Test 17.png

Sorry, but actions created this way did not automatically get the right tag (in previous expecco versions). Therefore, you may have to navigate to the test step's definition in the tree, got to the documentation tab and add the "TEST-CASE" tag (or better yet, a "MANUAL-TEST-CASE" tag. As usual, click on "Accept" when done.

Step 4: Define What is to be Shown in the Wizard[Bearbeiten]

We now define what the wizard should show when the "TS - Open Application" Step is executed.

Up to now, we did not actually define whether this step will be a manual or automated test step. This is done now, by placing an action from the Manual Test Library into our action step.

Notice:

in theory, we could have added the manual test step right into the test case. However, this has the disadvantage that this might require more rework if we ever want to change this step to a fully automated one. We would have to change the test case and probably all places were the test step is also used.
By wrapping the actual wizard action into separate action, enables use to change the inside at any time later, without a need to change all uses of that step. We'll thus get more maintainability and cost reduction in the future.
It is a good practice to place any action which might be changed in the future into a separate compound action / even if it seems strange to have such simple diagrams which only cnsist of a single step.

Now choose, how the text should be presented to the test and how you want to enter it (i.e. Text/HTML/Wiki). For that, select the test step action in the tree, go to the diagram and choose "New Step..." from the popup menu.
(this time not "New Action", as we want to select an existing action instead of creating a new one):

Manual Test 18.png

Choose the action to be placed into the diagram as step. If the list is too long, filter by listong only individual libraries (in this case: "Manual Test Library") or by name (you might afilter for "HTML" or "Wiki" here) or by tag.

In this example, we choose: "Manual Test Template (WikiStyle)" (the action shown in the screenshot is somewhat obsolete - therefore different).

Manual Test 19.png

Confirm with "OK" to add the step to the diagram.

Manual Test 20.png

Doubleclick on the input pin named "testStepDescription" to open an editor for the text. Alternatively, choose "Freeze as String" from the pin's context menu to open the text editor.

Manual Test 21.png

Type the text which is to be shown in the wizard eg. according to WikiStyle rules.

Manual Test 22.png

Again confirm with "OK". The text is now attached to the input pin as so called "Freeze Value".

Manual Test 23.png


Finally, click on "Accept" to make your changes permanent in the project tree.

Step 5: Execute the Test Case in Isolation and Verify the Wizard's Presentation[Bearbeiten]

Select the test case in the project tree and navigate to the "Diagram'"tab.

Click the "Play"-button in the toolbar of the diagram editor. The wizard appears and shows your text:

Manual Test 24.png

Expecco will now wait for your input in the wizard. If you click on the green Play-con there, the wizard will advance to the next step and finish after the last test step (in this case, as this is an individual test being executed in isolation, it will finish right after the first step). The execution will be marked as "Passed" (assuming that you clicked on "green").

The wizard stays open and can be closed by hand.

Step 6: Creation of a Test Plan, Automatic Open/Close of the Wizard[Bearbeiten]

Previously, the test case was executed in isolation ("single action test execution"). This is good practice: test your actions in isolation so that you don't have to step through long sequences every time you encounter an error (or wrong texts).

Anmerkung:

Only then should you test longer sequences in which multiple cases run one after another. In this simple example, we do not expect any problems; however, if you create eg. a GUI Test, every test case should leave the tested system in a defined state, so that the next text can start with well known initial conditions. Typical errors are left-open dialogs, popups, wrong selections in menus, tabs or selection lists etc.
Therefore, make sure that such situations are prevented by running individual actions in isolation.

Creating a Test Plan[Bearbeiten]

Manual Test 25.png

Select your new test plan (or create one, if you haven't yet) in the project tree. Drag the test case action and drop it into the list of test case items on the right. As you may have noticed: a test case action can be placed into multiple test plans, and thus be easily reused. Therefore, the position of an action in the tree is not relevant for the test plan; you may place actions wherever and however you want to organize your tree, and group test cases and action independent from where they are used in test plans.

Manual Test 26.png

Manual Test 27.png

If you like, repeat the previous steps to create different test cases - possibly with multiple test steps.

Defining Pre- and Post Actions[Bearbeiten]

Finally, we want to define actions to automatically open the wizard with a well-defined size at the beginning. As a reminder: you don't need this, if you are fine with its default appearance. However here we want it to come up in a full screen window.

Also, after the test plan, the wizard should be automatically closed.

For a solution, there are 3 ways to go, although we recommend the last one below:

  • (1) Open and Close actions as part of the test plan (i.e. first and last actions in the list)
  • (2) Open as Pre-Action; Close as Post-Action (see corresponding fields below the list in the test plan editor)
  • (3) Define wrapper actiones; one to open the wizard, another to close the wizard and define these two as pre- and post actions.

(1) is not a good Solution, as it may leave the wizard window open, in case the test plan runs into an error or is aborted. That is not a problem if you sit in front of the screen to close it, but it may be bad if you automize the test run (eg. via Jenkins) and run it unattended. A wizard's window may in this case be kept alive on the screen (until expecco is terminated, or another test ist started).

(2) is much better: the post action will always be executed (as a cleanup), even if the test plan's execution is aborted for whatever reason.

(3) is even better as we now have a common place to add any additional cleanup actions in the future (eg. turn off power, remove temporary file, etc). And more can be added later without much maintenance effort.

Therefore, we will define two new actions, which will be called "PreAction" and "PostAction", and define them as such in the ist plan editor: Add the "Manual test Wizard -Start" action as step to the PreAction (by drag & drop or via "New Step...");
and also the "Manual test Wizard -Stop" action to the PostAction.

A configuration might look like:

Manual Test 28.png

Finally, define those two actions as Pre- and Post actions in the test plan. For that, go to the test plan editor and drag the corresponding action into the fields at the bottom of the editor. Manual Test 29.png

Click "Accept" to save this change.

Now when running the testplan, a wizard will be automatically opened at the beginning and closed at the end.

More Examples[Bearbeiten]

A sample test plan is found in the Manual Test Library itself. It contains multiple manual steps.

Also, find and open "ManualTestExample.ets" in the delivered samples folder. This contains a ready to start suite with an example similar to one above.



Copyright © 2014-2024 eXept Software AG