Best Practices

Customers have different requirements and preferences in how suites are organized. Thus this document should be taken as a rough guideline, not as dogma.

A general rule of thumb is that test-development is - after all - very similar to "normal programming" and that patterns and anti-patterns of "normal" program development also apply to the development of test suites.

Add Documentation

This applies to any programming work - actually to anything you do. Add a readme which tells others where to start, what is needed as prerequisite etc.

Give your Items Descriptive Names

A common advice, which applies to any programming. Give your actions, pins and data attachments names which help others and yourself to understand the thing. Don't be lazy using short names: you will write things only once, but you and others will read it many times. If the accumulated time to figure out what a variable named "x" does is longer than the time to write "someDescriptiveName", then you have a net saving.

Allow for Cross-Referencing Items to Other Specs

Usually, test cases are written against either a test-spec or a requirements spec, and items there follow a more-or-less strict naming pattern. Although the concrete details vary among customers and tools they use, most use sequence or chapter numbers there. For example, requirements might be named "RQ-X" and test cases "TC-y" in those documents.

Make sure, that you enable those to be cross referenced against test cases inside expecco. This can be in the individual names and/or by adding tags to the action items.

As to our experience, using plain RQ-names as test case names makes the suite hard to read for test developers (i.e. "remember which test case was doing ...") and using plain English descriptive names makes it hard to automate interaction with QM (Quality Management) systems.

Therefore, the best approach is to combine them: concatenate the RQ-name with some English description as the test case name, and add the plain RQ-name as a tag to it.

The tag will be preserved in the saved ets-file (which is actually XML) and can therefore be extracted, associated or linked to corresponding artifacts in the QM system (by s custom XSLT script, or by expecco actions).

There are reflection facilities in expecco, which can be used to search/extract/generate such tags and/or tagged values. For example, to automatically send an XML-request to the QM system, when a particular test case fails, or to upload attachments to the corresponding item there, as post-execute action, or to generate CSV lists which can be imported back to other systems.

In summary: a good naming scheme might be: "RQ-001: test service startup", "RQ-002: data import/output test", "RQ-002.1: data import" etc. From that, the RQ id can be easily extracted for automation, and it is also human readable.

Of course, the details depend on what you already have and how your QM system organizes things (if you use expeccoALM, the mapping is done there, and it uses a test-functionUUID to requirement-UUID mapping already; so you don't have to care for this specially).

When test specs are imported (for example, from enterprise architect test specs), expecco will keep any ID (actually: any tagged value) and preserve it. For other import/export formats, the details may vary and may need a custom action block to extract/generate.

Add Sample Usages

Use the Test/Demo network page (of action blocks) to provide one or more sample uses for your actions. Beside being a great "executable documentation" for others, they are also useful as "unit tests" for actions you create.
Also, provide demos (usually for more complex scenarios) in a "playground" folder.

Make Actions (both Diagram- and Coded Actions) Short

Split bigger actions into smaller parts, where each performs a particular job. This makes them both easier to understand, easier to test in isolation and especially easier to reuse in other situations. Make it perform one operation only.

Big workflow diagrams tend to become hard to understand. Try to limit the number of steps and split complex actions in to separate smaller ones.

Don't be afraid of Small Scripted Actions

Although many of our customers are non-programmers and one of expecco's goals was to make it easier for non programmers, it is sometimes advisable to write little pieces of code (of course: if possible).

For example, some text-extraction or mass data-reduction algorithm might be much easier to write in a few lines of code, than using action blocks from the library. For example, communication package or bulk measurement data analysis, complex substring extraction from text (if regex and other existing action blocks are not sufficient) etc. can usually be written in a few lines of code.

Expecco gives you the choice among many programming languages: JavaScript, Python, Java, Smalltalk, VisualBasic and even C/C++. However, make sure to make this action perform one task well, and do not wrap complete and complex tests into it, in order to remain usable for non programmers in the future (although you could, technically).

See what is Already Provided

The standard library provides many action blocks, especially for string processing, searching in text/data, file operations etc. Use the string search features or the very convenient "Search by Result" tools. The later lets you give input and expected output, and expecco will find API functions and code blocks which compute that result (to be honest: it tries to find them).

Try to make Similar Action Sequences reusable by parametrizing them

A typical example is the interaction with graphical user interfaces: instead of recording every menu-selection or form-data-fill individually, record it once and add a parameter to which menu item or which input field is to be filled. For this, add an input pin to the recording and use that as substring in a locator pin (eg. adding a pin named "itemName" and using a locator such as "$(ui.menu)/menuItem[@name="$(itemName)]"). Notice, that both the path to the menu ("ui.menu") and the item's label ("itemName") are dynamically expanded in the locator, where "ui.menu" is typically coming from a global variable (object repository) and "itemName" from a corresponding input pin.

Test Plans, Test Cases and Utility Actions

Tree Organization of Test Plans and Cases

In the tree, you will end up with one or multiple test plan items, and usually multiple test case action blocks. Be reminded that in expecco, the test plan consists of a list of test case items, each implemented by an action block. Thus, a test plan lists the individual test case actions; it does not include or contain or hide them as private entities.

The location of the action block is irrelevant for the test-plan: the action can be located anywhere in the suite or even imported from another library.

This is in contrast to other such tools, where the tree-structure defines the test case items of a test plan. The decision to do it this way in expecco is by purpose: it allows to place a test-case action block into multiple test plan, possibly doing the same, possibly being parametrized by the test plan list. Especially the fact that it can come from a library gives added flexibility: you can put test case implementation into one library and create an ets file consisting of test plans only. Actually, even this can be done automatically by expecco via command line options, where test plans are constructed on the fly, driven by command line, scripts or CSV file arguments.

To summarize, in the tree, you can

• either organize test case actions under the corresponding test plan
• put all test actions into one or multiple separate folders inside your suite (possibly split by area, risk or whatever you prefer)
• put all test actions into a separate library, import that one, and let testers arrange test-sets as they like, from the given set of test actions

Test Case Action vs. Utility Action

First of all: for expecco, there is actually no semantic (i.e. operational) difference between a test-case action and any other action. Every action can be dropped into a test plan, and every test case action can also be placed as a step into another workflow diagram.

Within a diagram, any step which generates a non-successful verdict will make the outer action become non-successful (unless the exception is handled or ignored), and any error is signaled up along the caller chain up to the test item of the test plan in which it runs. Which then marks the test case item there as erroneous.

Typically what makes a test case action a "test case" is that it contains assertions and other data/value comparisons, whereas typical utility actions simply perform some operation (being data processing, protocol handling, UI interactions or whatever).

That sayd, you can give any action a "TEST-CASE" tag (via the popup menu), and thereby giving it a special icon presentation in the tree. This can improve the readability and tells other about the intended use of an action block.

Do not Copy-Paste Suites

A common anti-pattern with new users is to take any existing suite (ets-file), start to modify it and saving it under another name. And continue to work with the copy. This is ok, if the original is never to be used or modified again, for example if you take one of the examples from the expecco installation as a template to start with.

However, it may lead to a maintenance nightmare, if both suites have to be maintained in the future - especially, if new features, new tests or new interfaces are to be added to both.

Another problem with this anti-pattern is that it will later be hard to import both suites into one common "master suite", because if saved this way, both suites will have the same functional UUID, and expecco will assume that they are different versions of the same project (be reminded, that expecco always uses those unique IDs to determine whether two suite files have different functionality or are actually only two versions of the same. And that expecco will not allow that two versions of the same library are imported at the same time (on the same import level).
Don't panic: expecco has menu functions and tools to repair such a situation, but it costs you extra work time, which can be avoided.
Also, expecco will warn you when saving suites under another name, and ask if that is to be another suite or simply a new version of the old one.

Dealing with Versions, Releases and Variants

Usually, test suites (ets files) are versioned in a repository along the versions of the product (i.e. checked into git, svn, mercurial, cvs or other source code repository).

In the past, it has shown to be good practice to keep older versions inside the suite. And place them into separate folders and master test plans, instead of overwriting them between versions.

This has a few advantages and one disadvantage:

• + it makes it much easier to compare versions (inside a single suite vs. among multiple suites). Although expecco provides tools and UIs to compare multiple suite files, it is of course much more convenient, to have the versions inside one suite, when comparing.

• + by structuring test plans into multiple master plans (one per version/variant) and feature-specific sub test plans, most test cases can be reused (remember: you can drag a test action block into multiple test plans), as usually there is no need to modify older test cases when new features of the product are to be tested. In almost every setup, new features come with new versions and these need additional tests, making the test set bigger over time, without usually making older tests obsolete (although that is possible, and can be done in expecco as well).

• + there is no need to duplicate test plans for new versions with added features: simply create a new plan (name it "V2") and drag the "V1" test plan into it. Thus every new testplan will contain all of the previous tests in addition to the delta of tests. You can then run select the test plan to run either dynamically (create a block, which determines the app's version and runs one of the test plans) or manually via the command line ("expecco --testplan V2") or from expeccoALM by checking the corresponding testplan's checkbox in the test run configurator.

• + in addition to that, individual test- and utility actions can still come from separate libraries. For example, split by sub component or grouped by use cases.

• - the disadvantage is that the suite might become bigger than actually required, as it may contain actions which are no longer needed or which are different in new versions.

For the above reasons, the suite which is generated as initial sample suite (when starting expecco without suite argument) will contain such a structure. Of course, the generated actions there are all dummy actions (which small output some info-message), but the structure can be used as a template for your own suites.

---

Copyright © 2014-2024 eXept Software AG