Selftest plugin

The plugin runs selftest scripts on actions of a library. Each action has associated to it a test script which specifies the inputs and expected behavior of the action. The script's syntax is very simple and scripts can easily be created from a requirements or certification document.

Its main use is as input for a certification suite of expecco's standard library.

Running Selftests

The plugin adds another tab named "Test Spec" to every action block. In this tab, a script is defined for that particular action. A run button to execute the script is shown in the editor's toolbar.

Also, a similar tab is added to the project's tree item, which executes all existing self tests of the whole library and provides the outcome of each test in a list view. Double click on a list item there, to navigate to the corresponding individual action's self test script.

Testscript Storage

The scripts are stored outside the library in a separate file named "<suite>.est" (".est" standing for "expecco-self-test"). The file is a zip-file, containing individual archive files per action block, where the archive-file is named after the block's functionalID. Script files can be loaded and saved via the plugin's main menu ("Plugins" - "SelfTest" menu item). Future versions may allow import/export of such scripts in other formats, e.g. from an excel requirements document.

Script Syntax

Each script consists of individual textual command lines. When executed, these script lines are interpreted by a test runner sequentially.

Lines beginning with ";" or "#" are comment lines. For "run" commands, text after a ";"-character is ignored (end-of-line comments). Notice that ";" and "#" are not recognized as end-of-line comment inside other commands' arguments unless occurring at the beginning or right after the command-keyword, because those are valid syntactic character for symbol- and array-literal constants and cascade expressions. Use Smalltalk comments ("...") inside those commands.

Lines ending in "\” are concatenated with the following line(s).

Currently, the following commands are recognized:

• "in <expr1> . <expr2> ..."
  specifies the input value(s) to be given to the action. Any number of expressions (incl. zero) can be given.
• "out" <expr1> . <expr2> ..."
  specifies the expected output value(s). Any number of expressions (incl. zero) can be given.
• "prerequisite" <st-expression>
  a prerequistite for the action, to be executed before the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). If the prerequisite expression raises an error, the test is reported as failed.
• "postrequisite" <st-expression>
  a postrequistite for the action, to be executed after the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). If the postrequisite expression raises an error, the test is reported as failed.
• "time" <st-block>
  similar to a postrequistite, the block is executed after the next test run, getting the execution time as argument. "st-block" must be a valid Smalltalk 1-arg block. If the check-block returns false, the test is reported as failed.
• "cleanup" <st-expression>
  a cleanup action to be executed after the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). Errors are ignored in cleanup actions.
• "timelimit" <st-timeDuration>
  sets a time limit for the next run. The test is marked as failing, if the execution time exceeds this limit. The default time limit is 5s.
• "skip" <message>
  skip the next test (up to and including the next "run" command) and output a message
• "skipall" <message>
  skip remaining tests and output a message
• "run" <expectedOutcome>"
  runs the test setup (given by previous in & out commands) and verifies the outcome. The outcome must empty (expect success) or one of "fail", "error", "inconclusive". The "run" command must always be the last in a group of commands for a single test run.

"in" Command

Specifies input values for the following run as Smalltalk expressions separated by a period ("."). For each input pin, one expression should be given. For unconnected inputs, individual expressions can be left empty (a space between periods is required).
For example:

• in 1 . 2 . 3
  Provides 3 integer input values to the first 3 input pins, with values "1", "2" and "3" respecitvely.
• in 1 . . . 'abc'
  Provides the integer "1" to the first input pin, no values to the second and third input pin, and the string "abc" to the fourth input pin (notice the spaces between the periods).
• in
  no input values are provided

Notice that the input value expressions can be arbitrary Smalltalk expressions - not limited to literal constants. For example:

• in (OrderedCollection new) . 5

will provide a fresh-created OrderedCollection to the first input pin.

"out" Command

Similar to the above "in" command, this defines the expected output values of the next test run. For pins where no output is expected, an empty expression must be given.
For example:

• out 10
  the first output pin is expected to deliver the integer "10"
• out 10 . 'foo'
  the first output pin is expected to deliver the integer "10", the second is expected to deliver the string "foo".
• out 10 . . 30
  the first and third output pins are expected to deliver the integers "10" and "30" repectively; the second output should not receive any value.
• out
  no output expected

Multiple Values at Output Pin

Some actions generate multiple values at one or more output pins. This behavior is specified by a preceeding "!" (exclamation mark) character, and a collection of expected pin values.
For example:

• out 10 . ! #(20 30 40) . 20

specifies that the first output pin should receive a single integer (10), the second pin should receive 3 values (20, 30 and 40), and the third output pin should receive a single integer (20).

Non-Constant Value at Output Pin

Some actions generate non-predictable values. For example, the "Generate UUID" action will generate a random uuid, which cannot be verified using a constant as in the previous examples. For this, use a 1-arg block as expected output, which checks the generated value.
For example:

• out 10 . [:val | val isUUID]

specifies that the second output shall be verified to be a uuid.

Arbitrary Value at Output Pin

To specify that an arbitrary value may appear at an output (for example a trigger token, for which the actual type is unspecified), use "*" as expression.
For example:

• out 10 . * . 30

specifies that the second output may have any value (but it MUST have been written by the tested action).

Identity Comparison

To check that an output value is identical (in contrast to "equal") to a particular input pin's value, use "= <nr>", where "<nr>" is the input pin nr.
For example:

• out 10 . =1

specifies that the second output pin's value should be identical to the first input pin's value (i.e. the same object).

"prerequisites" Command

Defines a Smalltalk expression to be evaluated BEFORE the next test run is executed. If a prerequisite action raises an exception, neither the test run, nor any postrequisites are executed. In contrast, to cleanup actions, which are evaluated in any case. A typical use is the creation of a temporary file (eg. when testing the "File [Exists]" action block).

"postrequisites" Command

Defines a Smalltalk expression to be evaluated AFTER the next test run is executed. Postrequisites are not executed if the prerequisite or test run failed or raised an exception. Typical are verification of side effects using "self assert:" expressions (eg. when testing for file existance in a test of the "File [Create]" action block).

"time" Command

Defines a Smalltalk block to be evaluated AFTER the next test run is executed. The block gets the execution time (as TimeDuration) as argument, and should return true, if the execution time is ok, false if not. Timechecks are not executed if the prerequisite, test run or postrequisite failed or raised an exception.

"timelimit" Command

Defines a an execution time limit for the next run. The default limit is 5 seconds. The expression must evaluate to a TimeDuration (i.e. you can say "0.1 seconds" or "100 milliSeconds")

"skip" Command

Skips the next test in the spec (i.e. everything up-to and including the next "run"-command is ignored), and outputs a message. Useful to temporarily skip individual tests. The rest of the line (everything up to the end) is presented as message in the log.

"skipall" Command

Skips all of the remaining tests and outputs a message. The test spec is not counted as missing or erroneous. Useful to mark scripts for actions which cannot be easily tested automatically, for example tests which open up user interface dialogs, or which have other effects which cannot be easily verified. Such actions usually need manual tests vor verification. The rest of the line (everything up to the end) is presented as message in the log.

"cleanup" Command

Defines a Smalltalk expression to be evaluated AFTER the next test run is executed. Cleanup actions are executed even if the pre-, post or test run fails or raises an exception. A typical use is the deletion of a temporary file (eg. when testing the "File [exists]" action block.

"run" Command

Executes the test using the previously defined input values ("in" command), and verifies the expected outputs against the previously defined output values ("out" command). Also, previously defined prerequisite, postrequisites and cleanup actions are evaluated.

The expected success/fail/error state is given by a keyword after the "run" command. This can be one of:

• <empty>
  expect success
• "norun"
  expect a "no executable step, due to missing inputs"-error
• "fail"
  expect failure
• "error"
  expect an error
• "inconclusive"
  expect an inconclusive outcome

Syntax Summary (BNF)

TestScript ::= <empty>
               | <TestScript> <SingleTest>

SingleTest ::= [ <SetupCommands> ] <RunCmd>

SetupCommands ::= <empty>
               | <SetupCommands> <SetupCmd> 

SetupCmd ::= <PreReqCmd> | <PostReqCmd> | <CleanupCmd> | <TimeCmd> | <TimeLimitCmd> | <skipCmd> | <skipAllCmd> | <InCmd> | <OutCmd>

PreReqCmd ::= "prerequisite" <ST-expression>

PostReqCmd ::= "postrequisite" <ST-expression>

CleanupCmd ::= "cleanp" <ST-expression>

TimeCmd ::= "time" <ST-timeCheckBlock>

TimeLimitCmd ::= "timelimit" <ST-TimeDuration-expression>

SkipCmd ::= "skip" message

SkipAllCmd ::= "skipall" message

InCmd ::= "in" <InPinSpec>

InPinSpec ::= <empty>
              | <SingleInPinSpec> [ "." <SingleInPinSpec ]...

SingleInPinSpec ::= <ST-expression>
                        
OutCmd ::= "out" <OutPinSpec>

OutPinSpec ::= <empty>
               | <SingleOutPinSpec> [ "." <SingleOutPinSpec ]...

SingleOutPinSpec ::= <SingleValueOutSpec> | <MultiValueOutSpec>

SingleValueOutSpec ::= <empty>
                       | "*"
                       |  <ST-expression>
                       | <ST-valueCheckBlock>
                       | "=" <inPinNr>

MultiValueOutSpec ::= "!" "{" <SingleValueOutSpec>... "}"
                       | "!" "#(" <SingleValueOutSpec>... ")"

RunCmd ::= "run" [ norun | error | fail | inconclusive ]

Sample Scripts

The following sample script could used to test the "Arith [ Sum ]" action:

in        ; no inputs
out       ; no outputs expected
run norun ; expect an "not-executable" error (no inputs given)
 
in 1 . 2  ; two inputs given
out 3     ; expected output
run       ; no error expected

The next sample script could used for the "File [ Exists ]" action:

prerequisite '/tmp/testfile' asFilename contents:'hello'
cleanup      '/tmp/testfile' asFilename delete
in           '/tmp/testfile'
out       ; no output
run       ; expect no error
 
cleanup '/tmp/testfile' asFilename delete
in      '/tmp/testfile'  
out       ; no output
run fail  ; expected outcome

The following tests the "File [ Create ]" action:

prerequisite  '/tmp/testfile' asFilename delete
postrequisite assert:('/tmp/testfile' asFilename exists)
cleanup       '/tmp/testfile' asFilename delete
in            '/tmp/testfile'
out       ; no output
run       ; expect no error

The "Trigger" action, which writes its output multiple times, is tested with:

in        3
out       ! #( * * * )
run

---

Copyright © 2014-2024 eXept Software AG