Selftest plugin

The plugin runs selftest scripts on actions of a library. Each action has associated to it a test script which specifies the inputs and expected behavior of the action. The script's syntax is very simple and scripts can easily be created from a requirements or certification document.

Its main use is as input for a certification suite of expecco's standard library.

Running Selftests

The plugin adds another tab to every action block, in which a script is defined for that particular action. A run button to execute the script is shown in the editor's toolbar.

Also, a tab is added to the project's tree item, which executes all existing self tests of the whole library and provides the outcome of each test in a list view. Double click on a list item there, to navigate to the corresponding individual action's self test script.

Testscript Storage

The scripts are stored outside the library in a separate file named "<suite>.est" (".est" standing for "expecco-self-test"). The file is a zip-file, containing individual archive files per action block, where the archive-file is named after the block's functionalID. Script files can be loaded and saved via the plugin's main menu ("Plugins" - "SelfTest" menu item).

Script Syntax

Each script consists of individual textual command lines. When executed, these script lines are interpreted by a test runner sequentially.

Lines beginning with ";" or "#" are comment lines. For "in", "out" and "run" commands, text after a ";"-character is ignored (end-of-line comments). Notice that "#" is not recognized as end-of-line comment unless occurring at the beginning, because "#" is a valid syntactic cahracter for symbol- and array-literal constants. And also notice that the ";"-character is also not recognized as comment character with "prerequisite" and "cleanup" commands, because they are valid in Smalltalk expressions.

Currently, the following commands are recognized:

• "in <expr1> . <expr2> ..."
  specifies the input value(s) to be given to the action. Any number of expressions (incl. zero) can be given.
• "out" <expr1> . <expr2> ..."
  specifies the expected output value(s). Any number of expressions (incl. zero) can be given.
• "prerequisite" <st-expression>
  a prerequistite for the action, to be executed before the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). If the prerequisite expression raises an error, the test is reported as failed.
• "postrequisite" <st-expression>
  a postrequistite for the action, to be executed after the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). If the postrequisite expression raises an error, the test is reported as failed.
• "cleanup" <st-expression>
  a cleanup action to be executed after the next test run. "st-expression" must be a valid Smalltalk expression (which is given to eval). Errors are ignored in cleanup actions.
• "run" <expectedOutcome>"
  runs the test setup (given by previous in & out commands) and verifies the outcome. The outcome must empty (expect success) or one of "fail", "error", "inconclusive". The "run" command must always be the last in a group of commands for a single test run.

"in" Command

Specifies input values for the following run as Smalltalk expressions separated by a period ("."). For unconnected inputs, individual expressions can be left empty.
For example:

• in 1 . 2 . 3
  Provides 3 integer input values to the first 3 input pins, with values "1", "2" and "3" respecitvely.
• in 1 . . 'abc'
  Provides the integer "1" to the first input pin, no values to the second and third input pin, and the string "abc" to the fourth input pin.
• in
  no input values are provided

"out" Command

Similar to the above "in" command, this defines the expected output values of the next test run. For pins where no output is expected, an empty expression must be given.
For example:

• out 10
  the first output pin is expected to deliver the integer "10"
• out 10 . 'foo'
  the first output pin is expected to deliver the integer "10", the second is expected to deliver the string "foo".
• out 10 . . 30
  the first and third output pins are expected to deliver the integers "10" and "30" repectively; the second output should not receive any value.

"prerequisites" Command

Defines a Smalltalk expression to be evaluated BEFORE the next test run is executed. A typical use is the creation of a temporary file (eg. when testing the "File [Exists]" action block).

"postrequisites" Command

Defines a Smalltalk expression to be evaluated AFTER the next test run is executed. Typically are verification of side effects using "self assert:" expressions (eg. when testing for file existance in a test of the "File [Create]" action block).

"cleanup" Command

Defines a Smalltalk expression to be evaluated AFTER the next test run is executed. Cleanup actions are executed even if the pre-, post or test run fails or raises an exception. A typical use is the deletion of a temporary file (eg. when testing the "File [exists]" action block.

"run" Command

Executes the test using the previously defined input values ("in" command), and verifies the expected outputs against the previously defined output values ("out" command). Also, previously defined prerequisite, postrequisites and cleanup actions are evaluated.

The expected success/fail/error state is given by a keyword after the "run" command. This can be one of:

• <empty>
  expect success
• "norun"
  expect a "no executable step, due to missing inputs"-error
• "fail"
  expect failure
• "error"
  expect an error
• "inconclusive"
  expect an inconclusive outcome

Sample Scripts

The following sample script could used with the "Arith [ Sum ]" action:

in        ; no inputs
out       ; no outputs expected
run error ; expect an error (no inputs given)
 
in 1 . 2  ; two inputs given
out 3     ; expected output
run       ; no error expected

The next sample script could used with the "File [ Exists ]" action:

prerequisite '/tmp/testfile' asFilename contents:'hello'
cleanup      '/tmp/testfile' asFilename delete
in           '/tmp/testfile'
out       ; no output
run       ; expect no error
 
cleanup '/tmp/testfile' asFilename delete
in      '/tmp/testfile'  
out       ; no output
run fail  ; expected outcome

The following could used with the "File [ Create ]" action:

prerequisite  '/tmp/testfile' asFilename delete
postrequisite assert:('/tmp/testfile' asFilename exists)
cleanup       '/tmp/testfile' asFilename delete
in            '/tmp/testfile'
out       ; no output
run       ; expect no error

---

Copyright © 2014-2024 eXept Software AG