- 1 Glossary
- 1.1 Black Box Test
- 1.2 Compound Block
- 1.3 Compound Step
- 1.4 Elementary Block
- 1.5 Elementary Step
- 1.6 ".elf" File
- 1.7 ".ets" File
- 1.8 Gray Box Test
- 1.9 Groovy
- 1.10 Groovy Elementary Block
- 1.12 Shell/Batch Script Elementary Block
- 1.13 Smalltalk Elementary Block
- 1.14 SUT (System Under Test)
- 1.15 Syntactic Sugar
- 1.16 Verdict
- 1.17 VersionID
- 1.18 White Box Test
- 2 Abbreviations
Please keep this list alphabetically sorted!
Black Box Test
In a "black box test", the test-machine and test-application are completely separated from the System Under Test (SUT). No knowledge about the internals (data structures, object model etc.) of the SUT is known when testing. The test only operates against official external interfaces, such as input/output, data and configuration files, command line interfaces, GUI interfaces or communication protocols.
The opposite being a "White Box Test", which executes inside the SUT and has full access to all internal details (i.e. functions, data objects etc.).
Expecco supports both, "Black Box" by various libraries which provide interfaces to commonly used external interfaces, and "White Box" via code injection mechanisms, such as "Groovy" or "VisualBasic".
Mixed mode testing is called "Grey Box Testing".
An action block which is defined by an activity diagram. Its execution semantics are defined by a network of interconnected steps. These themselfes can be compound- or elementary steps. See Compound Block in the Tree Elements documentation.
A step inside an activity diagram, whose action definition is a compound block. When triggered, such a step will execute another activity diagram. See Compound Block in the Tree Elements documentation.
An non-compound action block. I.e. any action which is not defined by an activity diagram, but either builtin (well known to expecco), a call to an external function or application, or defined programmatically as a piece of program code in one of the supported programming languages.
A step inside an activity diagram, whose action definition is an elementary block. When triggered, such a step will execute either a builtin function, an existing function inside a DLL, another program or a piece of code written in one of the programming languages supported by expecco.
"elf" stands for "Expecco Log File" and is the native format in which result files are written. "elf" files contain all of the execution, trace and data-flow information and the original suite as executed. "elf"-files can be reloaded at any time into the expecco UI, values of that run be inspected and it is even possible to reexecute the original test. Also, any of the other reports can be regenerated, possibly with different report options (i.e. detail). Technically, "elf" files are zip-containers containing traces, logs and the original test suite in XML-format, and any attachments (screen shots, measurement data etc.) in whatever format they were written (images usually as PNG). They can (in theory) be read and processed by third party tools.
"ets" stands for "Expecco Test Suite". These are technically zip-containers, containing the test suite in XML-format and any attachments in whatever format they where imported. In addition, an index (manifest) is present, which allows for fast access to certain information without a need to read the whole zip archive. Also, signatures ensure that the manifest and remaining contents are in sync. Therefore, ".ets" files should not be manipulated manually, as the consistency of the suite may be harmed.
Gray Box Test
Some knowledge about internal interfaces of the System Under Test are known. For example, data structures or function entries inside the SUT are exploited and used. The categorization of a test as white- or gray box test is vague - there is often no clear line of separation between them. (see "Grey Box Testing" on Wikipedia).
A language interpreter which runs on top of a Java Virtual Machine (JVM), with a language very similar and almost compatible to Java. Used in expecco to call into Java interfaces or define required callback and listener classes, in case the SUT or the interface to the SUT require those.
Groovy code is compiled to bytecode inside the JVM, which is further Just-in-Time compiled to fast machine code within the target JVM. Thus, it executes at full Java speed.
Expecco allows for Groovy code to be executed both on the local machine (the one on which expecco itself is running) and on remote machines which are reachable from the local machine via a TCP/IP connection. Multiple of these connections are possible and handled simultaneously in parallel, making it possible to interact with multiple hosts in a distributed system or when testing both ends of a communication protocol. See "Groovy Elementary Blocks" in the Expecco API Document and Elementary Block in the Tree Elements documentation.
Groovy Elementary Block
An action block which is defined by an execute function which is written in the Groovy language. The block's code will be executed by a Groovy interpreter on a Java Virtual Machine (JVM). The JVM may run either on the local or on a remote machine. Expecco uses code injection techniques to implant the code into the SUT dynamically at run time. Except for sealed systems, it is normally not required to specially instrument or otherwise recompile the code in the SUT. See "Groovy Elementary Blocks" in the Expecco API Document and Elementary Block in the Tree Elements documentation.
Shell/Batch Script Elementary Block
An action block which is defined by a shell (or batch) script which is written in a shell command-line-interpreter language. The block's code will be executed outside expecco, by a script interpreter. See "ShellScript Elementary Blocks" in the expecco elementary block documentation.
Smalltalk Elementary Block
An action block which is defined by an execute function which is written in the Smalltalk language. The block's code will be executed inside expecco itself. See "Smalltalk Elementary Blocks" in the Expecco API Documentation.
SUT (System Under Test)
The tested system.
Expecco interacts with this either in a pure black box (only talking to external interfaces), or gray to white box fashion (some or a lot of knowledge about internals of the SUT are known).
Technically, black box tests interact with the SUT via (more or less official) external interfaces, such as messages, documents, communication interfaces, GUI interaction etc. White box tests can access data structures and interfaces internal to the SUT.
In expecco, White Box Tests are done by implanting code into the SUT via code injection techniques. Depending on the type of technology and environment used, this injection (or instrumentation) may be possibly dynamically (Java or DotNet) or has to be done statically by linking expecco-support libraries to the program (C, C++)
A feature of (typically) a programming language, which does not give you more semantic features, but instead exists for the convenience of the user. Typically "syntactic sugar" designates a feature which is easier to use or shorter to write, but which could also be defined in terms of other features of the language - albeit possibly more clumsy. In expecco, a number of syntactic sugar constructs are available, which could also be described in terms of regular action blocks. For example, the "attachment step", "shell step" and even "groovy" steps could be also defined in terms of regular elementary blocks which read a file, start a shell or talk to a JVM respectively.
The execution's summary-outcome-status from running a test plan, test case or action. In addition to detail information (traces, data, execution time, log-messages), every execution in expecco will always return one of 4 values as a summary status:
if all went well, and the execution did not encounter any problems
the execution detected an error in the System Under Test (SUT). For example, it measured a wrong value, detected invalid behavior of the SUT, invalid protocol behavior etc.
the test itself ran into some error. This means that during the test's execution, an error occurred inside expecco or one of the user-defined action blocks. For example, if a file containing test-data could not be found, if a division by zero occurred etc.
the test had no chance to perform its testing operations. For example, if a machine could not be reached in the network, if a resource (operator, measurement device, test-device etc.) was not available, etc.
The distinction is very useful as it should be kept in mind, that a test-manager/tester has to react differently depending on the reason for a test not resulting in a PASS:
- for a FAIL, he has to inform the developer of the SUT, to fix the problem in the product.
- for an ERROR, he has to inform the test-developer/analyst, to fix the problem in the test-case
- for INCONCLUSIVE, he has to check for the availability of required devices and possibly inform the network administrator or test-lab-manager. After that, he can rerun the test without a need to contact either the product-development, nor the test-development team.
Notice, that many test systems/frameworks only provide PASS/NOT PASS information as final verdict. If expecco is to be integrated into such a framework, FAIL, ERROR and INCONCLUSIVE will all be treated as NON-PASS (eg. when running under the control of Jenkins, QC, Polarion etc.)
Every item within expecco (action, type, testcase, testplan, the suite itself etc.) has an associated unique identifier, which is guaranteed to be world-wide unique (a so called "UUID"). This guarantees, that these objects can be uniquely identified, even if renamed, or if the same name is used for an action, type etc. in two different test suites. Thus, two suites can always be merged, and no name conflicts can happen. expecco will always refer to any object by its ID, never by its name. The versionID is updated, with every change made to an object. Thus it is guaranteed, that no two different objects have the same ID. And vice versa, that the equality (actually identity) can always be checked via the ID.
White Box Test
Designates a test which executes within the System Under Test and uses or refers to (but also depends on) the internals of the System Under Test. The test directly creates or manipulates data objects inside the SUT and/or calls internal functions. Most unit tests are typically implemented as white box tests (but not required to be so).
White Box Tests have both advantages and disadvantages.
- + because they know the internals, more of internal details can be tested, and more of existing internal utilities can be used as utility in the test. For example, data definitions, data structures, functions and interfaces etc. can be accessed from the test system. Depending on the type of test and the available frameworks which can be "white-boxed", this may save a lot of test-development time, because thise interfaces need not be reimplemented in the test system.
- - because they know the internals, they are also more reliant on them, and may also suffer from bugs in the internal frameworks. For one, if data structures and internal interfaces change, the white box test usually has to change with it. Whereas external interfaces usually remain more stable - especially if they are based on communication or data format standards. The other downside is that if internal interfaces of the SUT are heavily used, bugs in them are also injected into the test system. This may make some of the bugs invisible and undetectable to the test system. For example, if a communication data structure (say, a message or document format) is always accessed via a white-boxed SUT interface, but never actually tested physically, any bug in that framework will go unnoticed to both the SUT and to the test system. Both the SUT and the test system may see perfect data, while the data sent to the outside world may still be completely bogus.
In a real world test system, you would usually use a combination of whitebox and blackbox tests. The level of insider-knowledge of the whitebox test depends on the type of SUT, the risk analysis and trust in the frameworks being used.
See also: Black Box Test
(please keep this list alphabetically sorted)
Common used abbreviations in this document are:
- ALM - Application Lifecycle Management
- CB - Compound Block
- DCE - Distributed Computing Environment (an RPC mechanism)
- DLL - Dynamic Link Library (a synonym for SO - shared object)
- EB - Elementary Block
- ".elf" - Expecco Log File (file suffix)
- ".ets" - Expecco Test Suite (file suffix)
- GUI - Graphical User Interface
- QM - Quality Management
- RPC - Remote Procedure Call
- SO - Shared Object (a synonym for DLL)
- SUT - System Under Test
- UI - User Interface (often actually meaning: Graphical-UI)