WindowsAutomation Reference 2.0/en
This is the documentation for the Windows Automation 2.0 plugin.
It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.
- 1 Features
- 2 Known issues in the current Release 19.1.0
- 3 Installation and Connection
- 4 Settings
- 5 GUI Browser
- 6 Library
- 6.1 Connecting to an Application
- 6.2 Accessing GUI Elements
- 6.3 Interacting with GUI Elements
- 6.4 Automating Keyboard and Mouse
- 6.5 Recording and Taking Screenshots
- 7 Frequently Asked Questions
- 7.1 What is the difference between Click/Invoke/Select/Toggle?
- 7.2 My element supports a certain pattern but blocks of that pattern don't work?
- 7.3 I have designed custom elements, how can I automate them?
- 7.4 I have extended my GUI elements to have custom properties, how can I query them?
- 7.5 Hotkey "Get hovered element" selects the wrong element in the tree
- Automated GUI testing of WPF, WinForms, and Win32 applications.
- Contains an expecco block library and tools which help to model tests and inspect the GUI of the application.
- Parallel remote test-execution on multiple systems.
- Performs tests against already built executables. No source code changes / recompilation of the application is required.
- Access to GUI components using XPath locators.
- Access objects of the application live at execution time.
- Full access to keyboard and mouse of the target system.
- Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...
Known issues in the current Release 19.1.0
- The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing Pre-19.1.0 test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
- Instant follow mouse is very expensive in Windows and is therefore disabled. Use HotKeys to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey-service has to be started. Take care that the keys have not already been grabbed by another application. If that is the case, a warning will be shown, and you should either make sure that expecco is started before the other application, or that you use other hotkey combinations (in either expecco or the application).
Installation and Connection
The WindowsAutomation testing plugin uses the expecoo ".NET-Bridge" which consists of two parts: A plugin for expecco, and a server executable using the .NET framework running on the remote computer to connect to.
On the machine running the system under test:
- .NET Framework 4.6.2  or higher.
- One of the following operation systems:
- - Windows 7 SP1
- - Windows 8
- - Windows 8.1
- - Windows 10
- - Windows Server 2008 R2 SP1
- - Windows Server 2012
- - Windows Server 2012 R2
- - Windows Server 2016
- - Windows Server, version 1709
The plugin consists of the following parts:
- The GUI Browser, used to interactively explore your application.
- The Windows Automation Library, which provides blocks that you can use in your test cases.
- The .NET-Bridge server, which provides an interface between the application under test and your tests running in expecco.
Installing and Configuring the Plugin in Expecco
The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "
plugin" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You might need to restart your expecco session.
Preparing a local system for test execution
If you want to automate an application running on the same host as expecco, there is no need to set up anything. Everything should work out of the box.
Try to connect to an application via the GUIBrowser's WindowsAutomation-connect dialog:
Preparing a remote system for test execuction
Please make sure, that remote access is only possible from inside a secure local network. Under no circumstances should you grant access to the DotNetBridgeServer from untrusted hosts.
DotNet Bridge Server
The remote system has to run the ".Net-Bridge server" which in turn either actively connects to expecco or passively waits for an incoming connection from expecco.
For this, copy "
DotNetBridgeServer.exe" (in the "
packages/exept/bridgeFramework/dotNetBridge/dotNetBin" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.
To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments ("DotNetBridgeServer.exe --help" outputs a list of valid options):
|-s||Yes||"server mode". Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode, where it actively connects to the partner).|
|-k||Yes|| "keepalive". Tells the client to keep the connection open after a connection has been terminated.|
This saves you the hassle to restart it every time a test finishes.
|-a <ip/hostname>||Yes||"accepted hosts". The IP-address or hostname from which it will accept connections (in server mode) or to which it connects. To ensure that in slave mode no other machine connects to the server, set this to the IP of the machine your expecco is running on. To allow any machine (i.e. to have it listen on any IP-address), set it to 0.0.0.0.|
|-p <int>||No||"port". The port the server should listen on (in server mode) or to which is should connect (default is 34318).|
|-L <string>||No|| "assembly". Load a .NET assembly on startup. Omit the .dll extension (so use "WindowsAutomation" but not "WindowsAutomation.dll"). You can repeat this option to load multiple assemblies.
Note: this is not normally needed. expecco loads the required assemblies automatically into the bridge.
So you may execute on the remote host:
DotNetBridgeServer.exe -s -k -a 0.0.0.0
There are currently no configurable settings.
The GUI browser allows you to explore a running application. You can browse the application, inspect element properties and execute actions. For a more detailed description also see Gui Browser.
This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.
Connecting to an Application
To automate an application, you first have to connect to it. You can do this with the blocks from the "Connecting" group.
You can connect to an application by its executable name, or by its process id. If you use the executable name, expecco will attach to the main process of the specified executable. Be careful as this is not necessarily the process drawing the GUI! Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process.
All connection blocks also require you to specify a name by which the connection should be known in future. This is needed when you automate multiple applications at the same time and need to change connection contexts during test execution. Please note that the application to connect to has to be started before the corresponding connection block is executed. You can change in context of which connection a block is executed with the [Set Active Connection] block. This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.
Establishing a Local Connection
If you want to automate an application running on the same host as expecco, there is no need to set up anything. Use the [Create Local Connection] with the above mentioned parameters.
Establishing a Remote Connection
Make sure that the DotNet-Bridge server (DotNetBridgeServer.exe) is running on the remote system, as explained in the chapter "Preparing a remote system for test execuction" When the bridge is running, you can connect to it with the [Create Remote Connection] block using the hostname and IP-Address you specified when you started the bridge.
Notice, that the DotNet-Bridge server only accepts local connections by default. This is done for security reasons. Use the "-a x.x.x.x" command line argument to specify, which hosts are allowed to connect to the Bridge server. With "-a 0.0.0.0", any host is allowed. Do NOT use this, if the remote host is reachable from untrusted hosts.
Closing a Connection
Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources. One simple way to do this is to setup your connections in the Before Execution (pre-action) part of the test suite or test case, and put the [Close All Connections] block in the After exeuction (post-action) part.
Accessing GUI Elements
All Windows applications are built as a tree of GUI elements. Some elements act as container of other element - their children. Tables, for example, are containers for their cell elements which themselves might contain text labels. At the root of the tree are one or more windows . When you connect to an application, you get access to all windows at the tree's root. To manipulate any of the system under test's GUI element, you first have to gain access to it. The group named "GUI Element Access" provides action blocks to access elements. The actions are further categorized into sub-groups named " Access By XPath" , "Access By Property", "Existence Test" etc. For each category, blocks with the same interface exist. The only difference is the method of retrieval.
- Get GUI Element
- returns a single GUI element matching the search condition. If multiple elements match the condition, an arbitrary element matching the condition is returned.
- To make sure that there is only one single element matching, set ensureExclusive to true. You can also enforce this globally by setting the environment variable "ensureExclusive" in your environment.
- If ensureExclusive is set and more than one element is found, the block will fail and notify about the ambiguity. This is disabled by default, as checking for uniqueness is a very costly (ie. slow) operation.
- You can also specify a timeout after which the search is aborted. This is useful if you know an element should be there already (or soon but might currently be loading).
- Get GUI Elements
- returns all GUI elements matching the search condition. If no element is found, it returns an empty list Timeout and cacheXPath are also applicable.
- Works exactly the same as Get GUI Element but does not fail if no element is found. Additionally provides an output pin specifying if an element has been found.
Both of these blocks are also available in the "Relative To Anchor" variety. Those blocks also take a GUI Element as anchor in relation to which they search.
XPath is the query language used for selecting GUI elements in the application's element tree. You can find the XPath of an element with the Gui Browser or with third-party applications like "Inspect.exe"  or FlaUInspect . As an example, the XPath of the scroll button on the vertical scrollbar in the German Windows Notepad.exe application looks like this:
/Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten']
For each level in the GUI tree, a new Location Step containing the child elements Control Type has to be added. You do not have to use any predicates, the following XPath is valid as well:
but might lead to issues as there might be multiple GUI Elements matching this path. You can select by three predicates that can be mixed and matched in the same query:
- the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file.
/Window[@Name='Unbenannt - Editor']
- An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced on lazy or incompetent programmers.
- Many lazy developers leave the ID empty or reuse it, making it useless. You should nontheless use this ID wherever applicable as it is static
- and meant for this exact purpose.
- List Index
- If there are multiple siblings with the same Control Type, you can select one of them by (1-based) list index.
Please Note: You can only use the three predicates above in your XPaths. Other predicates DO NOT work! If you want to search an element by another property (e.g. its help text), use the By Property variant of the block instead. If the property is not unique, you can select a unique ancestor of the target by property or XPath relative to which the property is unique. You can then use that ancestor as anchor in the [Get GUI Element By Property (Relative to Anchor)] block.
Using Wildcards in XPaths
XPath also supports wildcards. You can:
- replace one tree level with a "*" wildcard:
- replace a control type with a "*" wildcard:
- replace an arbitrary amount of levels with a "//" wildcard:
Especially the last option is useful as it makes your tests more readable as the reader can easily focus on the relevant parts of the xpath. But be careful. Using wildcards comes at a steep performance cost. If your tests run unacceptably slow, first try to remove wildcards where they are not needed.
You can also search for elements by property by specifying a value an element's property with the given name must have. For a list of properties a GUI element might have, take a look at Microsoft's documentation . It specifies the supported properties for each pattern an element can support.
XPath Resolution in Action Blocks
Getting a reference to the GUI Element everytime you need to manipulate it in any way can make for very unclean looking test suites as two blocks are needed to perform any action - one for getting a reference to the element and another one for perform the action. The library therefore also allows you to directly feed an xpath into every block that would take a reference to a GUI Element.
The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent. While Variant 2 is more concise, Variant 1 is the right choice in most cases. While it has an additional block, you have control over the other input parameters. You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.
Rule of thumb: Always use Variant 1 except if you need the reference to the GUI element only once.
Separating Path Definitions
It is highly recommended to add definitions for element-lathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:
- the paths are hidden and the values at the steps become more readable
- your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.
- you can import those definitions from a CSV or XML file, and thus separate the locators from the actual test suite. In case of changed UI-layout / UI-hierarchy after a revision change of the tested application, all you need is a new excel or xml file containing those definitions.
If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. CVS/excel or XML file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.
Interacting with GUI Elements
Once you retrieved a GUI element, you can interact with it by making use of the library's blocks located in the groups Elements, Patterns, and Properties. Blocks in those categories take a GUI Element (and sometimes additional data) as input and manipulate it or query for its properties in some way.
The 'Elements' Group
This group contains blocks that are applicable to blocks that only work on GUI elements of a specific type. For a block's detailed description with examples , take a look at the block's documentation in expecco.
Even though not only Checkboxes can be 'checked', the [Check] block in the "Checkbox group" only works for checkboxes and not e.g. for radio buttons. Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific pattern instead and are therefore in the "Patterns" group.
The 'Patterns' Group
This group contains blocks that are applicable to GUI elements supporting a certain pattern. Patterns describe how a GUI element behaves rather than what it is. An element can also support multiple patterns at the same time. You can find out which patterns an element supports with the [Get Supported Patterns] block or by taking a look at Microsoft's documentation about patterns .
Pattern blocks can therefore be re-used between GUI elements of different types. For example, the [Expand] block in "ExpandCollapse pattern", can be used to expand Comboboxes, Menus, Submenus, or Trees.
The 'Properties' Group
Blocks in the "Patterns" and "Elements" group already allow you to query the value of properties pertaining to that element or pattern. Blocks in the "Properties" group allow you to query both properties which all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.
Please note that the [Get Arbitrary Property] block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide 
Automating Keyboard and Mouse
You can control keyboard and mouse with action blocks in the "Keyboard" and "Mouse" groups, as shown on the right. In contrast to the blocks above, which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.
You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the [Left/Double/Right Click On Element] blocks for this). If you do not want to wait for the mouse to move, you can also use the [Set Focus] block instead.
The plugin provides blocks to interact with the keyboard on different abstraction levels. The most versatile block is [Type With Control Keys Pressed] where you can specify a string of characters to be typed while any number of control keys (e.g. "esc", "alt", "shift", "ctrl", ...) are held down. All other blocks are specializations of this block.
[Press Control Keys] e.g. allows you to enter key commands such as renaming files ("ALT+F2"), starting the task manager ("CTRL+ALT+DEL"), or closing a window ("ALT+F4"). [Press Control Key] makes it easy to just press a single key (e.g. "RETURN" to start a new line in an edit box). The "Shortcuts" group provides an assorted collection of often used key combinations for your convenience.
Use the [Type] action to just type some text, or if you just need some throwaway demo text, use [Type Random Phrase].
If you need more fine grained control over the keyboard, you can make use of Scan Codes. The keyboard sends a scan code to the operating system whenever a key is pressed or released. Microsoft provides a list of the scan codes it supports . [Press Key By Scan Code] sends a signal to Windows that a key with a given scan code has been pressed (and is held down). The matching [Release Key By Scan Code] sends Windows the adverse signal. Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use [Type Key By Scan Code].
The plugin provides blocks manipulating the mouse in three ways: Clicking, Moving, and interacting with its Buttons.
Recording and Taking Screenshots
Frequently Asked Questions
What is the difference between Click/Invoke/Select/Toggle?
My element supports a certain pattern but blocks of that pattern don't work?
I have designed custom elements, how can I automate them?
I have extended my GUI elements to have custom properties, how can I query them?
Hotkey "Get hovered element" selects the wrong element in the tree
Sometimes there are elements which overlap or hide other elements. This element is then selected in the tree. In this case, you can use the Hotkey "Get focused element" to select the element in the tree