WindowsAutomation Reference 2.0/en

From expecco Wiki (Version 2.x)
Revision as of 16:31, 20 September 2018 by Mb (Talk | contribs)

Jump to: navigation, search

This is the documentation for the Windows Automation 2.0 plugin.

It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.


  • Automated GUI testing of WPF, WinForms, and Win32 applications.
  • Contains an expecco block library and tools which help to model tests and inspect the GUI of the application.
  • Parallel remote test-execution on multiple systems.
  • Performs tests against already built executables. No source code changes / recompilation of the application is required.
  • Access to GUI components using XPath locators.
  • Access objects of the application live at execution time.
  • Full access to keyboard and mouse of the target system.
  • Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...

Known issues in the current Release (August 2018)

  • The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
  • Instant follow mouse is too expensive in Windows and is not and will not be supported. Use HotKeys to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey service hast to be started. Take care that the Hotkeys have not already been grabbed by another application that is running on your desktop (a warning will be shown in this case).
  • Recording is still in Beta State. Recording of Button clicks and Keystrokes work well.
  • ContextMenus cannot be recorded in the current version.
  • When doing a clicking the first submenu entry of the first pulldown menu, a wrong element is clicked. All other menu entries are correctly selected. To get it working you can "Invoke" the submenu entry instead of using "Click on Element"

Installation and Connection

The WindowsAutomation testing plugin uses the expecoo ".NET-Bridge" which consists of two parts: A plugin for expecco, and a server executable using the .NET framework running on the remote computer to connect to.


On the machine running the system under test:

  • .NET Framework 4.6.2 [1] or higher.
  • One of the following operation systems:
- Windows 7 SP1
- Windows 8
- Windows 8.1
- Windows 10
- Windows Server 2008 R2 SP1
- Windows Server 2012
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server, version 1709

Plugin Components

The plugin consists of the following parts:

  • The GUI Browser, used to interactively explore your application.
  • The Windows Automation Library, which provides blocks that you can use in your test cases.
  • The .NET-Bridge server, which provides an interface between the application under test and your tests running in expecco.

Installing and Configuring the Plugin in Expecco

The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "plugin" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You might need to restart your expecco session.

Preparing a local system for test execution

If you want to automate an application running on the same host as expecco, there is no need to set up anything! Everything should work out of the box.


Preparing a remote system for test execuction

The remote system has to run the ".Net-Bridge server" which in turn connects to expecco. For this, copy "DotNetBridgeServer.exe" (in the "packages/exept/bridgeFramework/dotNetBridge/dotNetBin" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.

To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments:

Arguments Required Description
-s Yes Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode, where it actively connects to the partner).
-k Yes Tells the client to keep the connection open after a connection has been terminated.
This saves you the hassle to restart it every time a test finishes.
-a <ip/hostname> Yes The IP-address or hostname from which it will accept connections. To ensure that no other machine connects to the server, set this to the IP of the machine your expecco is running on. To allow any machine (i.e. to have it listen on any IP-address), set it to
-p <int> No The port the server should listen on (default is 34318).

So you may execute on the remote host:

   DotNetBridgeServer.exe -s -k -a


There are currently no configurable settings.

GUI Browser

The GUI browser allows you to explore a running application. You can browse the application, inspect element properties and execute actions. For a more detailed description also see Gui Browser.


This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.

Connecting to an Application

To automate an application, you first have to connect to it. You can do this with the blocks from the Connecting group.

The Library's Connection Blocks.

You can connect to an application by its executable name, or by its process id. If you use the executable name, expecco will attach to the main process of the specified executable. Be careful as this is not necessarily the process drawing the GUI! Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process. All connection blocks also require you to specify a name by which the connection should be known in future. This is needed for when you automate multiple applications at the same time and need to change connection contexts during test execution. Please note that the application to connect to has to be started before the corresponding connection block is executed. You can change in context of which connection a block is executed with the Set Active Connection block. This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.

Establishing a Local Connection

If you want to automate an application running on the same host as expecco, there is no need to set up anything! Just use the Create Local Connection with the mentioned above parameters.

Establishing a Remote Connection

Make sure that the DotNet Bridge server is started as explained in Preparing a remote system for test execuction When the bridge is running, you can connect to it with the Create Remote Connection block using the hostname and IP-Address you specified when you started the bridge.

Closing a Connection

Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources. One simple way to do this is to setup your connections in the Before Execution part of the test suite and put the Close All Connections block in the After exeuction part.

Accessing GUI Elements

The Library's GUI Element Access Blocks.
A Few Examples on How to Access Elements in Notepad.
Different Ways to Reference a GUI Element.

All Windows applications are built as a tree of GUI elements . Some elements act as container of other element - their children. Tables, for example, are containers for their cell elements which themselves might contain text labels. At the root of the tree are one or more windows . When you connect to an application, you get access to all windows at the tree's root. To manipulate any of the system under test's GUI element, you first have to gain access to it. The group GUI Element Access provides blocks that help you to gain access to the elements. There are 12 blocks in total, divided between Access By XPath , Access By Property, and Existence Test. For each category, blocks with the same interface exist. The only difference is the method of retrival.

Get GUI Element 
returns a single GUI Element matching the search condition. If multiple match the condition, an arbitrary element matching the condition is returned.
If you want to make sure that there is only one element matching, set ensureExclusive to true. You can also enforce this globally by setting the environment variable ensureExclusive in your environment.
If ensureExclusive is set and more than one element is found, the block will fail and notify you of the ambiguity. This is disabled by default as checking for uniqueness is a very costly operation.
You can also specify a timeout after which the search is aborted. This is useful if you know an element should be there soon but might currently be loading.
Get GUI Elements
returns all GUI Element matching the search condition. If no element is found, it returns an empty list Timeout and cacheXPath are also applicable.
Works exactly the same as Get GUI Element but does not fail if no element can be found. Additionally provides an output pin specifying if an element has been found.

Both of these blocks are also available in the Relative To Anchor variety. Those blocks also take a GUI Element as anchor in relation to which they search.

Using XPaths

XPath is the query language this plugin uses for selecting GUI elements in the application's element tree. You can find the XPath of an element with the Gui Browser or with third-party applications like Inspect.exe [2] or FlaUInspect [3]. An XPath for the scroll button on the vertical scrollbar in the german Windows Notepad application looks like this:

 /Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten'] 

For each level in the GUI tree, a new Location Step containing the child elements Control Type has to be added. You do not have to use any predicates, the following XPath is valid as well:


but might lead to issues as there might be multiple GUI Elements matching this path. You can select by three predicates that can be mixed and matched in the same query:

the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file.
/Window[@Name='Unbenannt - Editor']
An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced.
Many developersleave the ID empty or reuse it, making it useless. You should nontheless use this ID wherever applicable as it is static
and meant for this exact purpose.
List Index
If there are multiple siblings with the same Control Type, you can select one of them by (1-based) list index.

Please Note: You can only use the three predicates above in your XPaths. Other predicates DO NOT work! If you want to search an element by another property (e.g. its help text), use the By Property variant of the block instead. If the property is not unique, you can select a unique ancestor of the target by property or XPath relative to which the property is unique. You can then use that ancestor as anchor in the Get GUI Element By Property (Relative to Anchor) block.

Using Wildcards in XPaths

XPath also supports wildcards. You can:

  • replace one tree level with a "*" wildcard:


  • replace a control type with a "*" wildcard:


  • replace an arbitrary amount of levels with a "//" wildcard:


Especially the last option is useful as it makes your tests more readable as the reader can easily focus on the relevant parts of the xpath. But be careful. Using wildcards comes at a steep performance cost. If your tests run unacceptably slow, first try to remove wildcards where they are not needed.

Using Properties

You can also search for elements by property by specifying a value an element's property with the given name must have. For a list of properties a GUI element might have, take a look at Microsoft's documentation [4]. It specifies the supported properties for each pattern an element can support.

XPath Resolution in Action Blocks

Getting a reference to the GUI Element everytime you need to manipulate it in any way can make for very unclean looking test suites as two blocks are needed to perform any action - one for getting a reference to the element and another one for perform the action. The library therefore also allows you to directly feed an xpath into every block that would take a reference to a GUI Element.

The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent. While Variant 2 is more concise, Variant 1 is the right choice in most cases. While it has an additional block, you have control over the other input parameters. You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.

Rule of thumb: Always use Variant 1 except if you need the reference to the GUI element only once.

Separating Path Definitions

It is highly recommended to add definitions for element-lathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:

  • the paths are hidden and the values at the steps become more readable
  • your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.

If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. cvs or excel or xml file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.

Interacting with GUI Elements

Once you retrieved a GUI element, you can interact with it by making use of the library's blocks located in the groups Elements, Patterns, and Properties. Blocks in those categories take a GUI Element (and sometimes additional data) as input and manipulate it or query for its properties in some way.

The Library's Blocks for Interacting With Elements.

The 'Elements' Group

This group contains blocks that are applicable to blocks that only work on GUI elements of a specific type. For a block's detailed description with examples , take a look at the block's documentation in expecco.

Even though not only Checkboxes can be 'checked', the Check block in the Checkbox group only works for checkboxes and not e.g. for radio buttons. Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific pattern instead and are therefore in the Patterns group.

The 'Patterns' Group

This group contains blocks that are applicable to GUI elements supporting a certain pattern. Patterns describe how a GUI element behaves rather than what it is. An element can also support multiple patterns at the same time. You can find out which patterns an element supports with the Get Supported Patterns block or by taking a look at Microsoft's documentation about patterns [5].

Pattern blocks can therefore be re-used between GUI elements of different types. The Expand block of the ExpandCollapse pattern, for example, can be used to expand Comboboxes, Menus, Submenus, or Trees.

The 'Properties' Group

Blocks in the Patterns and Elements group already allow you to query the value of properties pertaining to that element or pattern. Blocks of the Properties group allow you to query properties all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.

Please note that the Get Arbitrary Property block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide [6]

Automating Keyboard and Mouse

The Library's Blocks for Interacting With Keyboard and Mouse.

You can control keyboard and mouse with the blocks in the groups Keyboard and Mouse that can be seen on the right. In contrast to the blocks above which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.

You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the Left/Double/Right Click On Element blocks for this). If you do not want to wait for the mouse to move, you can also use the Set Focus block instead.


The plugin provides blocks interacting with the keyboard on different abstraction levels. The most versatile block is Type With Control Keys Pressed where you can specify a string of characters to be typed while any number of control keys (e.g. esc, alt, shift, ctrl, ...) are held down. All other blocks are specializations of this block.

Press Control Keys e.g. allows you to enter key commands such as renaming files (ALT+F2), starting the task manager (CTRL+ALT+DEL), or closing a window (ALT+F4). Press Control Key makes it easy to just press a single key (e.g. RETURN to start a new line in an edit box). The Shortcuts group provides an assorted collection of often used key combinations for your convenience.

Use the Type to just type some text or, if you just need some throwaway demo text, use Type Random Phrase.

If you need more fine grained control over the keyboard, you can make use of Scan Codes. The keyboard sends a scan codeto the operating system whenever a key is pressed or released. Microsoft provides a list of the scan codes it supports [7]. Press Key By Scan Code sends Windows a signal that a key with a given scan code has been pressed (and is held down). The matching Release Key By Scan Code sends Windows the adverse signal. Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use Type Key By Scan Code.


The plugin provides blocks manipulating the mouse in three ways: Clicking, Moving, and interacting with its Buttons.

Recording and Taking Screenshots

Frequently Asked Questions

What is the difference between Click/Invoke/Select/Toggle?

My element supports a certain pattern but blocks of that pattern don't work?

I have designed custom elements, how can I automate them?

I have extended my GUI elements to have custom properties, how can I query them?

Copyright © 2014-2018 eXept Software AG