WindowsAutomation Reference 2.0/en

Aus expecco Wiki (Version 2.x)
(Weitergeleitet von WindowsAutomation Reference 2.0)
Zur Navigation springen Zur Suche springen

This is the documentation for the Windows Automation 2.0 plugin. It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.

Notice:
The vsn2 replaces the previous vsn1 plugin, which is no longer maintained. The change was necessary due to changes in the Windows operating system, and features which were discontinued by Microsoft. The vsn2 plugin uses Microsoft's UI-Automation2 API to interact with the application. This API was not avaliable in Windows-XP, Vista and other older OS versions, and many features used by the previous version are no longer supported in newer OS versions, or need different parameters.


Inhaltsverzeichnis

Features[Bearbeiten]

  • Automated GUI testing of WPF, WinForms, and Win32 applications.
  • Contains an expecco block library and tools which help to model tests and inspect the GUI of the application.
  • Parallel remote test-execution on multiple systems.
  • Performs tests against already built executables. No source code changes / recompilation of the application is required.
  • Access to GUI components using XPath locators.
  • Access objects of the application live at execution time.
  • Full access to keyboard and mouse of the target system.
  • Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...

Known issues in the current Release 19.1.0[Bearbeiten]

  • The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing Pre-19.1.0 test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
  • Instant follow mouse is very expensive in Windows and is therefore disabled. Use HotKeys to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey-service has to be started. Take care that the keys have not already been grabbed by another application. If that is the case, a warning will be shown, and you should either make sure that expecco is started before the other application, or that you use other hotkey combinations (in either expecco or the application).

Limitations[Bearbeiten]

  • WindowsAutomation only supports XPath1.0.
That means for example, that the "ends-with" function, which is an XPath2 function, is not supported and thus not possible as XPath attribute filter.
See [XPath vs. XPath2] for details. For some (especially "ends-with") there are workarounds by comparing against a substring which is based on the length of the attribute string or a regex match (see [1]).
Notice, that this is a limitation of the Windows operating system, which cannot be worked around in expecco.

Installation and Connection[Bearbeiten]

The WindowsAutomation testing plugin uses the expecoo ".NET-Bridge" which consists of two parts: A plugin for expecco, and a server executable using the .NET framework running on the remote computer to connect to.

Requirements[Bearbeiten]

On the machine running the system under test:

  • .NET Framework 4.7 [2] or higher.
  • One of the following operation systems:
- Windows 7 SP1
- Windows 8
- Windows 8.1
- Windows 10
- Windows 11
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
- Windows Server 2022

Plugin Components[Bearbeiten]

The plugin consists of the following parts:

  • The GUI Browser, used to interactively explore your application.
  • The Windows Automation Library, which provides blocks that you can use in your test cases.
  • The .NET-Bridge server, which provides an interface between the application under test and expecco.

Installing and Configuring the Plugin in Expecco[Bearbeiten]

The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "plugin" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You may have to restart your expecco session, if the plugin is installed later.

Preparing a local system for test execution[Bearbeiten]

To automate applications running on the SAME host as expecco, there is no need to set up anything - it should work out of the box.

Try to connect to an application via the GUIBrowser's WindowsAutomation-connect dialog:

WindowsAutomation2Connect.png

Preparing a remote system for test execuction[Bearbeiten]

Security Considerations[Bearbeiten]

Please make sure, that remote access is only possible from inside a secure local network. Under no circumstances should you grant access to the DotNetBridgeServer from untrusted hosts.

DotNet Bridge Server[Bearbeiten]

The remote system has to run the ".Net-Bridge server" which in turn either actively connects to expecco or passively waits for an incoming connection from expecco.

For this, copy "DotNetBridgeServer.exe" and "DotNetBridge.dll" (found in the "packages/exept/bridgeFramework/dotNetBridge/dotNetBin" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.

To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments ("DotNetBridgeServer.exe --help" outputs a list of valid options):

Arguments Required Description
-s Yes "server mode". Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode,
where it actively connects to the partner).
-k Yes "keepalive". Tells the client to keep the connection open after a connection has been terminated.
This saves you the hassle to restart it every time a test finishes.
-a <ip/hostname> Yes "IP address accepting connections". The IP-address or hostname of an interface on the host on which connections will be accepted (in server mode) or to which it connects (in client mode).
If you have multiple network interfaces, you can ensure that only incoming connections on the secure network will be accepted by setting the IP address of this interface.
To allow incoming connections on any network interface (i.e. to have it listen on any IP-address), set it to 0.0.0.0. If you are not inside a secure network, you should use a secure tunnel (e.g. a SSH or Powershell tunnel) between the machine running expecco and the machine running the bridge and listen only on 127.0.0.1 (the default).
-p <int> No "port". The port the server should listen on (in server mode) or to which is should connect (default is 34318).
-L <string> No "assembly". Load a .NET assembly on startup. Omit the .dll extension (so use "WindowsAutomation" but not "WindowsAutomation.dll").
You can repeat this option to load multiple assemblies.

Note: this is not normally needed, as expecco loads the required assemblies automatically into the bridge.
However, if security settings forbid external assemblies, you have to copy them to the target machine, and load them via command line option.

So you may execute on the remote host:

   DotNetBridgeServer.exe -s -k -a 0.0.0.0

For automatic tests, you must setup the target system for the bridge server to be running; either by adding it to some "auto exec' script, or by adding to the "scheduled tasks", or by remote starting it from your test-host via a powershell script, remote ssh command or similar.

If you get an "Assembly not Found" error, you may have an incompatible .NET framework or the "DotNetBridge.dll" is missing. If you get a network error, or the expecco-host cannot connect, make sure that your firewall settings allow access to the given ports (you may try an alternative port like 8080, which must then be specified on both sides).

Settings[Bearbeiten]

There are currently no configurable settings.

GUI Browser[Bearbeiten]

The GUI browser is used to explore a running application. You can browse the application, inspect element properties and execute actions. See " Expecco GUI Tests Extension Reference" for details.

Library[Bearbeiten]

This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.

Connect to Application vs. Connect to Screen[Bearbeiten]

You can either "connect to a single application", or "connect to the screen". The first provides faster access to the components, but will not work, if the app creates subprocesses with a GUI (eg. for dialogs or additional windows). There exist also applications which always create a subprocess, even for the main GUI.

For such applications, you have to "connect to the screen", and provide an additional window-reference in the XPath locator.
You must also use a screen connection, if your test contains interactions between multiple top-level windows, for example a drag&drop between windows.

Connecting to an Application[Bearbeiten]

To automate an application, you first have to connect to it or to its screen. You can do this with the blocks from the "Connecting" group.

The Library's Connection Blocks (old picture without screen connection blocks).

You can connect to an individual application by its executable name, or by its process id. If you use the executable name, expecco will attach to the main process of the specified executable. Be careful as this is not necessarily the process drawing the GUI! Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process.

All connection blocks also require you to specify a name by which the connection should be known in future. This is needed when you automate multiple applications at the same time and need to change connection contexts during test execution. Please note that the application to connect to has to be started before the corresponding connection block is executed. You can change in context of which connection a block is executed with the [Set Active Connection] block. This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.

Establishing a Local Connection[Bearbeiten]

To automate an application running on the same host as expecco, there is no need to set up anything. Use "[Create Local Connection]" with the above mentioned parameters.

Establishing a Remote Connection[Bearbeiten]

Make sure that the DotNet-Bridge server ("DotNetBridgeServer.exe") is running on the remote system, as explained in the chapter "Preparing a remote system for test execuction" When the bridge is running, you can connect to it with the "[Create Remote Connection]" block using the hostname and IP-Address you specified when you started the bridge.

Notice, that the DotNet-Bridge server only accepts local connections by default. This is done for security reasons. Use the "-a x.x.x.x" command line argument to specify, which hosts are allowed to connect. With "-a 0.0.0.0", any host is allowed. Do NOT use this, if the remote host is reachable from untrusted hosts.

Closing a Connection[Bearbeiten]

Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources. One simple way to do this is to setup your connection in the "Before Execution" (pre-action) part of the test suite or test case, and put the "[Close All Connections]" action block in the "After exeuction" (post-action) part.


Accessing GUI Elements[Bearbeiten]

The Library's GUI Element Access Blocks.
A Few Examples on How to Access Elements in Notepad.
Different Ways to Reference a GUI Element.

All Windows applications are built as a tree of GUI elements. Some elements act as container of other element - their children. Tables, for example, are containers for their cell elements which themselves might contain text labels. At the root of the tree are one or more windows . When you connect to an application, you get access to all windows at the tree's root. To manipulate any of the system under test's GUI element, you first have to gain access to it. The group named "GUI Element Access" provides action blocks to access elements. The actions are further categorized into sub-groups named " Access By XPath" , "Access By Property", "Existence Test" etc. For each category, blocks with the same interface exist. The only difference is the method of retrieval.

Get GUI Element 
returns a single GUI element matching the search condition. If multiple elements match the condition, an arbitrary element matching the condition is returned.
To make sure that there is only one single element matching, set ensureExclusive to true. You can also enforce this globally by setting the environment variable "ensureExclusive" in your environment.
If ensureExclusive is set and more than one element is found, the block will fail and notify about the ambiguity. This is disabled by default, as checking for uniqueness is a very costly (ie. slow) operation.
You can also specify a timeout after which the search is aborted. This is useful if you know an element should be there already (or soon but might currently be loading).
Get GUI Elements
returns all GUI elements matching the search condition. If no element is found, it returns an empty list Timeout and cacheXPath are also applicable.
Exists
Works exactly the same as Get GUI Element but does not fail if no element is found. Additionally provides an output pin specifying if an element has been found.

Both of these blocks are also available in the "Relative To Anchor" variety. Those blocks also take a GUI Element as anchor in relation to which they search.


Using XPaths[Bearbeiten]

XPath is the query language used to select GUI elements in the application's element tree. You can find the XPath of an element with the Gui Browser or with third-party applications like "Inspect.exe" [3] or "FlaUInspect" [4].
As an example, the full XPath of the scroll button on the vertical scrollbar in the german Windows "Notepad.exe" application looks like this:

 /Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten'] 

For each level in the GUI tree, a new Location Step containing the child element's Control Type was added in the above XPath.
You do not have to use any predicates, the following XPath is valid as well:

 /Window/Document/ScrollBar/Button 

but might lead to issues if multiple GUI Elements match this path.

You can select by three predicates (which can be mixed and matched in the same query):

Name
the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file).
/Window[@Name='Unbenannt - Editor']
AutomationId
An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced on lazy or incompetent programmers.
Many lazy developers leave the ID empty or reuse it, making it useless.
You should use this ID wherever applicable as it is static and meant for this exact purpose.
/Window/Document[@AutomationId='15']
List Index
If there are multiple siblings with the same Control Type, you can select one of them by (1-based) list index.
/Window[1]/Document[1]
/Window[1]/Document[last()]

Warning there are XPath Limitations[Bearbeiten]

Please Note: You can ONLY use the three predicates above in your XPaths.
Other predicates DO NOT work!
To search for an element by another property (e.g. its help text), use the By Property variant of the block instead. If the property is not unique, you can select a unique ancestor of the target by property or XPath relative to which the property is unique. You can then use that ancestor as anchor in the [Get GUI Element By Property (Relative to Anchor)] block.

Using Wildcards in XPaths[Bearbeiten]

XPath also supports wildcards. You can:

  • replace one tree level with a "*" wildcard:
/Window/Document[@AutomationId='15']/*/Button[1]
/*/*/*/Button[1]
  • replace a control type with a "*" wildcard:
/Window/*[@AutomationId='15']
  • replace an arbitrary amount of levels with a "//" wildcard:
//Button
/Window//Button[1]

Especially the last option is useful as it makes your tests more readable, in that the reader can easily focus on the relevant parts of the xpath.

But be careful: using wildcards comes at a steep performance cost (due to the implementation inside the Windows API). If your test runs unacceptably slow, first try to remove wildcards where they are not needed.

Using Properties[Bearbeiten]

You can also search for elements by property; i.e. by specifying a value an element's property with the given name must have.

For a list of properties a GUI element might have, take a look at Microsoft's documentation [5].

XPath Resolution in Action Blocks[Bearbeiten]

Xpath resolution may be slow in the Windows Automation API, depending on the number of windows/elements which have to be checked.

To speed up test execution, you should reuse a resolved XPath if possible. For this, use the "Get Element by XPath" action, and feed the "element handle" to multiple actions. All action- and property access blocks can take either an XPath or an element handle as locator input.

The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent. While Variant 2 is more concise, Variant 1 is the right choice in most cases. While it has an additional block, you have control over the other input parameters. You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.

Rule of thumb: Always use Variant 1 except if you need the reference to the GUI element only once, or if the application replaces the element dynamically.

Separating Path Definitions[Bearbeiten]

It is highly recommended to add definitions for element-pathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:

  • the paths are hidden and the values at the steps become more readable
  • your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.
  • you can import those definitions from a CSV or XML file, and thus separate the locators from the actual test suite. In case of changed UI-layout / UI-hierarchy after a revision change of the tested application, all you need is a new excel or xml file containing those definitions.

If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. CVS or XML file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.


Interacting with GUI Elements[Bearbeiten]

Once you retrieved a GUI element, you can interact with it by making use of the library's blocks located in the groups Elements, Patterns, and Properties. Blocks in those categories take a GUI Element (and sometimes additional data) as input and manipulate it or query for its properties in some way.

The Library's Blocks for Interacting With Elements.


The 'Elements' Group[Bearbeiten]

This group contains blocks that are applicable to blocks that only work on GUI elements of a specific type. For a block's detailed description with examples , take a look at the block's documentation in expecco.

Even though not only Checkboxes can be 'checked', the [Check] block in the "Checkbox group" only works for checkboxes and not e.g. for radio buttons. Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific pattern instead and are therefore in the "Patterns" group.

The 'Patterns' Group[Bearbeiten]

This group contains blocks that are applicable to GUI elements supporting a certain pattern. Patterns describe how a GUI element behaves rather than what it is. An element can also support multiple patterns at the same time. You can find out which patterns an element supports with the [Get Supported Patterns] block or by taking a look at Microsoft's documentation about patterns [6].

Pattern blocks can therefore be re-used between GUI elements of different types. For example, the [Expand] block in "ExpandCollapse pattern", can be used to expand Comboboxes, Menus, Submenus, or Trees.

The 'Properties' Group[Bearbeiten]

Blocks in the "Patterns" and "Elements" group already allow you to query the value of properties pertaining to that element or pattern. Blocks in the "Properties" group allow you to query both properties which all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.

Please note that the [Get Arbitrary Property] block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide [7]


Automating Keyboard and Mouse[Bearbeiten]

The Library's Blocks for Interacting With Keyboard and Mouse.

You can control keyboard and mouse with action blocks in the "Keyboard" and "Mouse" groups, as shown on the right. In contrast to the blocks above, which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.

You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the [Left/Double/Right Click On Element] blocks for this). If you do not want to wait for the mouse to move, you can also use the [Set Focus] block instead.

Keyboard[Bearbeiten]

The plugin provides blocks to interact with the keyboard on different abstraction levels. The most versatile block is [Type With Control Keys Pressed] where you can specify a string of characters to be typed while any number of control keys (e.g. "esc", "alt", "shift", "ctrl", ...) are held down. All other blocks are specializations of this block.

[Press Control Keys] e.g. allows you to enter key commands such as renaming files ("ALT+F2"), starting the task manager ("CTRL+ALT+DEL"), or closing a window ("ALT+F4"). [Press Control Key] makes it easy to just press a single key (e.g. "RETURN" to start a new line in an edit box). The "Shortcuts" group provides an assorted collection of often used key combinations for your convenience.

Use the [Type] action to just type some text, or if you just need some throwaway demo text, use [Type Random Phrase].

If you need more fine grained control over the keyboard, you can make use of Scan Codes. The keyboard sends a scan code to the operating system whenever a key is pressed or released. Microsoft provides a list of the scan codes it supports [8]. [Press Key By Scan Code] sends a signal to Windows that a key with a given scan code has been pressed (and is held down). The matching [Release Key By Scan Code] sends Windows the adverse signal. Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use [Type Key By Scan Code].

Mouse[Bearbeiten]

The plugin provides blocks manipulating the mouse in three ways: Clicking, Moving, and interacting with its Buttons.



Recording and Taking Screenshots[Bearbeiten]

Frequently Asked Questions[Bearbeiten]

Some elements are not presented in the UI browser's tree[Bearbeiten]

This happens if the application "draws" elements (such as checkboxes, buttons or text) without using Windows widgets and without providing proper automation support (btw: which a well behaved application should, because this also affects/prevents usability and makes these controls inaccessable to eg. users with vision disabilities). We encountered such in table-views containing checkboxes (probably OLE embedded database-tables). These elements must be addressed either via coordinates (relative to the containing parent element), or via an image search (grab and define the image in the GUI browser). Both have disadvantages and are inconvenient, but there is no other technical solution.

What is the difference between Click/Invoke/Select/Toggle?[Bearbeiten]

Click sends a click event; Invoke calls directly into the application's or widget's event handler; Select and Toggle usually call the corresponding automation functions (if the app-programmer provided that).

A click generates a click event to the widget; may disfunction if the widget is invisible or covered (eg. by a tooltip).

An invoke skips over the event mechanism and calls the event hander directly. It may also not work, if the event handler insists on keyboard focus, mouse position, shift- or ctrl-key or other state.

There exists no mechanism which work with any application: for some, you will have to record a mouse-move and/or a set-focus for the click to work. If possible, disable tooltips and other automatically appearing popups in the target app.

If none works, try the click at screen position action (which generates a real mouse-click, not via UI-automation to the widget, but via the Windows Event Queue). This should work in any case, but suffers most from popups, mouse position and keyboard focus.

My element supports a certain pattern but blocks of that pattern don't work[Bearbeiten]

I have designed custom elements, how can I automate them?[Bearbeiten]

I have extended my GUI elements to have custom properties, how can I query them?[Bearbeiten]

Hotkey "Get hovered element" selects the wrong element in the tree[Bearbeiten]

Sometimes there are elements which overlap or hide other elements. This element is then selected in the tree. In this case, you can use the Hotkey "Get focused element" to select the element in the tree

DevExpress controls are not visible in the GUI Browser[Bearbeiten]

DevExpress has some possibilities to disable UIAutomation support, which is also used by expecco.If you can't see one or some controls, please check e.g. https://supportcenter.devexpress.com/ticket/details/bc4069/devexpress-controls-now-clear-ui-automation-peers-for-the-application and other information on the DevExpress Support Site.

Connect fails with "Permission Denied"[Bearbeiten]

Some apps required admin access rights to connect to. You will get a "failed to connect" error and a walkback which looks like:

Could not connect to application: wireguard
System.ComponentModel.Win32Exception (0x80004005): Access denied
 bei System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)
 bei System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId, Boolean firstModuleOnly)
 bei System.Diagnostics.NtProcessManager.GetFirstModuleInfo(Int32 processId)
 bei System.Diagnostics.Process.get_MainModule()
 bei FlaUI.Core.WindowsAPI.WindowsApiTools.GetMainModuleFilepath(Process process)
 bei FlaUI.Core.Application.Attach(Process process)
 bei WindowsAutomation.Application..ctor(String name, UIA3Automation automation, PropertyRepository propertyRepository, String windowTitle)
 bei WindowsAutomation.Client.GetApplication(String name, String windowTitle)

You have two options:

  • run expecco as admin
  • start a dotNet-bridge manually as admin (in the local or remote machine):
   ...\bridgeFramework\dotNetBridge\dotNetBin>DotNetBridgeServer.exe -k -s -a 0.0.0.0 -p 34319 
then create a "remote connection" to it.



Copyright © 2014-2024 eXept Software AG