WindowsAutomation Reference 2.0/en: Unterschied zwischen den Versionen

Aus expecco Wiki (Version 2.x)
Zur Navigation springen Zur Suche springen
 
(111 dazwischenliegende Versionen von 3 Benutzern werden nicht angezeigt)
Zeile 1: Zeile 1:
This is the documentation for the Windows Automation 2.0 plugin.
This is the documentation for the Windows Automation 2.0 plugin.

It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.
It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.

Notice:<br>The vsn2 replaces the previous vsn1 plugin, which is no longer maintained.
The change was necessary due to changes in the Windows operating system, and features which were discontinued by Microsoft.
The vsn2 plugin uses Microsoft's UI-Automation2 API to interact with the application.
This API was not avaliable in Windows-XP, Vista and other older OS versions, and many features used by the previous version are no longer supported in newer OS versions, or need different parameters.



=Features=
=Features=
Zeile 13: Zeile 18:
*Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...
*Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...


= Known issues in the current Release (August 2018) =
= Known issues in the current Release 19.1.0 =
* The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
* The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing Pre-19.1.0 test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
* Instant follow mouse is too expensive in Windows and is not and will not be supported. Use HotKeys to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey service hast to be started. Take care that the Hotkeys have not already been grabbed by another application that is running on your desktop (a warning will be shown in this case).
* Instant follow mouse is very expensive in Windows and is therefore disabled. Use ''HotKeys'' to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey-service has to be started. Take care that the keys have not already been grabbed by another application. If that is the case, a warning will be shown, and you should either make sure that expecco is started before the other application, or that you use other hotkey combinations (in either expecco or the application).
= Limitations =
* Recording is still in Beta State. Recording of Button clicks and Keystrokes work well.
* WindowsAutomation only supports XPath1.0.
* ContextMenus cannot be recorded in the current version.
: That means for example, that the "ends-with" function, which is an XPath2 function, is not supported and thus not possible as XPath attribute filter.
* When doing a clicking the first submenu entry of the first pulldown menu, a wrong element is clicked. All other menu entries are correctly selected. To get it working you can "Invoke" the submenu entry instead of using "Click on Element"
: See [[https://en.wikipedia.org/wiki/XPath_2.0#Function_library XPath vs. XPath2]] for details. For some (especially "ends-with") there are workarounds by comparing against a substring which is based on the length of the attribute string or a regex match (see [https://stackoverflow.com/questions/40934644/xpath-testing-that-string-ends-with-substring]).
: Notice, that this is a limitation of the Windows operating system, which cannot be worked around in expecco.


=Installation and Connection=
=Installation and Connection=
Zeile 25: Zeile 32:
==Requirements==
==Requirements==
On the machine running the system under test:
On the machine running the system under test:
*.NET Framework 4.6.2 [https://www.microsoft.com/net/download/all] or higher.
*.NET Framework 4.7 [https://www.microsoft.com/net/download/all] or higher.
* One of the following operation systems:
* One of the following operation systems:
::- Windows 7 SP1
::- Windows 7 SP1
Zeile 31: Zeile 38:
::- Windows 8.1
::- Windows 8.1
::- Windows 10
::- Windows 10
::- Windows 11


::- Windows Server 2008 R2 SP1
::- Windows Server 2012
::- Windows Server 2012 R2
::- Windows Server 2012 R2
::- Windows Server 2016
::- Windows Server 2016
::- Windows Server, version 1709
::- Windows Server 2019
::- Windows Server 2022


==Plugin Components==
==Plugin Components==
The plugin consists of the following parts:
The plugin consists of the following parts:
* The GUI Browser, used to interactively explore your application.
* The [[GUI Browser]], used to interactively explore your application.
* The Windows Automation Library, which provides blocks that you can use in your test cases.
* The Windows Automation Library, which provides blocks that you can use in your test cases.
* The .NET-Bridge server, which provides an interface between the application under test and your tests running in expecco.
* The .NET-Bridge server, which provides an interface between the application under test and expecco.


==Installing and Configuring the Plugin in Expecco==
==Installing and Configuring the Plugin in Expecco==
The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "<code>plugin</code>" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You might need to restart your expecco session.
The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "<code>plugin</code>" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You may have to restart your expecco session, if the plugin is installed later.


==Preparing a local system for test execution==
==Preparing a local system for test execution==
If you want to automate an application running on the same host as expecco, there is no need to set up anything! Everything should work out of the box.
To automate applications running on the SAME host as expecco, there is no need to set up anything - it should work out of the box.

Try to connect to an application via the GUIBrowser's ''WindowsAutomation''-connect dialog:


[[Datei:WindowsAutomation2Connect.png]]
[[Datei:WindowsAutomation2Connect.png]]


==Preparing a remote system for test execuction==
==Preparing a remote system for test execuction==
The remote system has to run the "''.Net-Bridge server''" which in turn connects to expecco.
For this, copy "<code>DotNetBridgeServer.exe</code>" (in the "<code>packages/exept/bridgeFramework/dotNetBridge/dotNetBin</code>" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.


===Security Considerations===
To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments:
Please make sure, that remote access is only possible from inside a secure local network.
Under no circumstances should you grant access to the DotNetBridgeServer from untrusted hosts.

===DotNet Bridge Server===

The remote system has to run the "''.Net-Bridge server''" which in turn either actively connects to expecco or passively waits for an incoming connection from expecco.

For this, copy "<code>DotNetBridgeServer.exe</code>" and "<code>DotNetBridge.dll</code>" (found in the "<code>packages/exept/bridgeFramework/dotNetBridge/dotNetBin</code>" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.

To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments
("<code>DotNetBridgeServer.exe --help</code>" outputs a list of valid options):
{| class="wikitable"
{| class="wikitable"
|-
|-
! Arguments !! Required !! Description
! Arguments !! Required !! Description
|-
|-
| -s || Yes || Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode, where it actively connects to the partner).
| -s || Yes || "''server mode''". Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode,<br>where it actively connects to the partner).
|-
|-
| -k || Yes || Tells the client to keep the connection open after a connection has been terminated.<br>This saves you the hassle to restart it every time a test finishes.
| -k || Yes || "''keepalive''". Tells the client to keep the connection open after a connection has been terminated.<br>This saves you the hassle to restart it every time a test finishes.
|-
|-
| -a&nbsp;<ip/hostname>|| Yes || The IP-address or hostname from which it will accept connections. To ensure that no other machine connects to the server, set this to the IP of the machine your expecco is running on. To allow any machine (i.e. to have it listen on any IP-address), set it to 0.0.0.0.
| -a&nbsp;<ip/hostname>|| Yes || "''IP address accepting connections''". The IP-address or hostname of an interface on the host on which connections will be accepted (in server mode) or to which it connects (in client mode).<br>If you have multiple network interfaces, you can ensure that only incoming connections on the secure network will be accepted by setting the IP address of this interface.<br>To allow incoming connections on any network interface (i.e. to have it listen on any IP-address), set it to 0.0.0.0. If you are not inside a secure network, you should use a secure tunnel (e.g. a SSH or Powershell tunnel) between the machine running expecco and the machine running the bridge and listen only on 127.0.0.1 (the default).
|-
|-
| -p <int> || No || The port the server should listen on (default is 34318).
| -p <int> || No || "''port''". The port the server should listen on (in server mode) or to which is should connect (default is 34318).
|-
| -L <string> || No || "''assembly''". Load a .NET assembly on startup. Omit the .dll extension (so use "WindowsAutomation" but not "WindowsAutomation.dll").<br>You can repeat this option to load multiple assemblies.
Note: this is not normally needed, as expecco loads the required assemblies automatically into the bridge.<br>However, if security settings forbid external assemblies, you have to copy them to the target machine, and load them via command line option.
|}
|}


Zeile 73: Zeile 94:


DotNetBridgeServer.exe -s -k -a 0.0.0.0
DotNetBridgeServer.exe -s -k -a 0.0.0.0

For automatic tests, you must setup the target system for the bridge server to be running; either by adding it to some "auto exec' script, or by adding to the "scheduled tasks", or by remote starting it from your test-host via a powershell script, remote ssh command or similar.

If you get an "Assembly not Found" error, you may have an incompatible .NET framework or the "DotNetBridge.dll" is missing. If you get a network error, or the expecco-host cannot connect, make sure that your firewall settings allow access to the given ports (you may try an alternative port like 8080, which must then be specified on both sides).


=Settings=
=Settings=
Zeile 78: Zeile 103:


=GUI Browser=
=GUI Browser=
The GUI browser allows you to explore a running application. You can browse the application, inspect element properties and execute actions. For a more detailed description also see [[Expecco GUI Tests Extension Reference|Gui Browser]].
The GUI browser is used to explore a running application. You can browse the application, inspect element properties and execute actions. See "[[GUI Browser| Expecco GUI Tests Extension Reference]]" for details.


=Library=
=Library=
This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.
This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.

== Connect to Application vs. Connect to Screen ==
You can either "connect to a single application", or "connect to the screen".
The first provides faster access to the components, but will not work, if the app creates subprocesses with a GUI (eg. for dialogs or additional windows). There exist also applications which always create a subprocess, even for the main GUI.

For such applications, you have to "connect to the screen", and provide an additional window-reference in the XPath locator.
<br>You must also use a screen connection, if your test contains interactions between multiple top-level windows, for example a drag&drop between windows.


== Connecting to an Application ==
== Connecting to an Application ==
To automate an application, you first have to connect to it. You can do this with the
To automate an application, you first have to connect to it or to its screen. You can do this with the
blocks from the '''Connecting''' group.
blocks from the "''Connecting''" group.
[[Bild:WindowsAutomation2_Library_ConnectingBlocks.PNG|frame|The Library's Connection Blocks.]]
[[Bild:WindowsAutomation2_Library_ConnectingBlocks.PNG|frame|The Library's Connection Blocks (old picture without screen connection blocks).]]

You can connect to an individual application by its ''executable name'', or by its ''process id''.
If you use the executable name, expecco will attach to the main process of the specified executable.
Be careful as this is not necessarily the process drawing the GUI!
Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process.


All connection blocks also require you to specify a ''name'' by which the connection should be known in future.
You can connect to an application by its '''executable name''', or by its '''process id'''.
This is needed when you automate multiple applications at the same time and need to change
If you use the executable name, expecco will attach to the main process of the specified executable. Be careful as this is not necessarily the process
''connection contexts'' during test execution.
drawing the GUI! Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process.
All connection blocks also require you to specify a '''name''' by which the connection should be known in future.
This is needed for when you automate multiple applications at the same time and need to change
'''connection contexts''' during test execution.
Please note that the application to connect to has to be started before the corresponding connection block is executed.
Please note that the application to connect to has to be started before the corresponding connection block is executed.
You can change in context of which connection a block is executed with the '''Set Active Connection''' block.
You can change in context of which connection a block is executed with the [''Set Active Connection''] block.
This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.
This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.


=== Establishing a Local Connection ===
=== Establishing a Local Connection ===
If you want to automate an application running on the same host as expecco, there is no need to set up anything! Just use the '''Create Local Connection''' with the mentioned above parameters.
To automate an application running on the same host as expecco, there is no need to set up anything.
Use "[''Create Local Connection'']" with the above mentioned parameters.


=== Establishing a Remote Connection ===
=== Establishing a Remote Connection ===
Make sure that the DotNet Bridge server is started as explained in [[#Preparing a remote system for test execuction|Preparing a remote system for test execuction]]
Make sure that the DotNet-Bridge server ("<code>DotNetBridgeServer.exe</code>") is running on the remote system, as explained in the chapter [[#Preparing a remote system for test execuction|"''Preparing a remote system for test execuction''"]]
When the bridge is running, you can connect to it with the '''Create Remote Connection''' block using the hostname and IP-Address you specified when you started the bridge.
When the bridge is running, you can connect to it with the "[''Create Remote Connection'']" block using the hostname and IP-Address you specified when you started the bridge.

Notice, that the DotNet-Bridge server only accepts local connections by default.
This is done for security reasons. Use the "-a x.x.x.x" command line argument to specify, which hosts are allowed
to connect. With "-a 0.0.0.0", any host is allowed. Do NOT use this, if the remote host is reachable from untrusted hosts.


=== Closing a Connection ===
=== Closing a Connection ===
Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources.
Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources.
One simple way to do this is to setup your connections in the '''Before Execution''' part of the test suite
One simple way to do this is to setup your connection in the "''Before Execution''" (pre-action) part of the test suite or test case,
and put the '''Close All Connections''' block in the '''After exeuction''' part.
and put the "[''Close All Connections'']" action block in the "''After exeuction''" (post-action) part.

<br clear=all>


<br clear=all>
== Accessing GUI Elements ==
== Accessing GUI Elements ==
[[Bild:WindowsAutomation_Library_AccessBlocks.PNG|frame|The Library's GUI Element Access Blocks.]]
[[Bild:WindowsAutomation_Library_AccessBlocks.PNG|frame|The Library's GUI Element Access Blocks.]]
[[Bild:WindowsAutomation_Library_AccessBlocksExamples.PNG|frame|A Few Examples on How to Access Elements in Notepad.]]
[[Bild:WindowsAutomation_Library_AccessBlocksExamples.PNG|frame|A Few Examples on How to Access Elements in Notepad.]]
[[Bild:WindowsAutomation_Library_AccessBlocksModes.PNG|frame|Different Ways to Reference a GUI Element.]]
[[Bild:WindowsAutomation_Library_AccessBlocksModes.PNG|frame|Different Ways to Reference a GUI Element.]]
All Windows applications are built as a ''' tree''' of ''' GUI elements''' .
All Windows applications are built as a '' tree'' of '' GUI elements''.
Some elements act as container of other element - their children. Tables, for example, are containers
Some elements act as container of other element - their children. Tables, for example, are containers
for their cell elements which themselves might contain text labels.
for their cell elements which themselves might contain text labels.
At the root of the tree are one or more '''windows''' . When you connect to an application, you get access to
At the root of the tree are one or more ''windows'' . When you connect to an application, you get access to
all windows at the tree's root.
all windows at the tree's root.
To manipulate any of the system under test's GUI element, you first have to gain access to it.
To manipulate any of the system under test's GUI element, you first have to gain access to it.
The group '''GUI Element Access''' provides blocks that help you to gain access to the elements. There are 12 blocks in total, divided between ''' Access By XPath''' , '''Access By Property''',
The group named "''GUI Element Access''" provides action blocks to access elements. The actions are further categorized into sub-groups named "'' Access By XPath''" , "''Access By Property''", "''Existence Test''" etc.
and '''Existence Test'''. For each category, blocks with the same interface exist. The only difference is the method of retrival.
For each category, blocks with the same interface exist. The only difference is the method of retrieval.


;Get GUI Element : returns a '''single''' GUI Element matching the search condition. If multiple match the condition, an '''arbitrary''' element matching the condition is returned.
;Get GUI Element : returns a ''single'' GUI element matching the search condition. If multiple elements match the condition, an ''arbitrary'' element matching the condition is returned.
:If you want to make sure that there is only one element matching, set '''ensureExclusive''' to true. You can also enforce this globally by setting the environment variable '''ensureExclusive''' in your environment.
:To make sure that there is only one single element matching, set ''ensureExclusive'' to true. You can also enforce this globally by setting the environment variable "''ensureExclusive''" in your environment.
:If ensureExclusive is set and more than one element is found, the block will fail and notify you of the ambiguity. This is disabled by default as checking for uniqueness is a very costly operation.
:If ensureExclusive is set and more than one element is found, the block will fail and notify about the ambiguity. This is disabled by default, as checking for uniqueness is a very costly (ie. slow) operation.
:You can also specify a '''timeout''' after which the search is aborted. This is useful if you know an element should be there soon but might currently be loading.
:You can also specify a ''timeout'' after which the search is aborted. This is useful if you know an element should be there already (or soon but might currently be loading).


;Get GUI Elements: returns '''all''' GUI Element matching the search condition. If no element is found, it returns an empty list <b> Timeout </b> and <b>cacheXPath</b> are also applicable.
;Get GUI Elements: returns ''all'' GUI elements matching the search condition. If no element is found, it returns an empty list ''Timeout'' and ''cacheXPath'' are also applicable.


;Exists: Works exactly the same as '''Get GUI Element''' but does not fail if no element can be found. Additionally provides an output pin specifying if an element has been found.
;Exists: Works exactly the same as ''Get GUI Element'' but does not fail if no element is found. Additionally provides an output pin specifying if an element has been found.


Both of these blocks are also available in the '''Relative To Anchor''' variety. Those blocks also take a GUI Element as '''anchor''' in relation to which they search.
Both of these blocks are also available in the "''Relative To Anchor''" variety. Those blocks also take a GUI Element as ''anchor'' in relation to which they search.




=== Using XPaths ===
=== Using XPaths ===


'''XPath''' is the query language this plugin uses for selecting GUI elements in the application's element tree.
'''XPath''' is the query language used to select GUI elements in the application's element tree.
You can find the XPath of an element with the [[Expecco GUI Tests Extension Reference|Gui Browser]] or with third-party applications like Inspect.exe [https://msdn.microsoft.com/en-us/library/dd318521(v=vs.85).aspx] or FlaUInspect [https://github.com/FlauTech/FlaUInspect].
You can find the XPath of an element with the [[Expecco GUI Tests Extension Reference/en|Gui Browser]] or with third-party applications like "Inspect.exe" [https://msdn.microsoft.com/en-us/library/dd318521(v=vs.85).aspx] or "FlaUInspect" [https://github.com/FlauTech/FlaUInspect].
An '''XPath''' for the scroll button on the
<br>As an example, the full ''XPath'' of the scroll button on the
vertical scrollbar in the german Windows Notepad application looks like this:
vertical scrollbar in the german Windows "Notepad.exe" application looks like this:


<code>/Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten'] </code>
<code>/Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten'] </code>


For each level in the GUI tree, a new <b>Location Step</b> containing the child elements <b>Control Type</b> has to be added.
For each level in the GUI tree, a new ''Location Step'' containing the child element's ''Control Type'' was added in the above XPath.
You do not have to use any predicates, the following XPath is valid as well:
<br>You do not have to use any predicates, the following XPath is valid as well:


<code>/Window/Document/ScrollBar/Button </code>
<code>/Window/Document/ScrollBar/Button </code>


but might lead to issues as there might be multiple GUI Elements matching this path.
but might lead to issues if multiple GUI Elements match this path.
You can select by three predicates that can be mixed and matched in the same query:


You can select by three predicates (which can be mixed and matched in the same query):
; Name: the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file.

; Name: the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file).
: <code>/Window[@Name='Unbenannt - Editor'] </code>
: <code>/Window[@Name='Unbenannt - Editor'] </code>


; AutomationId: An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced.
; AutomationId: An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced on lazy or incompetent programmers.
: Many developersleave the ID empty or reuse it, making it useless. You should nontheless use this ID wherever applicable as it is static
: Many lazy developers leave the ID empty or reuse it, making it useless.
: and meant for this exact purpose.
: You should use this ID wherever applicable as it is static and meant for this exact purpose.
: <code>/Window/Document[@AutomationId='15'] </code>
: <code>/Window/Document[@AutomationId='15'] </code>


Zeile 165: Zeile 206:
: <code>/Window[1]/Document[last()] </code>
: <code>/Window[1]/Document[last()] </code>


=== Warning there are XPath Limitations ===
'''Please Note''': You can '''only''' use the three predicates above in your XPaths. Other predicates '''DO NOT''' work! If you want to search
'''Please Note''': You can ONLY use the three predicates above in your XPaths.
an element by another property (e.g. its help text), use the '''By Property''' variant of the block instead. If the property is not unique,
<br>Other predicates '''DO NOT''' work!
you can select a unique ancestor of the target by property or XPath relative to which the property is unique.
You can then use that ancestor as anchor in the '''Get GUI Element By Property (Relative to Anchor)''' block.
<br>To search for an element by another property (e.g. its help text), use the '''By Property''' variant of the block instead.
If the property is not unique, you can select a unique ancestor of the target by property or XPath relative to which the property is unique.
You can then use that ancestor as anchor in the [''Get GUI Element By Property (Relative to Anchor)''] block.


=== Using Wildcards in XPaths ===
=== Using Wildcards in XPaths ===
Zeile 174: Zeile 217:
XPath also supports '''wildcards'''. You can:
XPath also supports '''wildcards'''. You can:
* replace '''one tree level''' with a "*" wildcard:
* replace '''one tree level''' with a "*" wildcard:
<code>/Window/Document[@AutomationId='15']/*/Button[1]</code> <br />
: <code>/Window/Document[@AutomationId='15']/*/Button[1]</code> <br />
<code>/*/*/*/Button[1] </code>
: <code>/*/*/*/Button[1] </code>


* replace a '''control type''' with a "*" wildcard:
* replace a '''control type''' with a "*" wildcard:
<code>/Window/*[@AutomationId='15'] </code>
: <code>/Window/*[@AutomationId='15'] </code>


* replace '''an arbitrary amount of levels''' with a "//" wildcard:
* replace '''an arbitrary amount of levels''' with a "//" wildcard:
<code>//Button </code> <br />
: <code>//Button </code> <br />
<code>/Window//Button[1] </code>
: <code>/Window//Button[1] </code>


Especially the last option is useful as it makes your tests <b>more readable</b> as the reader can easily focus on the relevant parts of the xpath.
Especially the last option is useful as it makes your tests more readable, in that the reader can easily focus on the relevant parts of the xpath.

But be careful. Using wildcards comes at a steep <b>performance cost</b>. If your tests run unacceptably slow, first try to '''remove wildcards where they are not needed'''.
But be careful: using wildcards comes at a steep performance cost (due to the implementation inside the Windows API).
If your test runs unacceptably slow, first try to remove wildcards where they are not needed.


=== Using Properties===
=== Using Properties===


You can also search for elements by property by specifying a value an element's property with the given name must have. For a list of properties a GUI element might have, take a look at Microsoft's documentation [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview#control-pattern-classes-and-interfaces]. It specifies the supported properties for each pattern an element can support.
You can also search for elements by property; i.e. by specifying a value an element's property with the given name must have.

For a list of properties a GUI element might have, take a look at Microsoft's documentation [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview#control-pattern-classes-and-interfaces].


=== XPath Resolution in Action Blocks ===
=== XPath Resolution in Action Blocks ===


Xpath resolution may be slow in the Windows Automation API, depending on the number of windows/elements which have to be checked.
Getting a reference to the GUI Element everytime you need to manipulate it in any way can make for very unclean looking test suites as two blocks are needed to perform any action - one for getting a reference to the element and another one for perform the action. The library therefore also allows you to directly feed an xpath into every block that would take a reference to a GUI Element.

To speed up test execution, you should reuse a resolved XPath if possible.
For this, use the "Get Element by XPath" action, and feed the "element handle" to multiple actions.
All action- and property access blocks can take either an XPath or an element handle as locator input.


The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent.
The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent.
Zeile 199: Zeile 250:
You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.
You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.


'''Rule of thumb''': Always use Variant 1 except if you need the reference to the GUI element only once.
'''Rule of thumb''': Always use Variant 1 except if you need the reference to the GUI element only once, or if the application replaces the element dynamically.


=== Separating Path Definitions ===
=== Separating Path Definitions ===


It is highly recommended to add definitions for element-lathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:
It is highly recommended to add definitions for element-pathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:


* the paths are hidden and the values at the steps become more readable
* the paths are hidden and the values at the steps become more readable
* your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.
* your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.
* you can import those definitions from a CSV or XML file, and thus separate the locators from the actual test suite. In case of changed UI-layout / UI-hierarchy after a revision change of the tested application, all you need is a new excel or xml file containing those definitions.


If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. cvs or excel or xml file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.
If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. CVS or XML file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.


<br clear=all>
<br clear=all>
Zeile 221: Zeile 273:
=== The 'Elements' Group ===
=== The 'Elements' Group ===


This group contains blocks that are applicable to blocks that only work on '''GUI elements of a specific type'''. For a block's detailed description with examples ,
This group contains blocks that are applicable to blocks that only work on GUI elements of a specific type. For a block's detailed description with examples ,
take a look at the block's documentation in expecco.
take a look at the block's documentation in expecco.


Even though not only Checkboxes can be 'checked', the '''Check''' block in the '''Checkbox group''' only works for checkboxes and not e.g. for radio buttons.
Even though not only Checkboxes can be 'checked', the [''Check''] block in the "''Checkbox group''" only works for checkboxes and not e.g. for radio buttons.
Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific '''pattern''' instead and are therefore in the '''Patterns''' group.
Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific ''pattern'' instead and are therefore in the "''Patterns''" group.


=== The 'Patterns' Group ===
=== The 'Patterns' Group ===


This group contains blocks that are applicable to GUI elements supporting a certain '''pattern'''. '''Patterns''' describe how a GUI element '''behaves''' rather than what it '''is'''. An element can also support multiple patterns
This group contains blocks that are applicable to GUI elements supporting a certain ''pattern''.
Patterns describe how a GUI element ''behaves'' rather than what it ''is''. An element can also support multiple patterns at the same time.
at the same time. You can find out which patterns an element supports with the '''Get Supported Patterns''' block or by taking a look at Microsoft's documentation about patterns [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview].
You can find out which patterns an element supports with the [''Get Supported Patterns''] block or by taking a look at Microsoft's documentation about patterns [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview].


Pattern blocks can therefore be '''re-used''' between GUI elements of different types. The '''Expand''' block of the '''ExpandCollapse pattern''', for example, can be used to expand Comboboxes, Menus, Submenus, or Trees.
Pattern blocks can therefore be ''re-used'' between GUI elements of different types.
For example, the [''Expand''] block in "''ExpandCollapse pattern''", can be used to expand Comboboxes, Menus, Submenus, or Trees.


=== The 'Properties' Group ===
=== The 'Properties' Group ===


Blocks in the '''Patterns''' and '''Elements''' group already allow you to query the value of properties pertaining to that element or pattern. Blocks of the '''Properties''' group allow you to query properties all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.
Blocks in the "''Patterns''" and "''Elements''" group already allow you to query the value of properties pertaining to that element or pattern. Blocks in the "''Properties''" group allow you to query both properties which all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.


Please note that the '''Get Arbitrary Property''' block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview#control-pattern-classes-and-interfaces]
Please note that the [''Get Arbitrary Property''] block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide [https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-control-patterns-overview#control-pattern-classes-and-interfaces]


<br clear=all>
<br clear=all>
Zeile 245: Zeile 299:
[[Bild:WindowsAutomation_Library_KeyboardMouse.png|400px|The Library's Blocks for Interacting With Keyboard and Mouse.]]
[[Bild:WindowsAutomation_Library_KeyboardMouse.png|400px|The Library's Blocks for Interacting With Keyboard and Mouse.]]


You can control keyboard and mouse with the blocks in the groups '''Keyboard''' and '''Mouse''' that can be seen on the right. In contrast to the blocks above which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.
You can control keyboard and mouse with action blocks in the "''Keyboard''" and "''Mouse''" groups, as shown on the right. In contrast to the blocks above, which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.


You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the '''Left/Double/Right Click On Element''' blocks for this).
You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the [''Left/Double/Right Click On Element''] blocks for this).
If you do not want to wait for the mouse to move, you can also use the '''Set Focus''' block instead.
If you do not want to wait for the mouse to move, you can also use the [''Set Focus''] block instead.


=== Keyboard ===
=== Keyboard ===
The plugin provides blocks interacting with the keyboard on different abstraction levels. The most versatile block is '''Type With Control Keys Pressed''' where you can specify a string of characters to be typed while any number of control keys (e.g. esc, alt, shift, ctrl, ...) are held down. All other blocks are specializations of this block.
The plugin provides blocks to interact with the keyboard on different abstraction levels. The most versatile block is [''Type With Control Keys Pressed''] where you can specify a string of characters to be typed while any number of control keys (e.g. "esc", "alt", "shift", "ctrl", ...) are held down. All other blocks are specializations of this block.


'''Press Control Keys''' e.g. allows you to enter key commands such as renaming files (ALT+F2), starting the task manager (CTRL+ALT+DEL), or closing a window (ALT+F4). '''Press Control Key''' makes it easy to just press a single key (e.g. RETURN to start a new line in an edit box). The '''Shortcuts''' group provides an assorted collection of often used key combinations for your convenience.
[''Press Control Keys''] e.g. allows you to enter key commands such as renaming files ("ALT+F2"), starting the task manager ("CTRL+ALT+DEL"), or closing a window ("ALT+F4"). [''Press Control Key''] makes it easy to just press a single key (e.g. "RETURN" to start a new line in an edit box). The "''Shortcuts''" group provides an assorted collection of often used key combinations for your convenience.


Use the '''Type''' to just type some text or, if you just need some throwaway demo text, use '''Type Random Phrase'''.
Use the [''Type''] action to just type some text, or if you just need some throwaway demo text, use [''Type Random Phrase''].


If you need more fine grained control over the keyboard, you can make use of '''Scan Codes'''. The keyboard sends a scan codeto the operating system whenever a key is pressed or released.
If you need more fine grained control over the keyboard, you can make use of ''Scan Codes''. The keyboard sends a scan code to the operating system whenever a key is pressed or released.
Microsoft provides a list of the scan codes it supports [https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-6.0/aa299374(v=vs.60)].
Microsoft provides a list of the scan codes it supports [https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-6.0/aa299374(v=vs.60)].
'''Press Key By Scan Code''' sends Windows a signal that a key with a given scan code has been pressed (and is held down). The matching '''Release Key By Scan Code''' sends Windows the adverse signal.
[''Press Key By Scan Code''] sends a signal to Windows that a key with a given scan code has been pressed (and is held down). The matching [''Release Key By Scan Code''] sends Windows the adverse signal.
Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use '''Type Key By Scan Code'''.
Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use [''Type Key By Scan Code''].


=== Mouse ===
=== Mouse ===
The plugin provides blocks manipulating the mouse in three ways: '''Clicking''', '''Moving''', and interacting with its '''Buttons'''.
The plugin provides blocks manipulating the mouse in three ways: ''Clicking'', ''Moving'', and interacting with its ''Buttons''.




Zeile 269: Zeile 323:


== Recording and Taking Screenshots ==
== Recording and Taking Screenshots ==



= Frequently Asked Questions =
= Frequently Asked Questions =

== Some elements are not presented in the UI browser's tree ==
This happens if the application "draws" elements (such as checkboxes, buttons or text) without using Windows widgets and without providing proper automation support (btw: which a well behaved application should, because this also affects/prevents usability and makes these controls inaccessable to eg. users with vision disabilities).
We encountered such in table-views containing checkboxes (probably OLE embedded database-tables).
These elements must be addressed either via coordinates (relative to the containing parent element), or via an image search (grab and define the image in the GUI browser). Both have disadvantages and are inconvenient, but there is no other technical solution.


== What is the difference between Click/Invoke/Select/Toggle? ==
== What is the difference between Click/Invoke/Select/Toggle? ==
Click sends a click event; Invoke calls directly into the application's or widget's event handler; Select and Toggle usually call the corresponding automation functions (if the app-programmer provided that).


A click generates a click event to the widget; may disfunction if the widget is invisible or covered (eg. by a tooltip).
== My element supports a certain pattern but blocks of that pattern don't work? ==

An invoke skips over the event mechanism and calls the event hander directly. It may also not work, if the event handler insists on keyboard focus, mouse position, shift- or ctrl-key or other state.

There exists no mechanism which work with any application: for some, you will have to record a mouse-move and/or a set-focus for the click to work.
If possible, disable tooltips and other automatically appearing popups in the target app.

If none works, try the click at screen position action (which generates a real mouse-click, not via UI-automation to the widget, but via the Windows Event Queue). This should work in any case, but suffers most from popups, mouse position and keyboard focus.

== My element supports a certain pattern but blocks of that pattern don't work ==


== I have designed custom elements, how can I automate them? ==
== I have designed custom elements, how can I automate them? ==


== I have extended my GUI elements to have custom properties, how can I query them? ==
== I have extended my GUI elements to have custom properties, how can I query them? ==

== Hotkey "Get hovered element" selects the wrong element in the tree ==

Sometimes there are elements which overlap or hide other elements. This element is then selected in the tree. In this case, you can use the Hotkey "Get focused element" to select the element in the tree

== DevExpress controls are not visible in the GUI Browser ==

DevExpress has some possibilities to disable UIAutomation support, which is also used by expecco.If you can't see one or some controls, please check e.g. https://supportcenter.devexpress.com/ticket/details/bc4069/devexpress-controls-now-clear-ui-automation-peers-for-the-application and other information on the DevExpress Support Site.

== Connect fails with "Permission Denied" ==
Some apps required admin access rights to connect to.
You will get a "failed to connect" error and a walkback which looks like:

Could not connect to application: wireguard
System.ComponentModel.Win32Exception (0x80004005): Access denied
bei System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)
bei System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId, Boolean firstModuleOnly)
bei System.Diagnostics.NtProcessManager.GetFirstModuleInfo(Int32 processId)
bei System.Diagnostics.Process.get_MainModule()
bei FlaUI.Core.WindowsAPI.WindowsApiTools.GetMainModuleFilepath(Process process)
bei FlaUI.Core.Application.Attach(Process process)
bei WindowsAutomation.Application..ctor(String name, UIA3Automation automation, PropertyRepository propertyRepository, String windowTitle)
bei WindowsAutomation.Client.GetApplication(String name, String windowTitle)

You have two options:
* run expecco as admin
* start a dotNet-bridge manually as admin (in the local or remote machine):
...\bridgeFramework\dotNetBridge\dotNetBin>DotNetBridgeServer.exe -k -s -a 0.0.0.0 -p 34319
: then create a "remote connection" to it.

Aktuelle Version vom 29. Januar 2023, 15:38 Uhr

This is the documentation for the Windows Automation 2.0 plugin. It provides facilities to automate tests incorporating applications with a GUI based on Windows Presentation Foundation (WPF), WinForms, and the old Win32 API.

Notice:
The vsn2 replaces the previous vsn1 plugin, which is no longer maintained. The change was necessary due to changes in the Windows operating system, and features which were discontinued by Microsoft. The vsn2 plugin uses Microsoft's UI-Automation2 API to interact with the application. This API was not avaliable in Windows-XP, Vista and other older OS versions, and many features used by the previous version are no longer supported in newer OS versions, or need different parameters.


Inhaltsverzeichnis

Features[Bearbeiten]

  • Automated GUI testing of WPF, WinForms, and Win32 applications.
  • Contains an expecco block library and tools which help to model tests and inspect the GUI of the application.
  • Parallel remote test-execution on multiple systems.
  • Performs tests against already built executables. No source code changes / recompilation of the application is required.
  • Access to GUI components using XPath locators.
  • Access objects of the application live at execution time.
  • Full access to keyboard and mouse of the target system.
  • Lots of additional features such as screen capturing, taking screenshots, highlighting GUI elements, mouse tracking, ...

Known issues in the current Release 19.1.0[Bearbeiten]

  • The WindowsAutomation2.ets library has got a new functional id. When reimporting the library in existing Pre-19.1.0 test suites (which is mandatory), a warning about a changed functional id is shown once. It is ok to accept this dialog.
  • Instant follow mouse is very expensive in Windows and is therefore disabled. Use HotKeys to select a GUI element in the tree at the current pointer position or to update the element tree. HotKeys can be configured in the settings. The HotKey-service has to be started. Take care that the keys have not already been grabbed by another application. If that is the case, a warning will be shown, and you should either make sure that expecco is started before the other application, or that you use other hotkey combinations (in either expecco or the application).

Limitations[Bearbeiten]

  • WindowsAutomation only supports XPath1.0.
That means for example, that the "ends-with" function, which is an XPath2 function, is not supported and thus not possible as XPath attribute filter.
See [XPath vs. XPath2] for details. For some (especially "ends-with") there are workarounds by comparing against a substring which is based on the length of the attribute string or a regex match (see [1]).
Notice, that this is a limitation of the Windows operating system, which cannot be worked around in expecco.

Installation and Connection[Bearbeiten]

The WindowsAutomation testing plugin uses the expecoo ".NET-Bridge" which consists of two parts: A plugin for expecco, and a server executable using the .NET framework running on the remote computer to connect to.

Requirements[Bearbeiten]

On the machine running the system under test:

  • .NET Framework 4.7 [2] or higher.
  • One of the following operation systems:
- Windows 7 SP1
- Windows 8
- Windows 8.1
- Windows 10
- Windows 11
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
- Windows Server 2022

Plugin Components[Bearbeiten]

The plugin consists of the following parts:

  • The GUI Browser, used to interactively explore your application.
  • The Windows Automation Library, which provides blocks that you can use in your test cases.
  • The .NET-Bridge server, which provides an interface between the application under test and expecco.

Installing and Configuring the Plugin in Expecco[Bearbeiten]

The plugin is usually installed automatically by the installer; either included in the main expecco installation or provided as a separate installable add-on package. Its files are stored in the "plugin" subfolder of the expecco installation folder. You can also copy those files manually to that folder, if required. Expecco detects the plugin automatically during startup. You may have to restart your expecco session, if the plugin is installed later.

Preparing a local system for test execution[Bearbeiten]

To automate applications running on the SAME host as expecco, there is no need to set up anything - it should work out of the box.

Try to connect to an application via the GUIBrowser's WindowsAutomation-connect dialog:

WindowsAutomation2Connect.png

Preparing a remote system for test execuction[Bearbeiten]

Security Considerations[Bearbeiten]

Please make sure, that remote access is only possible from inside a secure local network. Under no circumstances should you grant access to the DotNetBridgeServer from untrusted hosts.

DotNet Bridge Server[Bearbeiten]

The remote system has to run the ".Net-Bridge server" which in turn either actively connects to expecco or passively waits for an incoming connection from expecco.

For this, copy "DotNetBridgeServer.exe" and "DotNetBridge.dll" (found in the "packages/exept/bridgeFramework/dotNetBridge/dotNetBin" folder under the expecco installation directory) to the computer where your system under test is running (you can copy it into any folder). No further installation is needed.

To start the bridge, navigate to the executable in a shell/cmd/console and start it with the following arguments ("DotNetBridgeServer.exe --help" outputs a list of valid options):

Arguments Required Description
-s Yes "server mode". Tells the executable to run in server mode (it will wait for expecco to connect to it, as opposed to client mode,
where it actively connects to the partner).
-k Yes "keepalive". Tells the client to keep the connection open after a connection has been terminated.
This saves you the hassle to restart it every time a test finishes.
-a <ip/hostname> Yes "IP address accepting connections". The IP-address or hostname of an interface on the host on which connections will be accepted (in server mode) or to which it connects (in client mode).
If you have multiple network interfaces, you can ensure that only incoming connections on the secure network will be accepted by setting the IP address of this interface.
To allow incoming connections on any network interface (i.e. to have it listen on any IP-address), set it to 0.0.0.0. If you are not inside a secure network, you should use a secure tunnel (e.g. a SSH or Powershell tunnel) between the machine running expecco and the machine running the bridge and listen only on 127.0.0.1 (the default).
-p <int> No "port". The port the server should listen on (in server mode) or to which is should connect (default is 34318).
-L <string> No "assembly". Load a .NET assembly on startup. Omit the .dll extension (so use "WindowsAutomation" but not "WindowsAutomation.dll").
You can repeat this option to load multiple assemblies.

Note: this is not normally needed, as expecco loads the required assemblies automatically into the bridge.
However, if security settings forbid external assemblies, you have to copy them to the target machine, and load them via command line option.

So you may execute on the remote host:

   DotNetBridgeServer.exe -s -k -a 0.0.0.0

For automatic tests, you must setup the target system for the bridge server to be running; either by adding it to some "auto exec' script, or by adding to the "scheduled tasks", or by remote starting it from your test-host via a powershell script, remote ssh command or similar.

If you get an "Assembly not Found" error, you may have an incompatible .NET framework or the "DotNetBridge.dll" is missing. If you get a network error, or the expecco-host cannot connect, make sure that your firewall settings allow access to the given ports (you may try an alternative port like 8080, which must then be specified on both sides).

Settings[Bearbeiten]

There are currently no configurable settings.

GUI Browser[Bearbeiten]

The GUI browser is used to explore a running application. You can browse the application, inspect element properties and execute actions. See " Expecco GUI Tests Extension Reference" for details.

Library[Bearbeiten]

This article gives a short overview over the design philosophy of the library. You can find more detailed documentation and examples attached to the blocks itself.

Connect to Application vs. Connect to Screen[Bearbeiten]

You can either "connect to a single application", or "connect to the screen". The first provides faster access to the components, but will not work, if the app creates subprocesses with a GUI (eg. for dialogs or additional windows). There exist also applications which always create a subprocess, even for the main GUI.

For such applications, you have to "connect to the screen", and provide an additional window-reference in the XPath locator.
You must also use a screen connection, if your test contains interactions between multiple top-level windows, for example a drag&drop between windows.

Connecting to an Application[Bearbeiten]

To automate an application, you first have to connect to it or to its screen. You can do this with the blocks from the "Connecting" group.

The Library's Connection Blocks (old picture without screen connection blocks).

You can connect to an individual application by its executable name, or by its process id. If you use the executable name, expecco will attach to the main process of the specified executable. Be careful as this is not necessarily the process drawing the GUI! Use the process id if an application spawns more than one process and you need to make sure you connect to the GUI process and not some background process.

All connection blocks also require you to specify a name by which the connection should be known in future. This is needed when you automate multiple applications at the same time and need to change connection contexts during test execution. Please note that the application to connect to has to be started before the corresponding connection block is executed. You can change in context of which connection a block is executed with the [Set Active Connection] block. This allows you, for example, to automate and compare the state of applications running on multiple systems in one test case.

Establishing a Local Connection[Bearbeiten]

To automate an application running on the same host as expecco, there is no need to set up anything. Use "[Create Local Connection]" with the above mentioned parameters.

Establishing a Remote Connection[Bearbeiten]

Make sure that the DotNet-Bridge server ("DotNetBridgeServer.exe") is running on the remote system, as explained in the chapter "Preparing a remote system for test execuction" When the bridge is running, you can connect to it with the "[Create Remote Connection]" block using the hostname and IP-Address you specified when you started the bridge.

Notice, that the DotNet-Bridge server only accepts local connections by default. This is done for security reasons. Use the "-a x.x.x.x" command line argument to specify, which hosts are allowed to connect. With "-a 0.0.0.0", any host is allowed. Do NOT use this, if the remote host is reachable from untrusted hosts.

Closing a Connection[Bearbeiten]

Remember to always close a connection when it is no longer needed, as it otherwise needlessly consumes resources. One simple way to do this is to setup your connection in the "Before Execution" (pre-action) part of the test suite or test case, and put the "[Close All Connections]" action block in the "After exeuction" (post-action) part.


Accessing GUI Elements[Bearbeiten]

The Library's GUI Element Access Blocks.
A Few Examples on How to Access Elements in Notepad.
Different Ways to Reference a GUI Element.

All Windows applications are built as a tree of GUI elements. Some elements act as container of other element - their children. Tables, for example, are containers for their cell elements which themselves might contain text labels. At the root of the tree are one or more windows . When you connect to an application, you get access to all windows at the tree's root. To manipulate any of the system under test's GUI element, you first have to gain access to it. The group named "GUI Element Access" provides action blocks to access elements. The actions are further categorized into sub-groups named " Access By XPath" , "Access By Property", "Existence Test" etc. For each category, blocks with the same interface exist. The only difference is the method of retrieval.

Get GUI Element 
returns a single GUI element matching the search condition. If multiple elements match the condition, an arbitrary element matching the condition is returned.
To make sure that there is only one single element matching, set ensureExclusive to true. You can also enforce this globally by setting the environment variable "ensureExclusive" in your environment.
If ensureExclusive is set and more than one element is found, the block will fail and notify about the ambiguity. This is disabled by default, as checking for uniqueness is a very costly (ie. slow) operation.
You can also specify a timeout after which the search is aborted. This is useful if you know an element should be there already (or soon but might currently be loading).
Get GUI Elements
returns all GUI elements matching the search condition. If no element is found, it returns an empty list Timeout and cacheXPath are also applicable.
Exists
Works exactly the same as Get GUI Element but does not fail if no element is found. Additionally provides an output pin specifying if an element has been found.

Both of these blocks are also available in the "Relative To Anchor" variety. Those blocks also take a GUI Element as anchor in relation to which they search.


Using XPaths[Bearbeiten]

XPath is the query language used to select GUI elements in the application's element tree. You can find the XPath of an element with the Gui Browser or with third-party applications like "Inspect.exe" [3] or "FlaUInspect" [4].
As an example, the full XPath of the scroll button on the vertical scrollbar in the german Windows "Notepad.exe" application looks like this:

 /Window[@Name='Unbenannt - Editor']/Document[@Name='Text-Editor']/ScrollBar[@Name='Vertikale Bildlaufleiste']/Button[@Name='Bildlauf nach unten'] 

For each level in the GUI tree, a new Location Step containing the child element's Control Type was added in the above XPath.
You do not have to use any predicates, the following XPath is valid as well:

 /Window/Document/ScrollBar/Button 

but might lead to issues if multiple GUI Elements match this path.

You can select by three predicates (which can be mixed and matched in the same query):

Name
the name the application developer has given an element. Be careful as this name might change during programm execution (See for example Notepad's window name which always begins with the name of the currently opened file).
/Window[@Name='Unbenannt - Editor']
AutomationId
An ID the application developer has given an element. It is meant to be unique between siblings but that is not enforced on lazy or incompetent programmers.
Many lazy developers leave the ID empty or reuse it, making it useless.
You should use this ID wherever applicable as it is static and meant for this exact purpose.
/Window/Document[@AutomationId='15']
List Index
If there are multiple siblings with the same Control Type, you can select one of them by (1-based) list index.
/Window[1]/Document[1]
/Window[1]/Document[last()]

Warning there are XPath Limitations[Bearbeiten]

Please Note: You can ONLY use the three predicates above in your XPaths.
Other predicates DO NOT work!
To search for an element by another property (e.g. its help text), use the By Property variant of the block instead. If the property is not unique, you can select a unique ancestor of the target by property or XPath relative to which the property is unique. You can then use that ancestor as anchor in the [Get GUI Element By Property (Relative to Anchor)] block.

Using Wildcards in XPaths[Bearbeiten]

XPath also supports wildcards. You can:

  • replace one tree level with a "*" wildcard:
/Window/Document[@AutomationId='15']/*/Button[1]
/*/*/*/Button[1]
  • replace a control type with a "*" wildcard:
/Window/*[@AutomationId='15']
  • replace an arbitrary amount of levels with a "//" wildcard:
//Button
/Window//Button[1]

Especially the last option is useful as it makes your tests more readable, in that the reader can easily focus on the relevant parts of the xpath.

But be careful: using wildcards comes at a steep performance cost (due to the implementation inside the Windows API). If your test runs unacceptably slow, first try to remove wildcards where they are not needed.

Using Properties[Bearbeiten]

You can also search for elements by property; i.e. by specifying a value an element's property with the given name must have.

For a list of properties a GUI element might have, take a look at Microsoft's documentation [5].

XPath Resolution in Action Blocks[Bearbeiten]

Xpath resolution may be slow in the Windows Automation API, depending on the number of windows/elements which have to be checked.

To speed up test execution, you should reuse a resolved XPath if possible. For this, use the "Get Element by XPath" action, and feed the "element handle" to multiple actions. All action- and property access blocks can take either an XPath or an element handle as locator input.

The image on the right shows this mode in action. Variants 1 and 2 are functionally equivalent. While Variant 2 is more concise, Variant 1 is the right choice in most cases. While it has an additional block, you have control over the other input parameters. You also can recycle the GUI element reference as seen in 3). This is invaluable as the XPath only has to be resolved once. This speeds up test execution by a lot as XPath resolution is very expensive operation.

Rule of thumb: Always use Variant 1 except if you need the reference to the GUI element only once, or if the application replaces the element dynamically.

Separating Path Definitions[Bearbeiten]

It is highly recommended to add definitions for element-pathes into an environment (typically the project environment), and use "Environment Value" references as input to the steps. This gives two benefits:

  • the paths are hidden and the values at the steps become more readable
  • your tests are easier to maintain: in case of a changed window layout/structure, only the variable definitions need to be changed at one place.
  • you can import those definitions from a CSV or XML file, and thus separate the locators from the actual test suite. In case of changed UI-layout / UI-hierarchy after a revision change of the tested application, all you need is a new excel or xml file containing those definitions.

If the development team of the application can provide the paths of the UI components in a machine readable form (i.e. CVS or XML file), you may even let expecco read and parse those path definitions at test start time, and let it automatically adapt to any change.


Interacting with GUI Elements[Bearbeiten]

Once you retrieved a GUI element, you can interact with it by making use of the library's blocks located in the groups Elements, Patterns, and Properties. Blocks in those categories take a GUI Element (and sometimes additional data) as input and manipulate it or query for its properties in some way.

The Library's Blocks for Interacting With Elements.


The 'Elements' Group[Bearbeiten]

This group contains blocks that are applicable to blocks that only work on GUI elements of a specific type. For a block's detailed description with examples , take a look at the block's documentation in expecco.

Even though not only Checkboxes can be 'checked', the [Check] block in the "Checkbox group" only works for checkboxes and not e.g. for radio buttons. Because of this limitation there are not that many blocks targeting specific elements. Most blocks target a specific pattern instead and are therefore in the "Patterns" group.

The 'Patterns' Group[Bearbeiten]

This group contains blocks that are applicable to GUI elements supporting a certain pattern. Patterns describe how a GUI element behaves rather than what it is. An element can also support multiple patterns at the same time. You can find out which patterns an element supports with the [Get Supported Patterns] block or by taking a look at Microsoft's documentation about patterns [6].

Pattern blocks can therefore be re-used between GUI elements of different types. For example, the [Expand] block in "ExpandCollapse pattern", can be used to expand Comboboxes, Menus, Submenus, or Trees.

The 'Properties' Group[Bearbeiten]

Blocks in the "Patterns" and "Elements" group already allow you to query the value of properties pertaining to that element or pattern. Blocks in the "Properties" group allow you to query both properties which all elements share as well as arbitrary properties. If you try to retrieve a property an element does not support, the block will fail.

Please note that the [Get Arbitrary Property] block can only query for properties that belong to a pattern. You cannot, for example, query "Text" from a textfield as this is not a property, use "Value" of the value pattern instead. Microsoft's documentation has a list of patterns and the properties they provide [7]


Automating Keyboard and Mouse[Bearbeiten]

The Library's Blocks for Interacting With Keyboard and Mouse.

You can control keyboard and mouse with action blocks in the "Keyboard" and "Mouse" groups, as shown on the right. In contrast to the blocks above, which work in context of a GUI element, the keyboard and mouse blocks work on a global level. You therefore do not need to provide a GUI element when using them.

You might want to execute some blocks in context of a GUI elemenet nontheless (e.g. enter a number into a text field instead of just pressing keys while the application is running). You can achieve this by selecting the element with the mouse beforehand as you would do when using the application normally (use the [Left/Double/Right Click On Element] blocks for this). If you do not want to wait for the mouse to move, you can also use the [Set Focus] block instead.

Keyboard[Bearbeiten]

The plugin provides blocks to interact with the keyboard on different abstraction levels. The most versatile block is [Type With Control Keys Pressed] where you can specify a string of characters to be typed while any number of control keys (e.g. "esc", "alt", "shift", "ctrl", ...) are held down. All other blocks are specializations of this block.

[Press Control Keys] e.g. allows you to enter key commands such as renaming files ("ALT+F2"), starting the task manager ("CTRL+ALT+DEL"), or closing a window ("ALT+F4"). [Press Control Key] makes it easy to just press a single key (e.g. "RETURN" to start a new line in an edit box). The "Shortcuts" group provides an assorted collection of often used key combinations for your convenience.

Use the [Type] action to just type some text, or if you just need some throwaway demo text, use [Type Random Phrase].

If you need more fine grained control over the keyboard, you can make use of Scan Codes. The keyboard sends a scan code to the operating system whenever a key is pressed or released. Microsoft provides a list of the scan codes it supports [8]. [Press Key By Scan Code] sends a signal to Windows that a key with a given scan code has been pressed (and is held down). The matching [Release Key By Scan Code] sends Windows the adverse signal. Please note that a key that is pressed, stays pressed until it is explicitly released. If you want to just press and release a key by scan code, use [Type Key By Scan Code].

Mouse[Bearbeiten]

The plugin provides blocks manipulating the mouse in three ways: Clicking, Moving, and interacting with its Buttons.



Recording and Taking Screenshots[Bearbeiten]

Frequently Asked Questions[Bearbeiten]

Some elements are not presented in the UI browser's tree[Bearbeiten]

This happens if the application "draws" elements (such as checkboxes, buttons or text) without using Windows widgets and without providing proper automation support (btw: which a well behaved application should, because this also affects/prevents usability and makes these controls inaccessable to eg. users with vision disabilities). We encountered such in table-views containing checkboxes (probably OLE embedded database-tables). These elements must be addressed either via coordinates (relative to the containing parent element), or via an image search (grab and define the image in the GUI browser). Both have disadvantages and are inconvenient, but there is no other technical solution.

What is the difference between Click/Invoke/Select/Toggle?[Bearbeiten]

Click sends a click event; Invoke calls directly into the application's or widget's event handler; Select and Toggle usually call the corresponding automation functions (if the app-programmer provided that).

A click generates a click event to the widget; may disfunction if the widget is invisible or covered (eg. by a tooltip).

An invoke skips over the event mechanism and calls the event hander directly. It may also not work, if the event handler insists on keyboard focus, mouse position, shift- or ctrl-key or other state.

There exists no mechanism which work with any application: for some, you will have to record a mouse-move and/or a set-focus for the click to work. If possible, disable tooltips and other automatically appearing popups in the target app.

If none works, try the click at screen position action (which generates a real mouse-click, not via UI-automation to the widget, but via the Windows Event Queue). This should work in any case, but suffers most from popups, mouse position and keyboard focus.

My element supports a certain pattern but blocks of that pattern don't work[Bearbeiten]

I have designed custom elements, how can I automate them?[Bearbeiten]

I have extended my GUI elements to have custom properties, how can I query them?[Bearbeiten]

Hotkey "Get hovered element" selects the wrong element in the tree[Bearbeiten]

Sometimes there are elements which overlap or hide other elements. This element is then selected in the tree. In this case, you can use the Hotkey "Get focused element" to select the element in the tree

DevExpress controls are not visible in the GUI Browser[Bearbeiten]

DevExpress has some possibilities to disable UIAutomation support, which is also used by expecco.If you can't see one or some controls, please check e.g. https://supportcenter.devexpress.com/ticket/details/bc4069/devexpress-controls-now-clear-ui-automation-peers-for-the-application and other information on the DevExpress Support Site.

Connect fails with "Permission Denied"[Bearbeiten]

Some apps required admin access rights to connect to. You will get a "failed to connect" error and a walkback which looks like:

Could not connect to application: wireguard
System.ComponentModel.Win32Exception (0x80004005): Access denied
 bei System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)
 bei System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId, Boolean firstModuleOnly)
 bei System.Diagnostics.NtProcessManager.GetFirstModuleInfo(Int32 processId)
 bei System.Diagnostics.Process.get_MainModule()
 bei FlaUI.Core.WindowsAPI.WindowsApiTools.GetMainModuleFilepath(Process process)
 bei FlaUI.Core.Application.Attach(Process process)
 bei WindowsAutomation.Application..ctor(String name, UIA3Automation automation, PropertyRepository propertyRepository, String windowTitle)
 bei WindowsAutomation.Client.GetApplication(String name, String windowTitle)

You have two options:

  • run expecco as admin
  • start a dotNet-bridge manually as admin (in the local or remote machine):
   ...\bridgeFramework\dotNetBridge\dotNetBin>DotNetBridgeServer.exe -k -s -a 0.0.0.0 -p 34319 
then create a "remote connection" to it.



Copyright © 2014-2024 eXept Software AG