Graphical Elements in GUI Technologies/en

Deutsche Version | English Version

Introduction

The GUI technologies generally provide information about available elements inside their applications. By a locator (usually an XPath) you can then define at a block, for which element a certain action shall be executed. But in some cases in can be difficult or even impossible to find such a path. Either because the structure is not static, there are no suitable properties, or there even is no element available by the technology, but only its surrounding element.

By using a screenplay it is possible, to define elements that can be found using the screenshot. The pronciple is the same as for the automation on the local screen or using VNC server, but can be used in parallel with the normal form of addressing elements.

Following technologies support an additional screenplay so far: Web Test (Selenium), Mobile Testing (Appium), WindowsAutomation 2, and Java.

Defining Screenplays

The easiest way to define a screenplay is using the GUI Browser. First connect to the application you want to use. In its context menu you will find then the entry Screenplay with several options, including to create a new screenplay. The elements inside the screenplay are divided into scenes. They can be used for structuring, but you can use just one scene as well. The screenplay will be saved as an attachment to the current project, as well all image files that are created to define the elements.

Using Screenplays

If a screenplay is attached to a connection, it can be used with it. In the context menu of an application in the GUI Browser you can in addition to creating a new screenplay also load an already existing one from an attachment or a file.

To load a screenplay in a test there are appropriate blocks. Some technology libraries contain such a block, in the Common GUI Library, which can be used in any case, it is called [UI] Set Screenplay for example. Place the attachment defining the screenplay to your compound block and set its output type to Contents as String, so you can connect the pin to the screenplay block.

The elements defined in the screenplay can be used with the blocks from the technology libraries or the Common GUI Library, but not all actions may work. Read in the section Notation further below how to write the elements at the input pins.

Elements

There are different kinds of elements or players, that can be defined in a screenplay. In the GUI Browser you can create a new element by selecting the screenplay or a scene and choosing the desired type of element under Define New Element ("Player" in Scene) in the context menu.

Graphical Elements

Graphical elements are all elements that can be found by the screenshot. Elements which are found like that are independent from the used technology. You can only execute basic actions, like click or get text (by OCR). Although these actions are executed using the respective technology, actions that need a true technology element (e.g. get property value) are not possible with these elements. There are different options to define such an element.

Position

The simplest option is to just provide a position. This doesn't even require the screenshot to be searched, but it classifies as graphical element nevertheless, as the possible action with such an element are the same.

Bounds

Define the bounds where an element is located by drawing a rectangle on the preview image. The position is considered relative to the top left corner of the screenshot.

Image Search

Elements can be defined by one or more images, which will then be searched in the current screenshot. There are several subcategories of this. The element can be defined directly by the image or at a position relative to the image. Furthermore it is possible to define an area bordered by two images.

	Area Under Bitmap	Define an image by drawing a rectangle on the preview image. You can crop the image afterwards. The defined element will be located where this image is found.
	Area Under Click (auto)	Same as the previous option, but you only have to click at the element position to define it. expecco then tries to create a useful image, which you can adjust if needed.
	Bitmap Anchored Area	In this case the element is not located where the associated image is found, but at a position relative to it. To define it, first draw a rectangle in the preview image where the element should be, then draw a second rectangle to define the anchor image.
	Area Between Bitmaps Left..Right	The element is delimited by two images that are on the left and the right side of the element. To define it, draw a rectangle around the overall area of the element and the images. After that you can crop the selection to the left and the right image.
	Area Between Bitmaps Top..Bottom	The element is delimited by two images that are at the top and the bottom of the element. To define it, draw a rectangle around the overall area of the element and the images. After that you can crop the selection to the top and the bottom image.
	Area Between Bitmaps TopLeft..BottomRight	Similar to both previous options, but the delimiting images don't have to take the whole height or width of the element.
	Area Defined by Two Bitmaps	In contrast to the previous option the images are not part of the element, but are anchors for the top left and bottom right corner of it.

Even if an application renders the same element, it can happen, that it is not to the pixel identical in the screenshot. To compensate the inaccuracies due to this effect, there are several search parameters. In the context menu of the element you can open an editor to edit these parameters (Locator (Image Search) Parameters...). There you can set for example the percentage how much the reference image may differ from the original or have only a grayscale or black-and-white search be done.

Image locators are described in more detail in the VNC plugin reference documentation.

Text Search

If OCR support is configured in expecco, it is possible to search for text inside the image. Be aware that the success obviously depends on the OCR result. A higher contrast may be an advantage, dark font on a light background is usually better recognized than vice versa. Icons or other graphical content might be interpreted as characters, which can cause problems if you search for a word next to it.

Area Under Text

Set the text that should be searched in the screenshot. The element then will be located in the area where this text is found.

XPath Elements

You can also define elements by XPath in the screenplay. This allows a path to be assigned a name, so to speak. On the one hand this contributes to the readability of the test, on the other hand this will make it easier to replace paths. Elements found like this are in the same way true technology elements as if you had written the path directly at the pin, i.e. they can be used for any action that is possible for the technology.

Relative Elements

Each of the screenplay elements described above can also be created relative to a technology element. In this case, the technology element will be searched first and further searching will then be done with its screenshot. An element is automatically created relatively, if a technology element is selected when defining it. Relative elements are displayed in the tree with a yellow tint.

Alternatively to this method, you can simply combine paths, see section Notation.

Notation

The @-sign is used to mark an element-ID when using an element from the screenplay at a block. E.g. @OkButton. There are other notation when using a screenplay with VNC or on local screen, but they are not allowed when using it with the other GUI technologies, as it must be distinguishable if a locator is an XPath or a screenplay element. An element ID may not contain the characters '@', '$', and '/'.

Besides it is possible to combine several elements in one path. The path will be parsed from left to right and only the previously found part will be used to search further. Each part has to be separated by"/". E.g. @leftArea/@OkButton. It is allowed to combine XPath of the technology with Element-IDs, but it has to be noted, that no Element-ID of a graphical element can be left of an XPath.

---

Copyright © 2014-2024 eXept Software AG