VNC Plugin Reference/en

Introduction

The VNC plugin interfaces to local or remote applications via the RFB protocol, which enables interaction with a remote screen on any other machine in the network.

Using VNC has one big advantage, and a number of disadvantages:

• [+] it allows interaction with and the testing of almost any application, and no instrumentation or recompilation of the tested application is needed, given that a VNC-server is running and reachable on the target system. VNC servers are available or already installed on almost any operating system or device, even for mobile devices, embedded control systems, head units in cars, etc.

• [-] whilst all other UI testing plugins interact with the tested application directly, via a protocol which enables expecco to reflect (query) on internals of the app, the VNC interface can only show the screen, and can only send events based on screen coordinates. No information about the widget element hierarchy or internal attribute can be interchanged via VNC.

Thus, with VNC, expecco cannot ask the application about the position of GUI elements and cannot ask for any internal attribute values.

• [-] the VNC plugin cannot provide the values of internal attributes of widget elements (text, labels, enable state etc.)

• [-] VNC is very sensible to changes in the layout and look of the application.

• [-] the geometry information (i.e. the location of widget elements) is kept inside expecco, and cannot be queried from the application.

In view of those limitations, you should use the VNC plugin, if no other plugin is usable with the application. i.e. if one of the following applies:

• it is written in C/C++ and cannot be instrumented or recompiled
• it is written in Java and the JVM cannot be started with additional jars being loaded or the JVM provides no debugging interface (i.e. embedded Java)
• it runs under Windows and does not support the Accessibility API
• it is an old style (hand-written) X-windows application (Xt or non-widget library)
• the target app or operating system of the target opens additional windows/dialogs, which cannot be controlled by other mechanisms and which need to be closed or confirmed. Examples are operating system warnings ("disk full") or keyboard entry windows on mobile devices.

Philosophy

To allow for a maximum of flexibility, the VNC plugin uses description objects, which describe where and how to find UI elements. These descriptions are called "Screenplays" (or "Storyboards") and are typically stored in attachments inside the test suite.

Each screenplay describes the UI-elements of an application. It is structured into multiple "Scenes", which correspond to pages/tabs or states of the application. Whenever the application changes its widget layout (i.e. changes to another UI-hierarchy or layout), a new scene (stage-description) is activated. Finally, each scene contains "Players" (UI-elements), and descriptions on how to find them.

The testsuite refers to UI-elements by scene- and element (player) identifiers, and the current active scene provides the information of where and how to find them on the screen.

Screenplays, Scenes and Players are described in a human-readable, textual format in attachments. The GuiBrowser can read, modify and write such attachments. They can also be imported/exported in some standard (XML) format. Import is especially useful, if app-developers can provide this information from their IDE/Window builder files. Expecco uses a format similar to the FXML format, so it may be easy to exchange such description files with app developers (if required, using an XSLT transformation procedure).

Preparation of the System Under Test

You need a VNCserver to be installed and running on the target system. Any freely available or commercial VNC/RFB server should work out of the box (Protocol version 3.3 or higher). Is is known to work (i.e. we tried) the following VNC servers (as target system):

• Android (Samsung tablet): alpha VNC server (Beta version) (no rooting required)
• OS-X (MacBook Pro/Yosemite 10.10.5): Remote Desktop (already part of OS-X) or Vine Server 4.01
• Xvnc (Gentoo Linux)

Recording in the GUIBrowser

The VNC plugin is well integrated into the expecco GUIBrowser framework. It now seamlessly integrates the definition of geometric information (screenplay, stage & player definition) with the recording process. The following gives a sample session, where a login procedure is recorded on a remote (OS-X) system.

Opening the GUIBrowser

Open the GuiBrowser by clicking on its icon in the toolbar.

Opening a VNC Connection

Click on the "Connect"-button and choose "VNC Gui Automation".

A connect dialog will appear, asking for the hostname (or IP address) of the target system, a port-number (which is 5900 for screen #0, 5901 for screen #1, etc.) and some other optional values (user, password and screenplay). For now, you can ignore those.

when connected, the guiBrowser's tree is updated and an initial "Main Scene" is appearing. Also, a screenshot is shown in the lower left area, if you have the "Preview" button enabled.

In this session, a connection was made to an OS-X system, and the OS-X login screen is shown in the preview:

Now, you have two options: either open a recorder, which combines interaction, recording and screenplay definition, or define the screenplay in the preview.

Both have similar functionality and allow the same operations; however, the recorder displays a live image of the target system, whereas the preview shows a static "snapshot", which needs to be updated manually. Depending on the bandwidth of your network/usb connection and/or the speed of your target system, either may be a better choice. Also, because the preview shows a frozen snapshot, it may be easier to define elements there, if the application changes its look dynamically (i.e. to specify the bounds of dynamically disappearing menus or popup dialogs).

Locators

Inside the suite, elements are referred to by their elementID (within the current scene). For example, a "Button-Press" action takes the elementID (a string), the mouse-button number and a position relative to the widget's origin as argument. Expecco will then find the element's geometry in the current scene, translate the coordinates to screen coordinates, and send a VNC button-press to the screen.

Currently, four types of locators are supported:

• bounds
• bitmap
• bitmap position + bounds
• between two bitmap images

Bounds Locator

With bounds, the origin+corner (i.e. the rectangular bounds) of the element are stored in the scene, and this is used to locate the element - independent of what is actually shown on the screen. This kind of locator is useful, if you can get the geometry information from the widget developers, or when the position of the elements remains constant, but its look is changing. Of course, expecco cannot verify if what is shown at this position is really the expected element - it will simply trust your given bounds and click or type into those bounds.

Bitmap Locator

Here, a "prototype bitmap" is kept in the scene information, and the bitmap is "searched" on the screen. The actual bounds are then computed dynamically at test-execution time.

This makes the test less affected by changes in the exact position of the element, but more affected by changes in the look (i.e. changed icons or changed labels). The bitmap to be searched can be kept inside the testsuite itself (as attachment) or in external bitmap files. In general, it is recommended to keep them inside the testsuite; however, if you can get the set of images from the app development team (from their source repository), it may also make sense to keep them separate.

When test test is executed, expecco will first scan the whole screen for occurrences of that bitmap pattern, and complain if not exactly one is found. If none is found, the pattern is obviously not present on the screen. Or was not found for some reason, even if it looks like (read more on image finding below).

If the element is present multiple times, you can change the locator's element-name to an indexed name: "name[n]". This will then refer to the n'th occurrence of the pattern, starting at the top-left towards the bottom-right, and n being an index starting at 1.

Combined Bitmap Position + Bounds Locator

This is similar to the bitmap locator described above, but here, the bitmap's found position defines an anchor to the actual bounds, and the bounds are translated relative to that position. This kind of locator is useful, if an input field is always located to the right or below some known bitmap pattern.

Between two Images Locator

Here two bitmaps are searched, and the bounds of the element are defined as the area between them. The two fencing bitmaps can be defined to delimit the area horizontally, vertically or diagonal. This kind of locator is useful to detect elements like scrollbars (defined by their two scroll button images) or buttons (defined by their frame-borders at opposite edges).

Interactive Definition of Bitmap Locators

Initial Capture from the Screen or Preview

Choose the locator's type from the menu, and select the area of the image in the appropriate view. Most of the time, you will select from the preview, but if you find the bitmap already on the screen, it can also be selected there (footnote: the OS-X version currently only allows images to be fetched from one of expecco's windows)

If the locator is a dual-image locator (i.e. fencing the area between to images), the image capture is performed in 3 steps:

• define the overall bounds,
• define the first partial image
• define the second partial image.

For a walkthrough of this process, see the chapter "Defining Buttons and Similar Elements" below.

Postprocessing

When selecting an image's area, you will often (most of the time) select an area too large and maybe also areas which should be ignored in the image search.

Many applications blend the colors of bitmap images; the boundary may be antialiased or there may be areas where the background shines through. These will look different whenever the background color of the application changes, and will therefore not be found in the image search.

To deal with this situation, you have two mechanism to solve this: cutting and masking.

Use the image editor's crop functions to cut the image down to size. Define a mask to exclude areas from the comparison.

Cutting (Crop)

Either cut the image incrementally (in the editor's "Crop" tab), or select a whole new subimage with the -button.

Cut the image down to an area which contains only pixels which are not affected by the background/alpha blending. In this case, you could cut it down to a small rectangle, only keeping the flower's inner area:

Masking

Masking is especially useful at the boundary of images which are antialiased or which have areas where the background shines through. Here is the example image from the OS-X login screen:

the highlighted area will contain different pixels, depending on where the flower-icon is positioned on the screen. Both because the screen's background has a color gradient (the brightness of the background changes) and because the colors of the image's boundary are blended with the background color (alpha channel blending).

In this particular situation, cutting as described above is sufficient. However, you could also mask out pixels which should be ignored:

Notice that the image search is much slower, if masking is involved. Therefore, cutting is always the first choice.

However, cutting is not sufficient, if the bitmap contains transparent areas, as in the following image:

which consist almost completely of transparent areas, with the background shining through. Here, you should define a mask, by flood-filling the background areas.

Flood-Filling with a Mask

To flood-fill an area with a mask, select the "Flood-Fill" button and click on any pixel which is inside the area to be filled.

By default, the flood fill operation fills all connected areas which have the same color as the pixel you clicked on. This works if there is a constant background. In the above example, you will find that only parts are filled, as a consequence of the background's color gradient.

To get around this, change the "Flood-Fill Tolerance" to some non-zero value. The values specify by how much the hue (that is the color) and the brightness of a pixel may deviate from the clicked pixel and still be considered to be "in the area". A value of zero means: "must be exact", a value of 0.5 means: "50% deviation allowed".

We suggest, you start with 0.1 (which works most of the time) and try again. You may first undo the previous incomplete fill. With "0.1" tolerance, the background area will be correctly filled:

Verifying that your Image is Found

When finished with the definition of an element's image, click on "Verify Element" to make sure that it is found on the screen. If not, you should repeat the post processing step by selecting the element's "Edit Bitmap" menu function.

Defining Buttons and Similar Elements

Elements like buttons or input fields often consist of a frame surrounding an icon or label. Typically, the frame looks the same for all elements of this type. Here is a sample image of a GUI with multiple such buttons:

Obviously, you do not want to capture the numerous buttons individually (although this might be less work, if you can import the bitmaps, as described below).

You can define a template, which will match all buttons and then use indices to refer to the concrete buttons in the test. To define all buttons from a single template, proceed as follows:

Capture one of the Buttons as a Template

For template matches, we have 3 possible matching algorithms to choose from:

• define left and right areas to match
• define top and bottom areas to match
• define topLeft and bottomRight (diagnal) areas to match

The first two will find only buttons which have the same height and with as the template. The diagonal search will find all buttons, independent of their size. Whichever is better for your application depends on the application.

Here, we will use the "diagonal" match:

Select "Area Between TopLeft and BottomRight Images" from the "New Element" menu.

Grab one of the buttons and cut it down to size in the editor:

Then close this first editor, to proceed with the next step, where the top-left fencing image is defined. The editor for the top-left image opens automatically. Cut this down to a few pixels using the "Extract Subimage" function. For this, first select "Special Functions" , then select the rectangle and finally choose "Extract Subimage":

Here, a small 6x6 top-left image is extracted because some of the buttons draw their textual label over the left or right frame area, and this area should therefore be ignored later in the image search. (because of this, a left-right fenced image locator will not be able detect all buttons). Now the top-left image looks like:

Proceed in the same manner with the bottom-right image to make it look like:

finally, give this template element a name ("button") and confirm the dialogs asking for the attachment names.

Verification correctly confirms that 16 buttons where found (including the small scroll button in the lower right). These can be addressed as "button[1]" .. "button[15]". As expected, the blue button at the bottom is not found by this locator.

If the small scroll button should not be matched, a larger top-left or bottom-right image (especially: a higher one) should be used. This can be accomplished by re-editing the images with the tree-element's "Edit Original Image" menu function (or by editing the attachment bitmaps in the regular expecco project tree). In this concrete example, using 7x7 images does the job, and the 15 buttons are found by the verifier.

Bulk Import of Bitmaps for Locators

If you already have the bitmap images, or can get them from the app developers (as PNG, TIFF, GIF or BMP files), you can bulk-import them via the "Extras" - "Tools" - "Import Attachments" menu function.

Then, in the GUIBrowser select a scene element, and choose its "Define ScreenPlay Elements from Bitmaps In..." context menu function. This will create bitmap locators for all those existing bitmaps.

Possible Problems with Image Finding

Mouse Pointer

The image matching algorithm can be disturbed by the mouse pointer, which may be located over an area and could be grabbed with the image. You should therefore either make sure that the cursor is invisible, or positioned in a non-interesting are.

At test execution time, the cursor will be disabled by expecco to avoid this problem (this is not done in the current version)

Font Size affecting Element Dimensions

Obviously, if you define elements not by diagonal fencing images, they will not match if the element's sizes depend on the application's font settings. In general, it is not a good idea, to allow different font and color settings in the tested application.

Language Setting of the Application

The same is true for national language variants.

---

Copyright © 2014-2024 eXept Software AG