Report Generation/en: Unterschied zwischen den Versionen

Aus expecco Wiki (Version 2.x)
Zur Navigation springen Zur Suche springen
Zeile 13: Zeile 13:
To create a report, a test has to be executed first. This can be either a single block test or a whole testplan. After the execution is finished, click on the "Print report" button ([[Bild:Icon Print Report.png|Report button]]) causing a dialog to open. Here you can select the output format (PDF, HTML, XML, CSV or TEXT), the report language, a paper format (either DIN A4 or Letter format) and a template for the report. This template can be the default Report template, a template from the current testsuite or a template which is stored separately in a file. You can also make ad-hoc changes to the printed report right before it is generated, by clicking on "''Edit Default Report Template''" in the 'Generate Report' dialog. There, you can specify the detail and the look of the report. (company, logos, colours, fonts, etc.)
To create a report, a test has to be executed first. This can be either a single block test or a whole testplan. After the execution is finished, click on the "Print report" button ([[Bild:Icon Print Report.png|Report button]]) causing a dialog to open. Here you can select the output format (PDF, HTML, XML, CSV or TEXT), the report language, a paper format (either DIN A4 or Letter format) and a template for the report. This template can be the default Report template, a template from the current testsuite or a template which is stored separately in a file. You can also make ad-hoc changes to the printed report right before it is generated, by clicking on "''Edit Default Report Template''" in the 'Generate Report' dialog. There, you can specify the detail and the look of the report. (company, logos, colours, fonts, etc.)


Next, choose the format of the generated file(s).
Further you can choose the format of the generated file(s).
You may select multiple output formats here.
You may select multiple output formats here.



Version vom 1. Oktober 2020, 12:35 Uhr

Automatic Report Generation[Bearbeiten]

Expecco can generate reports in a number of formats: as human readable PDF or HTML documents, or as machine readable XML-, CSV- or plain text containing all relevant information on the test execution. XML reports are further provided in 3 different formats: Junit compatible, Nunit compatible and the full detail XML.

The full detail XML contains all data as collected during the run, including timestamps and input-/output values of every logged activity. Bz using XML processors (such as XSLT), any custom report can be generated from it, if the other reports are not sufficient.

Report Templates[Bearbeiten]

Report templates are used to specify the content and layout of created reports. By default the so called "Default Report Template" is used to create reports. Other report templates can be created in testsuites by adding a report template element to the tree. They will be saved together with the testsuite. The "Default Report Template" is stored in your settings file and thus private, whereas attached report templates are contained in the saved test suite file, and thus automatically available to everyone who gets the testsuite.

How to create a report[Bearbeiten]

Report Settings

To create a report, a test has to be executed first. This can be either a single block test or a whole testplan. After the execution is finished, click on the "Print report" button (Report button) causing a dialog to open. Here you can select the output format (PDF, HTML, XML, CSV or TEXT), the report language, a paper format (either DIN A4 or Letter format) and a template for the report. This template can be the default Report template, a template from the current testsuite or a template which is stored separately in a file. You can also make ad-hoc changes to the printed report right before it is generated, by clicking on "Edit Default Report Template" in the 'Generate Report' dialog. There, you can specify the detail and the look of the report. (company, logos, colours, fonts, etc.)

Further you can choose the format of the generated file(s). You may select multiple output formats here.

Human readable formats are:

  • PDF - for printable document
  • HTML - to publish as an intranet or public web page

Machine readable formats are:

  • XML - contains full detail
  • jUnit - jUnit compatible XML (*)
  • nUnit - nUnit compatible XML
  • CSV - to be imported into excel or word
  • text - plain text

If the "Open Preview" box is checked, the generated reports are shown in a preview application. Otherwise, they are silently generated in the folder as specified in your report settings.

Now click on "Generate Report" in the 'Generate Report' dialog, (with the "Open Preview" checked, to see the output).

  • ) the jUnit and nUnit formats are compatible to the output generated by corresponding java and .NET unit testing frameworks. These files can be imported by eg. Jenkins to generate trend- and statistic reports.
    For how to integrate expecco into Jenkins, see "Integration with Jenkins" in the doc-Wiki.

How to customize a report template[Bearbeiten]

To customize a report template you have to select it in the navigation tree. In the related editor the template can be modified. The same editor is also used to modify the default report template. You can access it either via the already mentioned dialog or via the general expecco settings.

Filter Settings[Bearbeiten]

Filter Settings

These settings determine which items are part of the report. This way you can create either detailed or condensed reports.

Testsuite[Bearbeiten]

  • Do not Print Testsuite Environment
    This option determines whether the testsuite environment, i.e. the variables defined in the current testsuite, are part of the test report. Please note that only variables which are defined as read-only will be shown in the report because the value of other variables can change during the test execution.

Testplan[Bearbeiten]

  • Do not Print Testplan Environment
    This option determines whether the environment of each executed testplan, i.e. the variables defined in the respective testplan, are included in the report. Please note that only variables which are defined as read-only will be shown in the report.
  • Do not Print Successful Testcases
    This option removes all successful test cases from the report when checked, so that only failed tests are reported.
  • Only Print Testplan Summary
    If checked, this option will reduce every testplan to a summary of the results of the different testplan items, i.e. it removes all further information such as sub-activities or logs.
  • Print Unexecuted Test Cases
    This option also includes test cases to the report which have not been executed.
  • Print Pre/Post Condition
    This option determines whether the pre- and post executions are present in the report or not. There is also an option to include this information only for failed tests.

Activity[Bearbeiten]

  • Nesting Limit
    This option determines to which level nested sub-activities are reported. This affects the length and also the level of detail of the generated report.
  • Activity Tags
    In order to include or exclude activities spawned by certain blocks from the report, you can add tags to each block. These tags can be set in the documentation tab of the block. Later you can filter the executed blocks by their tags. Please note that an inclusion to the report has priority over an exclusion.
    • Include Tag(s)
      All blocks which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Exclude Tag(s)
      All blocks which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Sub-activities Include Tag(s)
      All sub activities which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Sub-activities Exclude Tag(s)
      All sub activities which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Search Tags in Children
      This option determines whether the sub-activities of an activity are searched for existing tags, when filtering.
  • Print Input/Output Pins
    This option determines whether the data of input and output pins is present in the report. You can also add this information only in case the execution failed.
  • Do not Print Output Pins
    This option removes the data of the output pins from the report, so that only input values are printed.

Logs[Bearbeiten]

These settings are used to specify how logs are reported.

  • Nesting Limit
    This option is used to determine down to which level logs of nested sub activities are reported.
  • Log Tags
    In order to include or exclude logs of certain blocks from the report you can add tags to each block. Please note that an inclusion to the report has priority over an exclusion.
    • Include Tag(s)
      All logs of blocks which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Exclude Tag(s)
      All logs of blocks which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
    • Search Tags in Children
      This option determines whether the sub activities of an activity are searched for existing tags when filtering.
  • Log Types
    These checkboxes are used to include or exclude different log message types from the report. Please note that there is a difference between logs that only change the state of an activity, e.g. "logFail", and real exceptions that stop the execution, e.g. "fail". The "All" button is used to toggle all types.

Presentation Settings[Bearbeiten]

Presentation Settings

These settings are used to determine how the report is presented.

Common[Bearbeiten]

  • Print Full Activity Name
    This option determines whether the name of the corresponding block is printed when a step is reported. If the step-name is the same as the block name (which is the common case), then that option does not make a difference. But if a step has been renamed from the underlying block's name, then both names will be printed if they are different.
  • Print Activity Name in Logs as Well
    This option determines whether the name of the activity which logged a message will be printed in front of the message.
  • Indention Level Limit
    This option is used to determine to which nesting level reported sub activities are indented.
  • Highlight Top State in Every Row
    This option adds a colored area with the top state color in every row. This helps to keep track of the overall test case result in case that many sub activities are reported.
  • Page Break after Testplan Summary
    This option will add a page break after a testplan summary.
  • Page Break after every Testcase
    This option will add a page break after every test case.

Fonts[Bearbeiten]

These settings are used to change the fonts used for the document, the column headers and the activity reports.

Colors[Bearbeiten]

These settings are used to change the color of different report items. Its possible to use the default color as defined in the settings for error and failure logs. You can also print a "Only Black & White" report by toggling the corresponding check-box.

Test Overview Settings[Bearbeiten]

Test Overview Settings

These settings can be used to customize the report summary.

  • Testplan Summary
    Here you can specify how the summary of a testplan is presented. The editor panel is described here: Editor Panel
  • Activity Summary
    Here you can specify how the summary of an activity is presented. The editor panel is described here: Editor Panel

Document Settings[Bearbeiten]

Document Settings

These settings apply on the general design of the report. Note that the header and footer line will only be present in a PDF report.

  • Document Header
    Here you can create a header for the report. The editor panel is described here: Editor Panel
  • Columns Width
    Here you can resize the three columns of the report. Please note that each column has to fill at least 10 percent of the entire width.
  • Header
    These text fields are used to edit the header line of the report. The left fields are used to enter left aligned text, the center ones for center aligned text and the right ones for right aligned text. You can also create links by typing a target into the "link" fields under the text fields.
  • Footer
    These text fields are used to edit the footer line of the report. The usage is similar to the header line fields.

Logo Settings[Bearbeiten]

These settings are used to specify images, e.g. company logos, which can be used in the report. By default the expecco logo is used. To replace the default images just click on the "replace" button behind the filenames and select the new image's file in the file dialog. Please note that the file type has to be ".jpg".

Description[Bearbeiten]

Here you can add a description for the report template. This can be useful to inform other users for example when to use this template.

Editor Panel[Bearbeiten]

This editor panel is used to create style information for report summaries and the report header.

The workspace is made up of a number of rows which are separated columns (limited to a maximum of 6 columns). A single field is called cell. It can contain either text or graphics, such as a pie chart or a logo.

To enter text, double click on the cell causing a text editor to open.

To insert graphics, right click on a cell and select "Insert Graphic". Now you can select the type of the graphic. Please note that pie charts are only available for testplan reports and logos only for the document header.

Inside text, the content of a placeholder variable, such as the name of the testsuite or the current date can be inserted: double click on a cell to open the editor and right click to open the context menu. Now select "Insert Variable" and choose one of the variables from the list. Currently, variables available in a testplan overview are:

  • %testplan_name% - The name of the testplan
  • %testcases_size% - The number of testcases in the testplan
  • %testcases_executed% - The number of testcases in the report
  • %hostname% - The hostname of the executing machine
  • %username% - The username (login name)
  • %start_date% - The date when the testplan was started
  • %start_time% - The time when the testplan was started
  • %run_time% - the overall execution time of the testplan (when finished)
  • %elapsed_time% - the elapsed execution time of the testplan (only useful if a GenerateReport block is used from within an executing testplan. same as run_time when finished)

Variables available in an activity-overview are:

  • %action_name% - The name of the activity
  • %hostname% - The hostname of the executing machine
  • %username% - The username (login name)
  • %start_date% - The time when the activity was started
  • %start_time% - The time when the activity was started
  • %run_time% - the execution time of the activity

Buttons[Bearbeiten]

  • Icon Insert Row.png Inserts a new row after the selected one
  • Icon Insert Column.png Inserts a new column right from the selected one. Please note that only the selected row is affected.
  • Icon Remove Row.png Removes the selected row
  • Icon Remove Column.png Removes the selected column
  • Icon Report Move Up.png Icon Report Move Down.png Moves the selected row up/down
  • Icon Report Move Left.png Icon Report Move Right.png Moves the selected column left/right
  • Icon Open Editor.png Opens an editor to modify the content of the selected column
  • Icon Save Close Editor.png Closes the editor with saving changes
  • Icon NoSave Close Editor.png Closes the editor without saving changes
  • Icon Left Alignment.png Icon Center Alignment.png Icon Right Alignment.png Aligns the content of the column left/center/right
  • Icon Show Hide Grid.png Shows/Hides a grid to resize the columns


Report vs. expecco Log-File[Bearbeiten]

In contrast to the generated reports, which are meant to be either human readable archived documents (HTML, PDF, text) or machine readable, to be processed by other programs (CSV, XML), there is another, expecco-internal storage format, the so called "expecco log files", with a ".elf" extension. These files store all relevant tree elements, log data, in- and output in values which are required to reconstruct a full activity log/result tree. This also means, that they include a copy of the executed blocks and their definitions, independent of any changes done to the test suite in the meantime. As such, they can be also archived to proof exactly which version of a suite was executed.

Just like regular reports, these log files can be generated after the execution of a test (both single block- and full testplan tests), and saved to a file:

Save elf1.png

 

expecco ALM (formerly called "expeccoNET") also provides central storage of all log files and makes them available later to team members with appropriate access rights.

Save elf2.png

As you can see, there is a menu item named "Load Test-Results (from expeccoNET)“ which loads those ".elf" files from this central database. If you do not have expecco ALM installed, you have to save/load those from a regular file. After loading, the result tree is presented as it was right after the execution. You can also navigate through the tree, see pin values and other artefacts (however, of course, objects which where present on remote systems and only referred to via a bridge connection are no longer available. If required, these should be marshaled to XML or any other textual format, if that is required).


Another advantage of this format is that additional reports can be generated at any time later. For example, you can generate a new report with a different layout, look or detail. For example, if your company logo has changed in the meanwhile, or the product department needs another, less detailed report.


Print elf.png

Example: Executive Summary vs. Developer Report[Bearbeiten]

Executive Summary
Developer Report.

The main advantage of report templates is the possibility to create standardized reports for different purposes. This way you can for example create a condensed report as a summary for an executive or a very detailed one for the development team to track down error causes or to document a test run.

In the shown example the executive summary solely shows the overall test results which are visualized by a pie chart. Other irrelevant information has been removed to keep the report clear and readable. A corresponding template can be found here: Datei:Executive Summary Template.ert

The shown developer report is more detailed and shows all relevant data to track down a possible error. In this case all irrelevant data from the developer's point of view is left out. For example the pie chart might not be useful in this case. A corresponding template can be found here: Datei:Developer Report Template.ert

Programmatic Report Generation[Bearbeiten]

In rare situations, the automatic generated reports are not sufficient to fullfill your needs. It may be required to conform to a particular document format, or contain additional data, or it may need to be postprocessed (for example: cryptographically signed).

For this, you can either take the automatic report and post-process it with an expecco action (this would be the case when it has to be signed), or create your report completely driven by your own actions. In the following, a few examples are provided to show possible solutions:

Postprocessing Reports[Bearbeiten]

You can define a spot processing action to be executed after a test run (find the field in the lower area of the test plan editor). This action will get the generated activity log as argument at its input pin. Notice that this activity log contains all traced pin data, timestamps, notes and attachment information. The action can walk through this object and generate arbitrary output. An example for this is found in the deployed demo/example folder ("LOG-Processing example" in "demos/d11_LOG_Processing_Example.ets").

Generating CSV Report Files[Bearbeiten]

Collect the CSV rows (i.e. the raw data to be put into the CSV columns) during the run, and write a CSV file at the end.

For this:

  • define an action, which creates an empty (sequencable) collection (i.e. "New OrderedCollection" will do).
  • inside the action, store this created collection in a test suite variable (let's say: "collectedData")

Pre-action csv report.png

  • make this action a "pre-action" of your test plan
  • during the run, add rows as required to the "collectedData" collection (give the vector to an "Add Element" step)

Collect data csv report.png

  • define an action which saves the collectedData to a CSV file (use the "Write CSV" action)

Post-action csv report.png

  • make this write-action a "post-action" of your test plan

Testplan csv report.png

All of the above actions are found in the standard library; there is no need to do any textual coding. However, if you prefer scripts, you can of course do so as well.

Generating a PDF Report[Bearbeiten]

PDFGeneratorLibrary.png

As part of the expecco installation, a PDF generator library is provided, which contains building blocks for PDF documents. Similar to the above strategy, start PDF generation in a pre-action, collect data during the run and write the final PDF in a post-action at the end.



Copyright © 2014-2024 eXept Software AG