Report Generation/en

2020-09-16T07:12:44Z

Anharman: /* Generating CSV Report Files */

= Automatic Report Generation =

Expecco can generate reports in a number of formats: as human readable PDF or HTML documents, or as machine readable XML-, CSV- or plain text containing all relevant information on the test execution. XML reports are further provided in 3 different formats: Junit compatible, Nunit compatible and the full detail XML.

The full detail XML contains all data as collected during the run, including timestamps and input-/output values of every logged activity. Bz using XML processors (such as XSLT), any custom report can be generated from it, if the other reports are not sufficient.

== Report Templates ==
Report templates are used to specify the content and layout of created reports. By default the so called "Default Report Template" is used to create reports. Other report templates can be created in testsuites by adding a report template element to the tree. They will be saved together with the testsuite.

== How to create a report ==
[[Bild:Report Settings.png|thumb|200px|Report Settings]]
To create a report, a test has to be executed first. This can be either a single block test or a whole testplan. After the execution is finished, click on the "Print report" button ([[Bild:Icon Print Report.png|Report button]]) causing a dialog to open. Here you can select the output format (PDF, HTML, XML, CSV or TEXT), the report language, a paper format (either DIN A4 or Letter format) and a template for the report. This template can be the default Report template, a template from the current testsuite or a template which is stored separately in a file. You can also choose to edit the default report template.

== How to customize a report template ==
To customize a report template you have to select it in the navigation tree. In the related editor the template can be modified. The same editor is also used to modify the default report template. You can access it either via the already mentioned dialog or via the general expecco settings.

=== Filter Settings ===
[[Bild:Report Filter Settings.png|thumb|200px|Filter Settings]]
These settings determine which items are part of the report. This way you can create either detailed or condensed reports.

==== Testsuite ====
*'''Do not Print Testsuite Environment''' This option determines whether the testsuite environment, i.e. the variables defined in the current testsuite, are part of the test report. Please note that only variables which are defined as read-only will be shown in the report because the value of other variables can change during the test execution.

==== Testplan ====
*'''Do not Print Testplan Environment''' This option determines whether the environment of each executed testplan, i.e. the variables defined in the respective testplan, are included in the report. Please note that only variables which are defined as read-only will be shown in the report.

*'''Do not Print Successful Testcases''' This option removes all successful test cases from the report when checked, so that only failed tests are reported.

*'''Only Print Testplan Summary''' If checked, this option will reduce every testplan to a summary of the results of the different testplan items, i.e. it removes all further information such as sub-activities or logs.

*'''Print Unexecuted Test Cases''' This option also includes test cases to the report which have not been executed.

*'''Print Pre/Post Condition''' This option determines whether the pre- and post executions are present in the report or not. There is also an option to include this information only for failed tests.

==== Activity ====
*'''Nesting Limit''' This option determines to which level nested sub-activities are reported. This affects the length and also the level of detail of the generated report.

*'''Activity Tags''' In order to include or exclude activities spawned by certain blocks from the report, you can add tags to each block. These tags can be set in the [[Documentation Editor|documentation tab]] of the block. Later you can filter the executed blocks by their tags. Please note that an inclusion to the report has priority over an exclusion.
**'''Include Tag(s)''' All blocks which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Exclude Tag(s)''' All blocks which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Sub-activities Include Tag(s)''' All sub activities which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Sub-activities Exclude Tag(s)''' All sub activities which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Search Tags in Children''' This option determines whether the sub-activities of an activity are searched for existing tags, when filtering.

*'''Print Input/Output Pins''' This option determines whether the data of input and output pins is present in the report. You can also add this information only in case the execution failed.

*'''Do not Print Output Pins''' This option removes the data of the output pins from the report, so that only input values are printed.

==== Logs ====
These settings are used to specify how logs are reported.

*'''Nesting Limit''' This option is used to determine down to which level logs of nested sub activities are reported.

*'''Log Tags''' In order to include or exclude logs of certain blocks from the report you can add tags to each block. Please note that an inclusion to the report has priority over an exclusion.
**'''Include Tag(s)''' All logs of blocks which have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Exclude Tag(s)''' All logs of blocks which do not have one or more of those tags will be present in the report. Multiple tags must be separated by ";".
**'''Search Tags in Children''' This option determines whether the sub activities of an activity are searched for existing tags when filtering.

*'''Log Types''' These checkboxes are used to include or exclude different log message types from the report. Please note that there is a difference between logs that only change the state of an activity, e.g. "logFail", and real exceptions that stop the execution, e.g. "fail". The "All" button is used to toggle all types.

=== Presentation Settings ===
[[Bild:Report Presentation Settings.png|thumb|200px|Presentation Settings]]
These settings are used to determine how the report is presented.

==== Common ====
*'''Print Full Activity Name''' This option determines whether the name of the corresponding block is printed when a step is reported. If the step-name is the same as the block name (which is the common case), then that option does not make a difference. But if a step has been renamed from the underlying block's name, then both names will be printed if they are different.

*'''Print Activity Name in Logs as Well''' This option determines whether the name of the activity which logged a message will be printed in front of the message.

*'''Indention Level Limit''' This option is used to determine to which nesting level reported sub activities are indented.

*'''Highlight Top State in Every Row''' This option adds a colored area with the top state color in every row. This helps to keep track of the overall test case result in case that many sub activities are reported.

*'''Page Break after Testplan Summary''' This option will add a page break after a testplan summary.

*'''Page Break after every Testcase''' This option will add a page break after every test case.

==== Fonts ====
These settings are used to change the fonts used for the document, the column headers and the activity reports.

==== Colors ====
These settings are used to change the color of different report items. Its possible to use the default color as defined in the [[Settings#State Colors|settings]] for error and failure logs. You can also print a "Only Black & White" report by toggling the corresponding check-box.

=== Test Overview Settings ===
[[Bild:Report Test Overview Settings.png|200px|thumb|Test Overview Settings]]
These settings can be used to customize the report summary.

*'''Testplan Summary''' Here you can specify how the summary of a testplan is presented. The editor panel is described here: [[Report Generation#Editor Panel|Editor Panel]]
*'''Activity Summary''' Here you can specify how the summary of an activity is presented. The editor panel is described here: [[Report Generation#Editor Panel|Editor Panel]]

=== Document Settings ===
[[Bild:Report Document Settings.png|thumb|200px|Document Settings]]
These settings apply on the general design of the report. Note that the header and footer line will only be present in a PDF report.

*'''Document Header''' Here you can create a header for the report. The editor panel is described here: [[Report Generation#Editor Panel|Editor Panel]]
*'''Columns Width''' Here you can resize the three columns of the report. Please note that each column has to fill at least 10 percent of the entire width.
*'''Header''' These text fields are used to edit the header line of the report. The left fields are used to enter left aligned text, the center ones for center aligned text and the right ones for right aligned text. You can also create links by typing a target into the "link" fields under the text fields.
*'''Footer''' These text fields are used to edit the footer line of the report. The usage is similar to the header line fields.

=== Logo Settings ===
These settings are used to specify images, e.g. company logos, which can be used in the report. By default the expecco logo is used. To replace the default images just click on the "replace" button behind the filenames and select the new image's file in the file dialog. Please note that the file type has to be ".jpg".

=== Description ===
Here you can add a description for the report template. This can be useful to inform other users for example when to use this template.

== Editor Panel ==

This editor panel is used to create style information for report summaries and the report header.

The workspace is made up of a number of rows which are separated columns (limited to a maximum of 6 columns). A single field is called cell. It can contain either text or graphics, such as a pie chart or a logo.

To enter text, double click on the cell causing a text editor to open.

To insert graphics, right click on a cell and select "''Insert Graphic''". Now you can select the type of the graphic. Please note that pie charts are only available for testplan reports and logos only for the document header.

Inside text, the content of a placeholder variable, such as the name of the testsuite or the current date can be inserted: double click on a cell to open the editor and right click to open the context menu. Now select "''Insert Variable''" and choose one of the variables from the list.
Currently, variables available in a testplan overview are:
* %testplan_name% - The name of the testplan
* %testcases_size% - The number of testcases in the testplan
* %testcases_executed% - The number of testcases in the report
* %hostname% - The hostname of the executing machine
* %username% - The username (login name)
* %start_date% - The date when the testplan was started
* %start_time% - The time when the testplan was started
* %run_time% - the overall execution time of the testplan (when finished)
* %elapsed_time% - the elapsed execution time of the testplan (only useful if a GenerateReport block is used from within an executing testplan. same as run_time when finished)

Variables available in an activity-overview are:
* %action_name% - The name of the activity
* %hostname% - The hostname of the executing machine
* %username% - The username (login name)
* %start_date% - The time when the activity was started
* %start_time% - The time when the activity was started
* %run_time% - the execution time of the activity

=== Buttons ===
*[[Bild:Icon Insert Row.png]] Inserts a new row after the selected one
*[[Bild:Icon Insert Column.png]] Inserts a new column right from the selected one. Please note that only the selected row is affected.
*[[Bild:Icon Remove Row.png]] Removes the selected row
*[[Bild:Icon Remove Column.png]] Removes the selected column
*[[Bild:Icon Report Move Up.png]] [[Bild:Icon Report Move Down.png]] Moves the selected row up/down
*[[Bild:Icon Report Move Left.png]] [[Bild:Icon Report Move Right.png]] Moves the selected column left/right
*[[Bild:Icon Open Editor.png]] Opens an editor to modify the content of the selected column
*[[Bild:Icon Save Close Editor.png]] Closes the editor with saving changes
*[[Bild:Icon NoSave Close Editor.png]] Closes the editor without saving changes
*[[Bild:Icon Left Alignment.png]] [[Bild:Icon Center Alignment.png]] [[Bild:Icon Right Alignment.png]] Aligns the content of the column left/center/right
*[[Bild:Icon Show Hide Grid.png]] Shows/Hides a grid to resize the columns

== Report vs. expecco Log-File ==


In contrast to the generated reports, which are meant to be either human readable archived documents (HTML, PDF, text) or machine readable, to be processed by other programs (CSV, XML),
there is another, expecco-internal storage format, the so called "expecco log files", with a ".elf" extension.
These files store all relevant tree elements, log data, in- and output in values which are required to reconstruct a full activity log/result tree.
This also means, that they include a copy of the executed blocks and their definitions, independent of any changes done to the test suite in the meantime. As such, they can be also archived to proof exactly which version of a suite was executed.

Just like regular reports, these log files can be generated after the execution of a test (both single block- and full testplan tests), and saved to a file:

[[Datei:Save_elf1.png‎]]

 

expecco ALM (formerly called "expeccoNET") also provides central storage of all log files and makes them available later to team members with appropriate access rights.


[[Datei:Save_elf2.png]]

As you can see, there is a menu item named "Load Test-Results (from expeccoNET)“ which loads those ".elf" files from this central database. If you do not have expecco ALM installed, you have to save/load those from a regular file.
After loading, the result tree is presented as it was right after the execution. You can also navigate through the tree, see pin values and other artefacts (however, of course, objects which where present on remote systems and only referred to via a bridge connection are no longer available. If required, these should be marshaled to XML or any other textual format, if that is required).



Another advantage of this format is that additional reports can be generated at any time later. For example, you can generate a new report with a different layout, look or detail. For example, if your company logo has changed in the meanwhile, or the product department needs another, less detailed report.



[[Datei:Print_elf.png]]

== Example: Executive Summary vs. Developer Report ==
[[Bild:Report Executive Summary.png|thumb|links|200px|Executive Summary]] [[Bild:Report Developer Report.png|thumb|200px|Developer Report.]]
The main advantage of report templates is the possibility to create standardized reports for different purposes. This way you can for example create a condensed report as a summary for an executive or a very detailed one for the development team to track down error causes or to document a test run.

In the shown example the executive summary solely shows the overall test results which are visualized by a pie chart. Other irrelevant information has been removed to keep the report clear and readable. A corresponding template can be found here: [[Datei:Executive Summary Template.ert]]

The shown developer report is more detailed and shows all relevant data to track down a possible error. In this case all irrelevant data from the developer's point of view is left out. For example the pie chart might not be useful in this case. A corresponding template can be found here: [[Datei:Developer Report Template.ert]]

= Programmatic Report Generation =

In rare situations, the automatic generated reports are not sufficient to fullfill your needs. It may be required to conform to a particular document format, or contain additional data, or it may need to be postprocessed (for example: cryptographically signed).

For this, you can either take the automatic report and post-process it with an expecco action (this would be the case when it has to be signed), or create your report completely driven by your own actions.
In the following, a few examples are provided to show possible solutions:

== Generating CSV Report Files ==

Collect the CSV rows (i.e. the raw data to be put into the CSV columns) during the run, and write a CSV file at the end.

For this:
* define an action, which creates an empty (sequencable) collection (i.e. "New OrderedCollection" will do).
* inside the action, store this created collection in a test suite variable (let's say: "collectedData")

[[Datei:Pre-action_csv_report.png|500px]]

* make this action a "pre-action" of your test plan
* during the run, add rows as required to the "collectedData" collection (give the vector to an "Add Element" step)

[[Datei:Collect data csv report.png|650px]]

* define an action which saves the collectedData to a CSV file (use the "Write CSV" action)

[[Datei:Post-action csv report.png|500px]]

* make this write-action a "post-action" of your test plan

[[Datei:Testplan csv report.png|650px]]

All of the above actions are found in the standard library; there is no need to do any textual coding.
However, if you prefer scripts, you can of course do so as well.

== Generating a PDF Report ==

[[Datei:PDFGeneratorLibrary.png|mini]]
As part of the expecco installation, a PDF generator library is provided, which contains building blocks for PDF documents.
Similar to the above strategy, start PDF generation in a pre-action, collect data during the run and write the final PDF in a post-action at the end.

Datei:Testplan csv report.png

2020-09-16T07:11:20Z

Anharman:

Testplan that demonstrates how to generate CSV Report file

Datei:Post-action csv report.png

2020-09-16T07:09:48Z

Anharman:

Post-action for generating CSV Report Files

Datei:Collect data csv report.png

2020-09-16T07:01:31Z

Anharman:

Adds rows to the data collected for the CSV report

Datei:Pre-action csv report.png

2020-09-16T06:57:24Z

Anharman: Anharman lud eine neue Version von Datei:Pre-action csv report.png hoch

Pre-action for generating CSV Report file

Datei:Pre-action csv report.png

2020-09-16T06:49:38Z

Anharman:

Pre-action for generating CSV Report file

expecco Wiki (Version 25.x) - Benutzerbeiträge [de]

Report Generation/en

Datei:Testplan csv report.png

Datei:Post-action csv report.png

Datei:Collect data csv report.png

Datei:Pre-action csv report.png

Datei:Pre-action csv report.png