Good Aspects of Test Reporting – What Story Do You Intend To Tell?

I was on the judges’ panel for STWC World Cup Asia Rounds, as a part of which, I had the privilege to review test reports of several teams. While there were few quality test reports, others were appalling. Some teams simply delivered a collection of copied bug reports, some talked about different tasks testers performed and few others contained information that was useless to stakeholders and decision-makers. Hence, the need for this article.

What is a Test Report?

A Test Report conveys a story – about the product, about its quality, about the testing requirements and how well / or to what extent testing has been performed. A Test Report is a window of information for stakeholders about how much testing was done and what are the key risks associated with the product in order to make a release or no release decision. So, it is very critical for testers to generate clear, concise and quality test reports. In this article, I would like to highlight important components of a test report and what sets a test report apart from the run-of-the-mill reports I described in the beginning.

Components of a Good Test Report

Overview

This section should provide a brief summary of what the report contains along with all the sections (No, this is not the same as a Table of Content).

Mission

What did you plan to accomplish through your testing efforts? Find bugs? Identify risks? Clear blockers? Make recommendations? Provide information to stakeholders? Who are the stakeholders? Answering these questions will help your readers understand why you have set out for testing in the first place.

Context

What is the context in which the product/project has been tested? What were the assumptions? Did you cover any real world conditions?

Test Coverage

Coverage

Indicate how testing was done. How much coverage (functional, platform configurations, test data) was accomplished?

Feature Coverage

List out all features that have been tested. It helps to provide mindmaps for visualization. Features that have not been tested, blocked or paused will have corresponding explanations as to why those couldn’t be tested.

Platform Coverage (Test Environment)

Test environments include hardware, software, application specific configurations, device settings in case of mobile devices and so forth. Platform coverage can be broken into these sub-categories:

OS Coverage

Operating systems that have been covered during testing

Browser Coverage

Browsers that have been used for testing

Device Coverage

Mobile devices models, screen sizes and resolutions, network carriers that have been covered during testing

Test Data Coverage

Test Data dependencies that have been handled as part of one-time test data setup or for ongoing testing need to be highlighted in the test report for data based products. Elaborate how you made sure that test data was generated, how database integrity was ensured and how test data has been anonymized, if required.

Testing Techniques used

Highlight different testing techniques that have been used to perform different tests. This can be showcased in a mindmap for better visibility.

Testing Results

A summary of all test results is an important aspect of test reporting [No, this is not a count of Pass vs Fail]. This contains a detailed story about the overall health of each and every feature in the product/project. This also contains a summary of what was tested, to what extent testing was done, what kind of bugs were found, what risks do they bring and how it might impact users. Testers must keep target stakeholders in mind while presenting testing status to highlight potential risks and problems which will help them make key decisions.

Risks

Why did you choose to perform some tests over others? How did you prioritize? It is important to say why you covered specific tests and how critical they were. Highlight the tests that were not performed and how they might contribute to the overall risk associated with the product. Identifying a list of problems in the product beforehand, using a mitigation plan to reveal these risks and planning tests to mitigate these risks is highly beneficial.

References

This section should include references to all documents such as requirement specifications, applied standards and SOPs, detailed test reports, detailed bug reports and related summary documents that might be of further interest to stakeholders.

Conclusion – What Story Do You Intend To Sell?

A Test Report is a sneak preview into what testers think about the state of the product. It is an opportunity to alert stakeholders about current risks in the product. It talks about how the overall health of the product can be improved after taking cognizance of the information provided in the test report and whether they can go ahead releasing the product or not.

A good test report is your window of opportunity!
How well do you intend to write your test report?

Write new comment

Comments (0)

No comments found!