Part of my job in the Chief Customer Office at SAP is to play detective. One particular case involved a situation where the Systems Integrator said 90% of the test scripts passed but after a go-live date was missed it was learned that only 30% of the scripts passed. The executive sponsor of the project told me he would have made different decisions if he would have know that there was so much more to test. So who was right? They both were...The problem was how the numbers were being reported. The Systems Integrator was not including scripts that had been deferred to future testing cycles where the Executive Sponsor thought everything was included.
The following are four ways to make sure you report your testing results accurately:
Always report on all the test scripts in your status meetings. During a project, test scripts go through different stages of when they are ready to be tested. Just because it isn't tested during the phase of the project you are in doesn't mean it should be reported.
Categorize the Different States of Test Scripts
Keep it simple..."Failed", "No Run" and "Passed". For the example above, a additional category of "Deferred" would help give an extra level of detail.
Clearly Define The Categories
I define "Failed" as any step in the script didn't work...No matter how major or minor. Others would give a script a "Passed" if there was a work around the tester could do to get the script to complete. Regardless, there should be a clear criteria and it should be communicated regularly to the Steering Committee.
Use a Colored Stacked Bar Graph
A good example, broken down by project area, is included in the graphic. I love a good picture. A stacked bar is easy to tell proportionally how much is in each category. Also include the number of scripts in each category so the raw numbers are clear. In this example, you can see that there are still 61 scripts that need to be tested in the PP area.
With these simple techniques, you will avoid confusion, have a much smoother project and not need the "Script Detective".
Let me know your experiences below or e-mail me @ firstname.lastname@example.org.