This page serves as a lessons learned during the HvN tests of the KBI system as well as an example of how to conduct Session Based Testing in a setting that requires a low overhead test approach in which I endeavored to:
- Get to know the product as soon as possible
- Find the blocking bugs as well as any critical malfunctions before the client does
- Follow up on where you spent your time and where to focus on next
- Have a report ready the moment it is asked.
In order to achieve these goals I invested time in designing an excel sheet.
This sheet was supposed to give me the tool in which to register my test ideas and give me the opportunity to quickly turn an ‘overview page’ into a clear report.
This way of working was originally conceived by Sarah Teugels, afterwards discussed fiercely and eventually poured into a report by me.
This report, however, is flawed. Like any other report it can be optimized further but if I’d like to bring it to the next level this would require me to invest more time in it.
Time better spent in studying the functional documents, the product, the client, the designer, the development team,…
If you have ideas to better the tool, please feel free to let me know. Or take the ideas and adapt them to your context.
I built the excel as one overview page where all Logical Modules are listed. The other tabs embody the Test ideas I gathered for a Logical Module.
| Overview Page | LM1 | LM2 | LM3 | … |
The Overview Page has these fields:
All fields are calculated, except for the ‘Major+’ column.
Logical Modules: I cannot tell you what Logical Modules should be in your situation. These can be derived from Requirements, grouped by the customer, separated by the Functional team and so on. You work with what suits your project best. In this example they represent a functional document. A group of Use Cases.
Weight: In consensus with the client or functional team, weigh the LM’s. Rate them from 1 to 10, or higher, or lower. Just make sure it reflects the test effort they are going to require. If “logging in” weighs 2 and “Sending an email” is 4, then you are going to put in twice as much test effort in sending emails than you are going to test the “logging in” functionality
Test Ideas Planned, Executed, ToDo: These Logical Modules will guide you in your test sessions. Keep in mind that these should not be your only guidance. An explorer dares to go off the path in front of him, climb high mountains and jumps over creeks. So should you while testing. A treasure is only a treasure if it hasn’t been found yet.
While doing these testing sessions, note down every possible idea you may have in the excel in the correct tab. These ideas you will follow in the next test session, or execute them right away.
This is what your test progress depends on. Test Ideas Planned minus the ones executed is your remaining, current, workload.
Mind the word ‘current’, because your work will never be finished. While testing I advise you to keep on listing the ideas that pop up. If nothing pours in, invest some time in reading a new technique, heuristic.
You could defocus and go on it a different way. Completely change the way you think about the product and look at it as someone else. These and tons of other aids can help you to list another myriad of test ideas.
Especially do this for the heavy-weight functionalities.
The alert function: The alert function compares the weight of the functionality against the total weight of the product and the number of test ideas executed of the functionality versus the total planned test ideas of the product.
Meaning: If the Weight % is bigger than the TI %, it alerts you to schedule more sessions. I’ve added a 20% margin in this comparison.
Use this to keep track of what you still need to do. Mind that the more test ideas you register for the low weight functionalities, the more you’ll have to come up with for the larger ones.
Major+: If your project isn’t all too concerned with issues that are lower than Major then this type of ‘filter’ can be useful for you.
If not, you can easily switch this out for another since this is the only field in the report that needs manual editing.
For my project, I included a column in which the number of issueswith priority Major or higher per Logical Module were shown.
The progress bar: Here is where the report could use some tweaking. It puts the Test Ideas ToDo versus the Test Ideas planned in a percentage, subtracting the number of Major+ issues.
This means a failed Test Idea resulting in a Major+ bug would subtract a percentage equal to 2 failed test cases.
This progress bar does have a lot of perks when used to present: Visual ‘progress’, percentages and colors.
You may find the XLS file here: Session based method example