Session based Example

Software Testing

This page serves as a lessons learned during the HvN tests of the KBI system as well as an example of how to conduct Session Based Testing in a setting that requires a low overhead test approach in which I endeavored to:

  1. Get to know the product as soon as possible
  1. Find the blocking bugs as well as any critical malfunctions before the client does
  2. Follow up on where you spent your time and where to focus on next
  1. Have a report ready the moment it is asked.

In order to achieve these goals I invested time in designing an excel sheet.
This sheet was supposed to give me the tool in which to register my test ideas and give me the opportunity to quickly turn an ‘overview page’ into a clear report.

This way of working was originally conceived by Sarah Teugels, afterwards discussed fiercely and eventually poured into a report by me.

This report, however, is flawed. Like any other report it can be optimized further but if I’d like to bring it to the next level this would require me to invest more time in it.

Time better spent in studying the functional documents, the product, the client, the designer, the development team,…

If you have ideas to better the tool, please feel free to let me know. Or take the ideas and adapt them to your context.

The report

I built the excel as one overview page where all Logical Modules are listed. The other tabs embody the Test ideas I gathered for a Logical Module.

Worksheets overview

|    Overview Page   |    LM1   |    LM2    |    LM3    |    …    |

The Overview Page has these fields:

All fields are calculated, except for the ‘Major+’ column.

Logical Modules: I cannot tell you what Logical Modules should be in your situation. These can be derived from Requirements, grouped by the customer, separated by the Functional team and so on. You work with what suits your project best. In this example they represent a functional document. A group of Use Cases.

Weight: In consensus with the client or functional team, weigh the LM’s. Rate them from 1 to 10, or higher, or lower. Just make sure it reflects the test effort they are going to require. If “logging in” weighs 2 and “Sending an email” is 4, then you are going to put in twice as much test effort in sending emails than you are going to test the “logging in” functionalityLM

Test Ideas Planned, Executed, ToDo: These Logical Modules will guide you in your test sessions. Keep in mind that these should not be your only guidance. An explorer dares to go off the path in front of him, climb high mountains and jumps over creeks. So should you while testing. A treasure is only a treasure if it hasn’t been found yet.

While doing these testing sessions, note down every possible idea you may have in the excel in the correct tab. These ideas you will follow in the next test session, or execute them right away.

This is what your test progress depends on. Test Ideas Planned minus the ones executed is your remaining, current, workload.

Mind the word ‘current’, because your work will never be finished. While testing I advise you to keep on listing the ideas that pop up. If nothing pours in, invest some time in reading a new technique, heuristic.

You could defocus and go on it a different way. Completely change the way you think about the product and look at it as someone else. These and tons of other aids can help you to list another myriad of test ideas.

Especially do this for the heavy-weight functionalities.

The alert function: The alert function compares the weight of the functionality against the total weight of the product and the number of test ideas executed of the functionality versus the total planned test ideas of the product.

Meaning: If the Weight % is bigger than the TI %, it alerts you to schedule more sessions. I’ve added a 20% margin in this comparison.

Use this to keep track of what you still need to do. Mind that the more test ideas you register for the low weight functionalities, the more you’ll have to come up with for the larger ones.

Major+: If your project isn’t all too concerned with issues that are lower than Major then this type of ‘filter’ can be useful for you.

If not, you can easily switch this out for another since this is the only field in the report that needs manual editing.

For my project, I included a column in which the number of issueswith priority Major or higher per Logical Module were shown.

The progress bar: Here is where the report could use some tweaking. It puts the Test Ideas ToDo versus the Test Ideas planned in a percentage, subtracting the number of Major+ issues.

This means a failed Test Idea resulting in a Major+ bug would subtract a percentage  equal to 2 failed test cases.

This progress bar does have a lot of perks when used to present: Visual ‘progress’, percentages and colors.

You may find the XLS file here: Session based method exampleOverview SB

Flow

Software Testing

Flow as a testing activity

Flow is a state of mind.  A psychological condition of being fully immersed in a feeling of involvement. It’s commonly referred to as “On a roll”, “to be on fire”, “Getting your groove on” and my favorite “Going beast mode”.

There’s a lot to read about this concept and most of it is very interesting to us testers. However, treat it as a heuristic, as we should most concepts. While a lot of principles about the idea apply to testing, some don’t match. I will explain on this further on.

I simulate this mindset by going into seclusion for an hour or two, connect my earphones and turn on a good Pink Floyd album. This conjures up a feeling of well-being for me and gives me the opportunity to have a 100% focus on the system before me, no sidetracks to anything else than the product. (sidetracks inside the system are highly recommended!)

It is possible to go in to great detail, create wonderful test ideas, analyze specific functions and test extremely efficient.

Doing this kind of testing usually gets great results. If you don’t experience the returns you would expect, you need a different approach. Go talk to a developer and get some needed insight for example.

While being wired in you can easily forget to stop. There are a few indicators on when to stop your testing session.

 

  1. The feeling of diminishing returns
  2. You don’t get surprised anymore
  3. Your timing is done
  4. Other features require your time and energy more

Pitfalls of Flow

 

“The hallmark of flow is a feeling of spontaneous joy, even rapture, while performing a task although flow is also described (below) as a deep focus on nothing but the activity – not even oneself or one’s emotions.”

 

Pasted from <http://en.wikipedia.org/wiki/Flow_(psychology)>

 

Emotions, feelings, self-awareness are very needed aspects while testing. Keep these constantly in mind while being in a state of flow. Do not lose these out of sight for you might miss important signals if you do.

 

Conditions to achieve flow:

– Goals are clear

– Feedback is immediate

– A balance between opportunity and capacity

 

Pasted from <http://en.wikipedia.org/wiki/Flow_(psychology)#Music>

 

These conditions are not necessary to achieve a testing flow. You might want to learn about the product, catch bugs, gather test ideas or much more or everything at once. Feedback doesn’t need to be immediate, but generally is.

When testing a completely new system the opportunity is always overwhelmingly larger than capacity. The tester diminishes this feeling of being overwhelmed by using heuristics such as equivalence partitioning, but still, the options are infinite.

 

Flow as an aspect of Project Management

 

Probing the product in a state of flow is one way of going about testing. I see this concept in a different way as well.

 

Remembering the project management triangle used in many PM methodologies I found that Flow has a place in the
triangle.
As a tester your time should be apparent. You have X time to test the product.

Cost should be kept to a minimum. Testers are quite frequently seen as a cost or a necessary evil. It is our responsibility to show this misconception to be wrong.

Scope, however, is a fickle thing for any tester. Sure, you could start testing your requirements. Create a test case or a couple for every requirement in the list and flag off passed/failed. While this is by no means a complete testing strategy, it has been the nominator for way too many projects. As any tester knows: for every non-trivial system ,the testing possibilities are infinite.

A professional tester makes intelligent and skillful decisions to decide scope.

Some of these decisions can be made before starting the actual testing. But most are made while feeling out the project, product and while gathering information about what’s at hand. It is influenced by how you experience the process of “testing the waters”. What you see in the project, how you are responded to by your peers, by management, what you have learned about the product, how it is documented, how responsive the developers are,…

 

So, when have we  covered enough scope? Here’s where I think Flow is an important indicator.

It has been described as “the feeling of diminishing returns” or “the piñata heuristic”.
In short, “you stop hitting it if there aren’t coming enough bugs out of it anymore”.
Have you completed your scope? No? Do your Testing sessions give you enough return on investment? No? Can you bring yourself to go into the flow state with the product? No?

 

To me, that’s an indicator to go test something else, your work here is done…
For the moment.