The Nature of Computation

An advert showed up on my LinkedIN today.
This is what it said:

Dramatic reduction in manual effort. [The client] identified that automation is approximately 97% faster than their prior manual approach to quality assurance.

It had a link to an article in which a presentation could be viewed. I’m very interested in these adverts for I hope to someday find an honest, clear representation of how a test automation tool can actually be profitable to your project.

I once heard the following being stated: “People who believe in these statistic-based-sales-stories blindly, deserve losing their money on it.”
It was meant as a jest and I admit, I’ve added some more drama to it.

However, it is a deep running frustration of many testers that test automation is sold as automagic. Let me tell you a bit about computers.

Computers don’t think, computers don’t lie, computers don’t tell stories, computers don’t tell the truth, computers can’t give you advice and computers can not (yet) substitute for an intelligent, sentient human being.
They take numbers and they give numbers. They calculate.

It is in our nature to believe they can do more than that. There’s no magic in there, unless we put it there. But I don’t know any magicians.

Don’t get me wrong. I love automation. It helps me speed up the boring, simple, obvious checks that come with every daily build. There’s developers creating these checks, there’s testers creating other kinds of scripts. Most of the time, they give us useful information in the form of “none of the things that worked previously, are now broken”. Sometimes they fail and we get to check whether the script itself needs maintenance, or some functionality actually broke.

The 97% profit, or any time-wise comparisons are complete huey.
Did they calculate in the actual creation of the scripts? The maintenance? The infrastructure? The time it took the developers to adapt the application to accommodate the automation? The tooling cost?

Even if all of this was. Do the tests give the feedback you want? Does it report possible problems or does it give a false sense of quality? Can it replace anything?

I sincerely hope that the people handling the money, who take the decisions about implementing new tools, processes,… take a step back and talk to someone who can put these numbers in perspective.

Language as a medium

Software Testing

Communicating is one actor, the sender, trying to get something across to another actor, the receiver using any possible medium.
That “something” is the message, undefined as it is. This complete procedure is completely dependent on both of the actors, not just the sender or receiver.

There are tons of reasons how this can go wrong. I.e. emotional and attitudinal barriers can interfere.
Only a day ago I had to cool down a heated discussion between family members. A discussion that started largely because one person understood a word to mean one thing, while the other party meant the other.

Language and any form it comes in, (spoken, written, body,… language) is a very liable to miscommunication. Think about emoticons. These filled a huge gap in the Instant Messaging mania a decenia ago. Before these small yellow faces reared their expressive heads, kids would lose BFF-status because they couldn’t add a :), and a 😡 was interpreted.

For language to work you need both the sender and the receiver to try their hardest to interpret the message as it was meant by the other party.
Even then, other stimuli and noise can derail communication between the two.

This became, again, painfully clear this past sprint.
Context: Since deadlines are still king, and teams are ought to focus on “what’s visible for the customer”, other important non-visible tasks are postponed. This included Unit Tests and ‘focus on quality’ in all forms.
My two colleagues and me, who do all of the testing tasks, weren’t all to happy with this decision. We informed management that, if focussing on the deadline and trying to reach it by neglecting quality tasks, risk would go up, quality would go down and more time would have to be investing in righting a product that was built timely, but crooked.
They understood, but stayed on course. Straight ahead but slightly starboard to those rocks on the horizon.
Language had failed. Bugboard

We saw only one thing we could do: If we fear for bad quality, we have to find out just how bad things get and report.
We found bugs by the handful and pinned them on the wall. They did not like that.

Past retrospective, we were asked feedback on why so many bugs were on the wall and if the team could do anything to mitigate the time lost on bugfixing.
We had our answer ready.

Sometimes, we have to hit a wall, or distant sea-rocks, to realize the stars weren’t showing us the right course. Sometimes, we hit that wall together and sometimes we have to let others hit a wall.

It might not be the easiest or most pleasant of the media to be used for communication, but it’s definitely one of the more effective. Be sure to interpret it right and learn a thing or two.

Identity Crisis

Experience Reports, Software Testing

Everybody’s a tester.
Product owners, developers, stay-home moms, anyone who buys any kind of application,…
You have tested some kind of complex application before. Some of us do it professionally, others subscribe to a Crowd testing program and even others just buy an app, toy around with and discard it after seeing the first sign of issues.

If testing is so easy that anyone can do it, why are there still professional testers?
Just pick a person from the street, give him the specs and say: “Now Cross-check this with what we’ve made. You can read can you? Can you click? Alright, you’re set!”

A friend and me had a very detailed discussion today about testing in ‘an agile project’, without much further context.
He’s working as an iOS developer and my experience is mostly in big, slow, complete business solutions. There is quite a difference in perspective, but these kind of discussions are always interesting.

Apparently, in the projects he works in, there are no full-time Testers, nor functional or business analysts,… There are however a few ‘Proxy’ people working on these projects. To me this sounded completely foreign, but I assumed this was another word for a kind of Product Owner. The description kind of matched. A proxy is an FTE who writes the User Stories and specs, checks with the customer and tests the application.

The reason these people are called Proxies and why they do just about anything but develop is that they are easily replaceable. One proxy falls ill? Another leaves the company? No matter, we’ve got more lined up.

Resource = Resource
I find this disturbing. I see no added value in having everyone so easily replaceable and generalist. If I’d had to be this omnipotent IT-wizard that body-shifts from product owner to change manager to developer to tester to developer and back to tester and so on,… I’d be terrible at everything. There’s a reason why I became a professional tester and I am content to be one.
I’m terrified of a Proxy writing code for the specs she agreed to with the customer and afterwards testing it. (This sounds awfully 1960-ish to me.)

People are not easily interchangeable. It always comes at a cost of knowledge, convenience, team spirit, time and budget…
Imagine good old Larry who you’ve been working together with for 1,5 year, occasionally shared meals with and who you’d ask all questions regarding finance functionality to.
Now imagine him leaving the firm and being replaced by Anna, who’s worked in a bank for 20 years and has a unibrow. Are you sure you’d work with her as smoothly as with Larry?
Even a change within a project requires a flip in mindset, or maybe even in skillset if it requires you to switch language (Java –> .net?) or from technical to functional. It is incredibly tiring and time inefficient.

When talking purely theoretical, it’s easy to look at your colleagues as 40-ish hours a week. But it’s wrong.
You can very well be called a Proxy. You can even be a Proxy. Hell, I’ve been a Proxy for a few months. But I know I’m not a good Functional Analist. I know I’m certainly not a developer.

You are not your job description
I am first and foremost a tester, but can fulfill coördination tasks, analyst tasks, Client management tasks and many more, if required. I also know that I can not efficiently do development tasks, or Test automation that goes further than a replay, nor will I do ‘Testing tasks’ that some people would expect of a ‘standard testing profile’, such as writing test cases with test steps for every single thing you can think about.
Take advantage of the different people in your team, don’t categorize them by their job title.

I’m not saying I’m against the idea of Proxies, nor will I advocate specified roles per person are the way to go. I do believe that both could work in different contexts and that some kind of combination of the two will surely work as well.

Session based Example

Software Testing

This page serves as a lessons learned during the HvN tests of the KBI system as well as an example of how to conduct Session Based Testing in a setting that requires a low overhead test approach in which I endeavored to:

Get to know the product as soon as possible

Find the blocking bugs as well as any critical malfunctions before the client does
Follow up on where you spent your time and where to focus on next

Have a report ready the moment it is asked.

In order to achieve these goals I invested time in designing an excel sheet.
This sheet was supposed to give me the tool in which to register my test ideas and give me the opportunity to quickly turn an ‘overview page’ into a clear report.

This way of working was originally conceived by Sarah Teugels, afterwards discussed fiercely and eventually poured into a report by me.

This report, however, is flawed. Like any other report it can be optimized further but if I’d like to bring it to the next level this would require me to invest more time in it.

Time better spent in studying the functional documents, the product, the client, the designer, the development team,…

If you have ideas to better the tool, please feel free to let me know. Or take the ideas and adapt them to your context.

The report

I built the excel as one overview page where all Logical Modules are listed. The other tabs embody the Test ideas I gathered for a Logical Module.

| Overview Page | LM1 | LM2 | LM3 | … |

The Overview Page has these fields:

All fields are calculated, except for the ‘Major+’ column.

Logical Modules: I cannot tell you what Logical Modules should be in your situation. These can be derived from Requirements, grouped by the customer, separated by the Functional team and so on. You work with what suits your project best. In this example they represent a functional document. A group of Use Cases.

Weight: In consensus with the client or functional team, weigh the LM’s. Rate them from 1 to 10, or higher, or lower. Just make sure it reflects the test effort they are going to require. If “logging in” weighs 2 and “Sending an email” is 4, then you are going to put in twice as much test effort in sending emails than you are going to test the “logging in” functionality

Test Ideas Planned, Executed, ToDo: These Logical Modules will guide you in your test sessions. Keep in mind that these should not be your only guidance. An explorer dares to go off the path in front of him, climb high mountains and jumps over creeks. So should you while testing. A treasure is only a treasure if it hasn’t been found yet.

While doing these testing sessions, note down every possible idea you may have in the excel in the correct tab. These ideas you will follow in the next test session, or execute them right away.

This is what your test progress depends on. Test Ideas Planned minus the ones executed is your remaining, current, workload.

Mind the word ‘current’, because your work will never be finished. While testing I advise you to keep on listing the ideas that pop up. If nothing pours in, invest some time in reading a new technique, heuristic.

You could defocus and go on it a different way. Completely change the way you think about the product and look at it as someone else. These and tons of other aids can help you to list another myriad of test ideas.

Especially do this for the heavy-weight functionalities.

The alert function: The alert function compares the weight of the functionality against the total weight of the product and the number of test ideas executed of the functionality versus the total planned test ideas of the product.

Meaning: If the Weight % is bigger than the TI %, it alerts you to schedule more sessions. I’ve added a 20% margin in this comparison.

Use this to keep track of what you still need to do. Mind that the more test ideas you register for the low weight functionalities, the more you’ll have to come up with for the larger ones.

Major+: If your project isn’t all too concerned with issues that are lower than Major then this type of ‘filter’ can be useful for you.

If not, you can easily switch this out for another since this is the only field in the report that needs manual editing.

For my project, I included a column in which the number of issueswith priority Major or higher per Logical Module were shown.

The progress bar: Here is where the report could use some tweaking. It puts the Test Ideas ToDo versus the Test Ideas planned in a percentage, subtracting the number of Major+ issues.

This means a failed Test Idea resulting in a Major+ bug would subtract a percentage equal to 2 failed test cases.

This progress bar does have a lot of perks when used to present: Visual ‘progress’, percentages and colors.

You may find the XLS file here: Session based method example Overview SB

Flow

Software Testing

Flow as a testing activity

Flow is a state of mind. A psychological condition of being fully immersed in a feeling of involvement. It’s commonly referred to as “On a roll”, “to be on fire”, “Getting your groove on” and my favorite “Going beast mode”.

There’s a lot to read about this concept and most of it is very interesting to us testers. However, treat it as a heuristic, as we should most concepts. While a lot of principles about the idea apply to testing, some don’t match. I will explain on this further on.

I simulate this mindset by going into seclusion for an hour or two, connect my earphones and turn on a good Pink Floyd album. This conjures up a feeling of well-being for me and gives me the opportunity to have a 100% focus on the system before me, no sidetracks to anything else than the product. (sidetracks inside the system are highly recommended!)

It is possible to go in to great detail, create wonderful test ideas, analyze specific functions and test extremely efficient.

Doing this kind of testing usually gets great results. If you don’t experience the returns you would expect, you need a different approach. Go talk to a developer and get some needed insight for example.

While being wired in you can easily forget to stop. There are a few indicators on when to stop your testing session.

The feeling of diminishing returns
You don’t get surprised anymore
Your timing is done
Other features require your time and energy more
…

Pitfalls of Flow

“The hallmark of flow is a feeling of spontaneous joy, even rapture, while performing a task although flow is also described (below) as a deep focus on nothing but the activity – not even oneself or one’s emotions.”

Pasted from <http://en.wikipedia.org/wiki/Flow_(psychology)>

Emotions, feelings, self-awareness are very needed aspects while testing. Keep these constantly in mind while being in a state of flow. Do not lose these out of sight for you might miss important signals if you do.

Conditions to achieve flow:

– Goals are clear

– Feedback is immediate

– A balance between opportunity and capacity

Pasted from <http://en.wikipedia.org/wiki/Flow_(psychology)#Music>

These conditions are not necessary to achieve a testing flow. You might want to learn about the product, catch bugs, gather test ideas or much more or everything at once. Feedback doesn’t need to be immediate, but generally is.

When testing a completely new system the opportunity is always overwhelmingly larger than capacity. The tester diminishes this feeling of being overwhelmed by using heuristics such as equivalence partitioning, but still, the options are infinite.

Flow as an aspect of Project Management

Probing the product in a state of flow is one way of going about testing. I see this concept in a different way as well.

Remembering the project management triangle used in many PM methodologies I found that Flow has a place in the
triangle.
As a tester your time should be apparent. You have X time to test the product.

Cost should be kept to a minimum. Testers are quite frequently seen as a cost or a necessary evil. It is our responsibility to show this misconception to be wrong.

Scope, however, is a fickle thing for any tester. Sure, you could start testing your requirements. Create a test case or a couple for every requirement in the list and flag off passed/failed. While this is by no means a complete testing strategy, it has been the nominator for way too many projects. As any tester knows: for every non-trivial system ,the testing possibilities are infinite.

A professional tester makes intelligent and skillful decisions to decide scope.

Some of these decisions can be made before starting the actual testing. But most are made while feeling out the project, product and while gathering information about what’s at hand. It is influenced by how you experience the process of “testing the waters”. What you see in the project, how you are responded to by your peers, by management, what you have learned about the product, how it is documented, how responsive the developers are,…

So, when have we covered enough scope? Here’s where I think Flow is an important indicator.

It has been described as “the feeling of diminishing returns” or “the piñata heuristic”.
In short, “you stop hitting it if there aren’t coming enough bugs out of it anymore”.
Have you completed your scope? No? Do your Testing sessions give you enough return on investment? No? Can you bring yourself to go into the flow state with the product? No?

To me, that’s an indicator to go test something else, your work here is done…
For the moment.