The 136th time

Experience Reports, Software Testing

There’s so much ado about Test Automation I wouldn’t want to feel left out doing it.
For years now, we’ve been made to believe that test automation is the future of testing. For instance, no ‘manual testing’ would be needed because a machine would be able to do just as well allegedly.

I often feel the trending of test automation to be a dangerous thing, but today, it helped me greatly! I’m not a technical person, nor do I have the ambition to spend my days coding. I do have been experimenting for a while with tools. A Ruby course here, some tryouts with the SamouraiWTF virtual machine and so on.

Selenium IDE is a very easy to use, easily installable add-on and incredibly useful. It’s great for throw-away scripts but bad at anything else (like every record-and-play tool). Today I used it by recording a few steps to log in to the webapp, execute a query and consult record details. From those details I triggered the functionality “Go to next record”.
I had noticed before that this functionality behaved slower than others, but testing this by “clicking and waiting” was tedious. Some tweaks to the recorded steps by adding some waiting time and copy-pasting 2 relevant steps about 200 times, I had the script I needed.

Fired up the browser, pressed F12 for the developer tools, clicked play and went for coffee. The script ran for longer than this, but the beauty of selenium is that it can run in the background. On the 136’th time, the application client ran out of memory resulted by a javascript overload.

A good find and a good use of test automation.

Remember though:
It was a human having the test idea;
It was a human who had the hunch and how to pursue it;
It was a human who created the script;
And it was a human who investigated the results.

It took a few others to interpret the error and eventually fix it.

The Nature of Computation

Software Testing

An advert showed up on my LinkedIN today.
This is what it said: Automation1

Dramatic reduction in manual effort.  [The client] identified that automation is approximately 97% faster than their prior manual approach to quality assurance.

It had a link to an article in which a presentation could be viewed. I’m very interested in these adverts for I hope to someday find an honest, clear representation of how a test automation tool can actually be profitable to your project.

I once heard the following being stated: “People who believe in these statistic-based-sales-stories blindly, deserve losing their money on it.”
It was meant as a jest and I admit, I’ve added some more drama to it.

However, it is a deep running frustration of many testers that test automation is sold as automagic. Let me tell you a bit about computers.

Computers don’t think, computers don’t lie, computers don’t tell stories, computers don’t tell the truth, computers can’t give you advice and computers can not (yet) substitute for an intelligent, sentient human being.
They take numbers and they give numbers. They calculate.

It is in our nature to believe they can do more than that. There’s no magic in there, unless we put it there. But I don’t know any magicians.

Don’t get me wrong. I love automation. It helps me speed up the boring, simple, obvious checks that come with every daily build. There’s developers creating these checks, there’s testers creating other kinds of scripts. Most of the time, they give us useful information in the form of “none of the things that worked previously, are now broken”. Sometimes they fail and we get to check whether the script itself needs maintenance, or some functionality actually broke.

The 97% profit, or any time-wise comparisons are complete huey.
Did they calculate in the actual creation of the scripts? The maintenance? The infrastructure? The time it took the developers to adapt the application to accommodate the automation? The tooling cost?

Even if all of this was. Do the tests give the feedback you want? Does it report possible problems or does it give a false sense of quality? Can it replace anything?

I sincerely hope that the people handling the money, who take the decisions about implementing new tools, processes,… take a step back and talk to someone who can put these numbers in perspective.