Agreeing on uncertainty

Experience Reports, Software Testing

Coverage in Scope

In a software project, you can come across quite a few kinds of coverage. There’s Line coverage, Decision coverage, Use Case coverage, Branch coverage,… It makes sense to talk in terms of percentages when addressing these.

The above examples are descriptions of exhaustive, finite lists. The lists may not be complete, but somehow, by someone, somewhere these lists are conceived and decided upon.

They will grow in time, as new information becomes apparent and new important issues are uncovered.

This is important: the changes to these lists are manageable, even if they don’t seem to be at first. You can prioritize, as some of the items are more important than others. You start from what is apparent and deal with what comes your way. Use cases are added, lines are added, decisions are clarified, branches are checked,… It’s a work in progress and eventually the project will reach a point where change isn’t relevant anymore. Coping with that constant change is being agile.

Coverage Estimation

Suppose you have 1.000.000 lines of code. There are already tool-enabled checks in place that support 90% line coverage. Could you estimate how much time it would take to reach 95% line coverage?

It’s a straightforward question but a herculean task to gather information to offer as precisely as possible an estimate. Even then the solution is ambiguous  and the result uncertain. But you can give an informed estimate, if you desire. An educated guess, calculating in some margin for error and unforeseen difficulties.

Possible parameters  for line coverage estimation:

  • # of lines
  • Complexity of lines
  • Disparity of lines

In order to agree upon an estimation, the influencing parameters must be agreed upon. To agree upon a parameter, it must be known. Estimations are more than guesses, they are an agreement between two parties. One side calculates in the time for his or her planning, the other wants to get results within the borders of the estimate.

Test Coverage

By definition, testing is infinite. You have knowledge of only so many influencing parameters and there are many more in hiding.
Test Coverage, in percentages, is an absurd concept.

Consider the following proposition:

Will you work for me? It involves pushing buttons.

I won’t tell you how long, how many times, what happens when buttons are pushed, where you’ll be doing this, in which environment or whether it involves trained wild-life.

I’m guessing you find this absurd. Estimating Test Coverage is just as preposterous as you can’t put your finger on the infinite parameters flying around.
People, by nature, are unwilling to agree to a set of uncertainties.

However, if you, or anyone on your team is still hell-bent on getting estimates for testing, there are possibilities.

  • You both could agree on a set of known parameters. Keeping in mind that new ways will open up and temper with your estimates.
  • Or both parties could outline a model of the feature and work from there.
  • You could agree on spending X time on the feature and nothing more.
  • An initial hour or two could be estimated for a first probing session on which a clearer ‘estimate’ could be given.

All of the above are severely lacking in either meaningful coverage or estimation. There are many workarounds and different possibilities, but they are usually artificial and bound to escalate.

Estimates seem harmless at first, but:
They have an impact on people’s planning, frustrating them as estimates are shattered. They can hit a dent in your credibility as a tester.
They can severely implicate your freedom as a tester.

You need to be able to push where it hurts, being told where and how long you can test restricts this.

Chesting and Tecks

Software Testing

I had an amusing experience a few weeks ago. A colleague of mine was explaining how he was automating a regression test set and how it greatly enhanced their testing experience. I believed him, because I know he’s a professional worker with good testing and automating skills.
But I wanted to tease him a bit, as I’ve learned that he sometimes mixes up terminology and definitions.

I asked him: Today, I asked for an extract of a certain table from our database. I visualized it in excel, added some formulas and conditional markup. Next, I sorted on that markup and data errors were immediatley visible.

Was that automated testing or manual testing?

To many people, it’s a confusing question. Hell, it’s even confusing for me.
A better name might be “tool-enabled-checking/testing”. But this doesn’t clarify whether my example is testing or checking.

I noticed that we, testers, see checking and testing as surrogate terminology for automation and manual. This is not completely true, it’s too easy. There’s bound to be a gray area. A twilight zone where testing overflows in checks.

One tests a theory. My theory was that the table I selected might contain errors. I devised a formula in which I could show these mistakes. This is where theoretical testing flows over into the actual checking. The moment I start working in the excel sheet I’m beginning my practical checks. But checking is still a subset of testing. Be they tool-enabled or not, checking has to be done to either, prove or falsify your theories.

There’s a funny ebb and flow kind of feeling to testing and checking as they tend to alternate. While exploring your product you come across new ideas constantly. Further exploring ideas in depth or jumping to the next idea or even exploring ideas within ideas makes the whole concept extremely fluid.

Black on white, in the given example one goes into checking as soon as you start comparing your data to it’s expected result. But while you are doing it, you’re still testing. There is much verification, validation and checking in testing, but it’s the testing itself that’s interesting and which can result in infinitely more important paths to follow.

Be wary, that the above mentioned explainability heuristic does not hold into account false positives and negatives, human epiphanies (oh! That looks weird, let’s take a closer look) and more human errors by the analyst, developer, tester or anyone else whose had a say. It serves to show that “automated” is not the same as checking. It can be used to explain that automated checks are a subset of all possible checks and that all possible checks are a subset of Testing.

 

 

BugCoin

Experience Reports, Software Testing

Bitcoin has been around for some time now. In a nutshell: it’s a currency that is generated by software which was created by humans. It consists of 0’s and 1’s and is only as valuable as people deem it to be.

288_european_mantis_cf1 Guess what! So are bugs! Hopefully, they weren’t generated on purpose though… but they can be extremely valuable as a currency. Treat the following as a heuristic, it won’t work in every project and in some contexts it might even be hazardous.
It’s usually better to report bugs in a dry, clear manner while keeping the stakeholders feelings in mind. It’s important that you don’t antagonize anyone on your team, for that will impact your testing for the worse.

Sometimes, however… There’s moments and situations in which it is helpful that you have a certain power to influence the bug-flow. Either by increasing the priority, describing the bug more clearly so it gets picked up earlier, having someone implement a quick fix as a favor… There’s a ton of ways to influence this bugaboo and people might benefit from it. Sometimes you find your pockets bulge with BugCoin. Whilst pondering on this concept I found that I have:

  • Bartered bugs as salesware on a market in order to adhere to contracts;
  • Have complex bugs fixed fast as a powerplay show to prove effectiveness;
  • Pinned high amounts of bugs on a wall to show problematic quality;
  • Used bugs to get buy-in with stakeholders.

Most of the times, the easiest way to show added value as a tester is by finding important defects fast. There are cases where your insistent searching is dismissed as it being “just-your-job”.
Showing a handful of BugCoin can sway many people, just make sure they know its value. In my currenct context, this is how we get things done. Today I spent a few hours searching for a mysterious bug for the accounting department. I’ll make sure they’ll know where that coin came from. 😉 I’m sure this will come in handy sooner or later.

Kolb’s Testing Cycle

Software Testing

We’ve all been done a terrible wrong.

I wasn’t born at the time, but it must’ve been somewhere in the 60’s, when the first ‘software tester’ was named as such.
How did this came about? I can only imagine.
As so many innovations it was probably a scheme to make money where there wasn’t initially any.

From the first tester on, us professionals were branded “Tester” and no finer name exists. Because Testing is such a broad an interesting concept, that it’s really more of an aspiration, a goal than a thing you can fully master.

BUT! Somehow, through time, in people’s perception, the word “testing” degraded. Is it because it is associated with unwelcome surprises? Is it because everyone sometimes does it and can do it? Or maybe testing doesn’t invoke much imagination, digressing in a ‘checking culture’?

Testing is in the same league as “Playing”. A child can play with blocks and dolls, a group can perform a play at a theater. Everyone can do some form of playing. Sure, not everyone can play the piano. Everyone can learn though, if they are willing. Learning, just as testing and playing, is key.
This is an important similarity I would like to make.

Learning and Testing

One can not test, if you are not willing to learn. One can not learn if you don’t test.
They are not exactly the same, but they are immensely congruent.

Testing and learning have virtually the same process.
The chart in this post visualizes Kolb’s Learning Cycle and is part of every teacher’s education.
Everyone, while learning, is considered to go through each of the four stages.
You can begin at any stage, but will eventually complete the cycle. You’ll also spend more time on one or two of the stages as that will be your personal preferred learning style.

learning-kolb

 

 

I’m more of a do-er, focusing on Concrete Experience. This is how a typical test cycle usually goes for me:

 

1. Get hands-on experience with the product;
2. While or afterwards create some kind of mindmap of what I experienced;
3. Analyze where and how I could important defects fast;
4. Go back and find those buggers..

See how Testing and Learning go entwined? We could be named ‘Software learner’ or ‘Software player’ just as well. However, neither of those evoke any more awe than ‘tester’ in the general perception.
Personally, while I can’t effectively master testing as a whole, I can learn about it and hope that one day, I’ll be truly worthy of the name “Tester”.

We’ve all been done a terrible wrong.
We’ve been given a job title we can’t possibly fulfill.

 

Motivating intelligence

Software Testing

“Even if you’re not a teacher, be a teacher”
-Tim Minchin

Teaching, Learning, Mentoring, Listening, Coaching, Advising.
These continuous verb forms are ingrained into the testing field so heavily, it feels as if it is the single most valuable skill a tester should have.

A willingness to learn and to question, is perhaps more valuable than the cliché ‘attention to detail’ which you find on the top of 90% of testing vacancies. This latter has shifted a bit though, to automation.

As a youth, I aimed to become a teacher of English and Informatics to educate teenagers. During the few years I spent training to become a classic teacher, I found myself growing ever so bitter against the current school system. After a few years that kept me tormented over either “pursuing that certificate and walking between the lines” and “Quitting, but facing a life of uncertainty”, I passed a milestone. That milestone being a second consecutive fail.

It felt as if I had lost everything, but freed me up to take matters into my own hands.
That’s when I came into testing.
Where I felt imprisoned in the academic world, I found a second wind in the private business.

Freed from curricula and exams, you can pursue just about anything. Drop it, leave it some months and pick it up again, however you like.

I believe I’ve gotten in contact with many more intelligent people, who are willing to share knowledge, learned more and studied more in a year of working, than my 4 years of higher education.

The choices trap

There are many people who face a lifetime of possibilities. I believe it is a sickness of today, that we raise our children to always keep our options open. I believe in shutting doors. We can’t progress if we can’t find peace in choosing.

A few months ago, a friend asked me to contact his sister, who found herself in the same situation I was in. I haven’t ever seen her, but since last December we had regular contact through mails and skype. I sent her books, gave her exercises and followed her progression, giving feedback on anything she sent my way.

She’s been working as a test consultant for a month now. I was ecstatic! It felt good to give back.

Per Sholas, Speak Easy, Weekend Testing, uTest, Software Testing Club… Are organizations that are focused on teaching and learning both, specifically for software testing.

Whether you find yourself in the same situation, you can reach out and you will be helped. It helps when you try to produce literate sentences and expect only as much time from others, as you put in yourself.

If you wish to enter the Testing craft, but doubt you’ll like it/have the qualifications/need help with a job interview, don’t hesitate to contact me.
You don’t need paper to prove your intelligence, you need only demonstrate it.