Words are important
I often practice good and thoughtful wording. We’re in the business of gathering information and supplying it. Working in a business where most people have no deep understanding of what their colleagues do, we need to be able to clarify, even translate information in a way that our recipient can understand the message optimally.
This is no easy feat and requires much practice. That’s why I train myself in it and you should too. At some point this summer, the idea popped into my head that “Test should never be used as a noun”.
Since then I’ve had the pleasure to discuss this on Twitter and Skype with Chris(@Kinofrost, QualityExcellentness), Patrick Prill (@TestPappy), Sarah Teugels(@Sarah Teugels) and James (@rightsaidjames).
This is the conclusion I have come to. (which is not necessarily, what the others have found.)
Shaping of ‘a test’
Consider these two definitions (from Context Driven Testing – James bach)
“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”
“A test is an instance of testing.”
To me it felt that the second definition did not bring any value but more likely supported confusion and abuse of the term.
- ‘A test’ is a countable noun. Which makes it dangerous for (non-)testers who like to attach meaning and value to numbers.
- ‘A test’ is easily confused with ‘a check’. It’s much too easy to substitute the difficult to comprehend concept with a more simple one. The question: “Could this information impact value to anyone who matters?” is forgotten and replaced with “Can I push this button and is the background blue?”. ‘A test’ involves human interaction, tacit knowledge, interpreting results,…
- Calling your activity ‘a test’ implicates that it is a thing. Things are tangible and generally mean the same thing. A dog means a dog, even though there’s a few hundred different races. You can further specify what kind of dog it is you’re talking about.
‘A test’ involves tacit knowledge, which can’t as easily be expressed.
In addition, calling it a thing, gives the illusion that it can be repeated. You don’t want other people to misunderstand it that way.
I have been convinced by the aforementioned people that there are indeed cases in which ‘a test’ can be a valuable. It comes with a few caveats though.
– It’s an activity; like ‘a run’ or ‘a play’ it isn’t specific about anything, but gives notion of a certain activity.
– It’s focused on finding important information (to a person who matters).
– It’s requires human intelligence. (tacit and explicit knowledge)
– It’s boundaries are irrelevant. Where it started and where it ended are not important to the concept.
We’ve come up with the following definition, trying to stay true to the nature of Context Driven Testing:
“A test is a human-controlled activity, any activity, that is focused on discovering information that is important to a person who matters”
Which is inclusive with: “Quality is value to some person who matters.”
Conclusion
“When you use ‘test’ as a noun, find a better way to phrase it.”
… and if you don’t, be mindful of what it represents.
Which has it’s exceptions of course, like every heuristic.
I’m open for further discussion, but Twitter didn’t take kind to this blog post format.
Isle of IT 
Hi Beren!
After giving your postulations some thought, I mostly agree with you and suffer still every day from managers and colleagues that got trapped into the easy confusions that tests are pre-chewed procedures, aimed on validation of Software specs. That tests can be counted and that the amount of tests somehow indicates a progress or even better a coverage of any kind.
At first I thought I lived in the future…lately I start seeing it more as I’m living in a parallel reality. For some, the future will never come.
A test is in my opinion an activity that should be repeatable as from the moment you start testing. A test focuses on finding issues, problems and/or inconsistencies and helps you understanding the reality of the application under test. Counting tests is as useful as counting clouds if you try to make conclusions out of it to predict future testing progress or to obtain a quality indication. Counting tests within a certain focus area vs time spent might be a heuristic useful to evaluate your progress and use it in combination with other things you learn, to decide to stop or continue testing.
Testing as such is therefore also not a process but a craftmanship.
LikeLike
Always a pleasure hearing from you, Mike!
I do hope that the situation for testing is changing for the better.
At least at my current project, we aren’t asked for charts or lists anymore. We’re asked for what we think about the product.
That’s because the people who need to know the important details will already have been informed and the people asking want to know whether their high level planning might be impacted.
There’s two things in your comment that had me wondering:
1. “A test is in my opinion an activity that should be repeatable as from the moment you start testing.”
–> Could you elaborate?
2. “Counting tests within a certain focus area vs time spent might be a heuristic useful to evaluate your progress”
–> I’m not sure that counting tests here gives any extra value on top of the “time spent on feature X”.
Is there a difference between these reports?
– Coverage of feature X = 2h spent testing on feature X
– Coverage of feature X = 10 tests for 2h on feature X
I think both need some kind of in depth explanation of what was tested, what wasn’t tested and what was found. Those three are much more important than adding the number of tests to the equation.
Counting tests to measure progress might be useful in some cases (heuristic), but I wouldn’t mindfully put it out there. It’s a concept that backfires quite easily.
LikeLike
Hi Beren,
Thanks for considering and thinking about my comments.
To come back to your questions
1. “A test is in my opinion an activity that should be repeatable as from the moment you start testing.”
–> Could you elaborate?
Once you start testing, it is important you keep track of what you did so that once you run into a something, you know how you got there and you can repeat it.
2. “Counting tests within a certain focus area vs time spent might be a heuristic useful to evaluate your progress”
–> I’m not sure that counting tests here gives any extra value on top of the “time spent on feature X”.
I think it can help, but you will need to put these numbers in a context for them to have meaning.
To keep track of where you are, you could in certain situations keep count of the different paths you already walked, to find out if it is worth testing further.
Imagine you are looking into boundaries of different parameters of a certain feature and you tried different combinations of boundaries and tested interdependencies for less than half of your settings without finding a bug, you might want to evaluate if it is worth continuing to test this way. Maybe it’s a good time to start testing from a different angle.
I don’t think we should stop counting because of testers and managers that think counting tests provide unambiguous reports. Counting can be useful to keep some kind of track of where you are within a test session you conduct.
“Is there a difference between these reports?
– Coverage of feature X = 2h spent testing on feature X
– Coverage of feature X = 10 tests for 2h on feature X”
This comparison is not the one I had in mind when I talk about the usefulness to count.
I would rather report:
In 2 h I tested basic and complex deviatons of 20 parameters of a feature. I decided to test each parameter more or less in equal patterns. So I performed about 5 different but for every parameter similar tests. After testing every parameter, I tested some interdependencies with other paramerers. I decided to do 3 interdependency tests per parameter and tried to always make new combinations so that eventually 20 tested parameters have all been combined with one another and about 60 parameters were indirectly triggered trough those onterdependency tests. The feauture under test has 75 parameters. So I did not look at 55 of them in details and did test far from all combinations possible. However, I did touch 60 parameters and tested 20 more thoroughly. Until now I did not find a single bug. I have 6 hours left and things to test are integrations with other features, layout, browser support, usability and I have to increase my regression set. I did notice some browser related issues as I needed to switch browsers to continue testing. Also did it seem that some tweaks I made, made response times slower. Need to look into that in more details.
Based on this knowledge I decide my testing to be ineffective if I continue with what I was doing – The chance I will find issues here is low. I better focus on larger integrations as well as on performance and browsers.
To say I tested 2 hours tells a lot less about what I did and more importantly, will not test and makes it hard to decide to stop or continue testing – So I think you need a report. And that report can contain some numbers of tests you did, features you tested, what you tested and what you did not test.
As I like metaphores, I would say: I think it’s not wise not use wood while constructing a house, because houses made of stone are better 🙂
Cheers!
Mike
LikeLike
I agree mostly.
It is indeed the journey and the story about the journey that matter. The numbers can be used to support the story or even as an explainability heuristic.
It is, however, important to consider that including such numbers in your report can be misinterpreted and misused by people reading your report. Maybe these people aren’t in direct communication with you, but can influence your work. This makes it dangerous.
I believe that the numbers you talk about could better be described as checks. Checks are countable and useful to count, but shouldn’t be confused with testing.
While I’m sure it’s often interesting to count certain aspects of your testing and logging these numbers, I’m unsure reporting this holds much value. Trading off the added value tot the risk of misuse doesn’t make it worth it to me.
A stone house might be stronger, but it’s definitely more expensive, takes longer to build and who’s saying you’re not moving anytime soon? 😉
LikeLike