How To Improve Quality

Beren Van Daele, The Core of Things

“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”

― Abraham Lincoln

What’s next?

Another famous quote is: plans are worthless, but planning is everything. I’d argue that following through is at least as important. We used to only have 3 phases of RiskStorming. The Online version introduced a 4th: Commitments, Deliverables & Measurements.

We try to achieve two things in this phase:

  1. Broaden the backlog past Functionality
  2. Plan little steps to move forward on our Quality Journey
User Stories typically deal with functionality.

Past Functionality

Back in Phase 1, you identified 6 Quality Aspects of which Functionality is or isn’t one. In my experience, the team’s backlog is filled with functionality stories. There might be an odd security, performance or accessibility story or acceptance criteria or requirement interwoven in it, but that’s rather rare.

By picking at least 5 other Quality Aspects, identifying their priority issues and discussing possible mitigation steps you virtually built up several new ‘backlogs’ of work items that need some attention. Now I’m not suggesting you create 5 separate backlogs, or worse yet, start silo’ing these things up further in teams or roles or,… Please no.

Pick the highest priority issues, something of a minimum viable product and integrate them in your backlog/kanban board/requirement list,…
Don’t try to complete everything, you can’t, but tackle the most pressing things first. Convert these requirements/plans into work items for your backlog.

Move Forward On Your Journey

At the very beginning of your RiskStormingOnline workshop, you discussed where you are on your journey towards the a goal. In this stage, you can decide together how you’ll achieve the next step. Are you worried about performance being an issue?

  • Analysis: You might first want to define benchmarks for performance. What’s acceptable? What isn’t?
  • Operations: Can we check what the current average times are? What are bottlenecks?
  • Testing: How can we measure safely, smartly and reliably how the system will react under more stress, persistent load and large number of requests?
  • Development: Can we optimise our systems to make them more fluent? Do we need separate testing environments? Would they give realistic results?

Start with defining benchmarks and measuring what the current results are. Maybe ask a tester to explore and investigate potential problems and report back. Once you get a feel of whether things are satisfactory or not you can implement measures and drill deeper into the new problems, if they matter enough.

See how this ties into the heuristics, planning, strategising discussion from phase 3? Just like the squad that was caught behind enemy lines: First get your bearings, evaluate and scan your possibilities before taking a well thought-out action. …or take action and be ready to deal with the consequences.

How To Act For Quality?

Phase 4 is here to make things concrete. The discussions we have gone through to get here were invaluable. They gave you purpose, vision and means to achieve your goal. Here’s the next step to make them actionable, trackable and measurable.

  1. Make commitments
  2. Provide means to stay accountable
  3. Visualise your journey

Make Commitments

In Phase 4, we structure the information from the previous phases into ‘streams’. This means we see combinations of one Quality Aspect, a risk and it’s possible solutions. This gives you a great overview of what your team discussed.
From here, people or roles can make commitments to take action.

Compare it too action points following a retrospective. A retrospective without action points is a complaining session. A RiskStorming Session without action points can give direction, but has no plan.
Phase 4 helps you translate your discussion into actionable tasks, user stories, sticky-notes,… Any body of work you normally work with.

What you’ll be looking for is commitments from teammembers to take up part of the work that was discussed. Raising possible solutions but waiting for others to achieve them doesn’t help at all.

Accountability

As far as possible, define what success would look like. How does the person making the commitment and other teammembers know when they succeeded in their commitment? What are they measured against?
Measurements have a bad reputation in IT, but they can be very helpful when applied specifically and when the team carries them. In other words, when the team defines the measurements together, the measurements are valuable and acted upon.

Measurements can be numbers, but also feelings, happiness, an achievement, milestones, KPI’s, OKR’s, the result of a questionaire,…

Goodhart's Law for Data Science and what happens when a measure becomes a  target?

Visualise Your Journey

Over the years, I’ve experimented with several ways to visualise where we are with our quality.
I encourage you to be creative and try your own ideas, things that make sense in your context.

The important factor here is that you find a way to make your progress visible to people within the team. This might be more difficult in a remote setting, but with various communication tools you’ll find plenty of ideas to share your charts, models, walls,… I’m sure.

Here are a couple of ideas:

A one-page-test-plan to capture the most important ideas on one page. Especially highlighting issues the team may have which they can’t solve themselves.

A RiskMap to hang on a wall to show phase 1 and 2 of RiskStorming. Buy some red and green stickers to highlight on the map what is going well and what isn’t.

A wall of bugs with actual pictures of bugs to indicate the severity. Butterflies are trivial, scorpions are blockers. Start exterminating!

What’s next?

Hopefully, this has been a wonderful exercise for you and the team. Not only did you learn about how to deal with quality and risks, but you applied it to your current project and your current situation.

Among many other benefits, these are probably the

  • Work that was previously invisible are considered wasteful is now brougt out in the open.
  • You can track quality in a more meaningful way
  • Whenever a new teammember joins, walking them through the results will be very enlightening for them.
  • The shared understanding of your product and project by your team has greatly improved.

Now is the time to make the next step on your journey. I hope you feel much better prepared now.

Virtual Tolkien Is A Free VR Recreation Of An Iconic LOTR Film Scene
Good luck on your journey, traveller.

How To Discuss Solutions?

Beren Van Daele, The Core of Things

“If your only tool is a hammer then every problem looks like a nail.”
– Abraham Maslow

The Problem Solving Business

In IT, our job is to solve problems. Whether you look at the project as a whole, a requirement, a user story or potential risks to be tackled, there are problems all around us. Nobody has exactly done this thing before.

We’ve taken the first important steps previously: We’ve identified our problems  stating them in event, cause, result plus we have defined what is important to us.

“A problem well stated is a problem half solved.”

– John Dewey

Moving onwards, two important facts need to be understood when trying to plan or strategise for solutions to our problems:

Everything is heuristic & Sometimes, the best solution is to fail gracefully

Anything we do to solve problems is bound to be heuristic. A heuristic has the potential to both succeed and fail. The outcome is mainly influenced by experience and expertise of the person or team using it, their ability to estimate their current situation and the future situation(s) they can reach by applying the heuristic.

Being great at chess means you’ve absorbed a number of heuristics and know how to apply them succesfully.

An example: A squad of soldiers find themselves behind enemy lines. Several heuristics they could employ are:

  • Find higher ground to get bearing of their position and surroundings;
  • Try to establish connection with homebase via technology;
  • Use stealth or decoys to find their way back;
  • Fight their way back as quickly as possible;
  • Do the exact oposite and push through to a safe position and reevaluate.

These aren’t all mutually exclusive. Some logically follow up one after another. Sometimes you’ll have several successful outcomes, only to fail in the end, like having the upper-hand in a chessmatch, only to lose in the end.

Plans & Strategies

The difference between a plan and a strategy is contingency. A plan is like a booklet to put together an IKEA table. Don’t get creative with that stuff, just follow the plan or it will likely not work.
You can have a meeting with several people and come out having a step-by-step plan to release your product to production. If any of the steps fail, you’ll likely have another meeting to patch it up.
In Chess, I often open with a 4-step-to-checkmate plan. These are 4 steps I need to carefully take in the beginning of the game, without miss and without having my opponent shortcutting it. This is a plan which is part of my strategy. When it fails, I need to start thinking in terms of contingency and build a new plan.

A strategy can almost be seen as a set of plans, each building upon one another or providing alternatives when something goes bad. Build strategies based on your experiences, expertise and skills by discussing them among each other, learning from failures, extrapolating what worked and sharpening what didn’t.

Mitigating Risks by Implementing Solutions

Following How to Identify Risks? you’ve defined what’s important and what could impact your product negatively. Now, you’ll want to start making sure those bad things, don’t happen. We’ve moved away from an over focus on functionality and are now broadening the Team’s idea of which activities influence Quality as a whole.
You can not solve these problems with a tester doing test cases. If this was a big part of your project’s ‘strategy’ you’ll come a long way by the end of this phase.
Here’s the step-by-step:

  1. Gather people with specialities in: analysing, building, testing and operating
  2. For each Risk, discuss what possible solutions are available
  3. Find a balance between roles and activities

Gather people With Different Specialities

Oh yes, you’ll be tired of reading about this step by now, so I’ll keep it short.
If your RiskStorming Phase 3 consists of only carpenters, all your risks will look like a nail for which a hammer will be the solution. Testers will test, developers will craft code, analysts will analyse,… It would be sad to have come all this way in RiskStorming, only to discuss one facet of influencing quality.

However, doing RiskStorming within a group of like-skilled people could be benefitial as a learning opportunity, such as within a community of practice.

RiskStorming Phase 3

For Each Risk, match possible solutions

In RiskStormingOnline.com we offer a grand 120 possible activities that can help you tackle your risks. They include Techniques, Heuristics, Patterns, Observability and Dealing with Change cards that will help you connect possibilities to problems. In the future, we’ll structure the cards differently to match the skillsets: analysing, building, testing and operating. For now, you can flip, scroll & search through many ideas and prompts that will help you tackle risks.

The idea is to find common ground between Team members. ‘oh, you’d like to do stress testing? I can see an issue that we don’t have a controlled and isolated environment for that. Maybe I can help set that up!’ & ‘That’s a great idea! I’ll activate extra logging so we can learn more.’,…

The focus is on sharing ideas, extending helping hands and finding comon ground. If people have been saying ‘Quality is a Team Responsibility’, then this is the moment it actually takes shape.
Only now have your teammembers understood sufficiently what quality actually means and how they can work together, from within their roles, to achieve it.

Obama’s famous Mic-Drop

Find Balance

If your solutions are mainly showing a similar colour, that means your solutions are too one-sided. Try to balance out the colours and discuss how one failing activity might be supported by a different activity.

E.g. If we don’t catch issues during the story mapping, we might find them during our End-to-end UI automation or when our testers go through the business scenario’s. If all that fails, we might catch the inconsistencies during beta-testing during our Dark Release or during Real User Monitoring.

Find balance. Harmony.

Model by Dan Ashby, adapted and coloured by Janet Gregory

What’s next?

You’ve gone through the three most important stages of RiskStorming:

  1. What’s important to us?
  2. What Risks will impact that negatively?
  3. How can we deal with those Risks?

Lots of ideas and discussions float around, supported by cards, hopefully some epiphanies as well.
You’re probably lacking something concrete at this point. We’ll explore this in a next post.

How to Identify Risks?

Beren Van Daele, The Core of Things

Risks are things that keep people up at night. Not just you, anyone. Taking a risks can make you feel alive, scared, overwhelmed, excited,…

Risks are based on Feelings

People often try to quantify risks. Any project with higher compliance requirements will tell you they estimate Likelihood and Impact, that they then put those in matrix and show which ones are minor, major or critical risks. Apart from helping you pass an audit, these practices also look nice and make you feel safer…
Feel safer’. No matter which process you use to quantify risks, it won’t ever remove that it’s all based on feelings. It’s a crucial piece of insight you need when discussing Risks: whatever you think, predict, foresee, expect,… is all based on feelings.

You’re a rope walker, who needs the right amount of skills, confidence and situational awareness to survive. Fear will paralyse you, overconfidence will unbalance you.

An actual rope walker

Trade-offs and Feelings

Remember this very same header in the “defining quality” section? It’s simple: Risks are the reverse of the quality medal. In a world where you’d have infinite resources you would still need do the same balancing act. They are two different ways to think and talk about the same thing. Ying & Yang, but dealing with them is exactly the same.
From a Quality point of view, in order to increase the fluency and flow for the User Friendliness aspect you gather data to find out where people leave, are blocked or face issues. In the same sense, added monitoring and user tracking, which might help you identify and deal with risks quicker, might infringe on people’s perception of privacy.

Ugh. Feelings? Balancing acts? Can’t measure or quantify any of this? Bah!

– Many people in IT

If you can accept this though, it will help you understand that the things we do in our field to deal with these problems are entirely beside the question. Without going too far into detail, there’s this model called Cynefin. It teaches us that different types of problems need different types of solutions.

If you believe Likelihood x Impact = Importance and your next step is checking several boxes to show the risk is dealt with, you believe the problem space is either Obvious or Complicated.
If you believe that risks are based on feelings which are different on who you ask, are bound to change and might need several possible and probable solutions you may or may not keep or discard, you believe your problem space is Complex or Chaotic

The important distinction is that you understand that nobody has done exactly the same thing as you are right now. There are no best practices, no guidelines, no experts who have done the exact same thing again. Almost by default, your first realisation should be that you are lacking information.

Don’t get into Cynefin right now, read this post first, ok?

Defining Risks

This blogpost is part of a series that uses RiskStormingOnline.com to structure a discussion towards a better understanding of quality, risks and what to do with them.

Following Phase 1: Defining Quality you should have a selection of important Quality Aspects, but more importantly: a better and shared understanding of what’s important for your product and project. This is an excellent starting point to explore and identify potential risks.

  1. Gather people with specialities in: analysing, building, testing and operating
  2. For each Quality Aspect, One by one:
    • Think about 2 to 3 ways it might be impacted negatively
    • Write down your ideas
    • Include for every risk: Event, Cause, Result
  3. Have an open discussion about Risks

Gather people with different specialities

Traditionally, someone doing Risk Analysis would create a spreadsheet, make a list of all the risks they could think of and then consult the following people:

  • A Business Analyst to add a number to each risk for Impact. Something like: 1 if there’s almost no loss of money if the risk would happen and 5 if we’d lose a lot of money.
  • A lead Developer or Architect to do the same number-game but for Likelihood. 1 if it’s very unlikely the risk would happen, 5 if it could happen any day.

I’m not entirely against this process and especially like the interaction with other roles. However, some things we want to change in RiskStorming:

  • Include more specialities into the discussion such as testing & operations
  • Invite more people to quantify Likelihood & Impact values
  • Use the values, not as goals, but as discussion points

Writing down Risks

“If you can’t think of three things that might go wrong with your plans, then there’s something wrong with your thinking.”

Gerald Weinberg – Secrets of Consulting

I’m less strict than the famous Mr. Weinberg and will only ask you for at least two examples of how your plans could go wrong. Yet, seriously take your time to think in different angles about your product and project. Testers are particularly trained in this skill, give them some input and see which questions they come up with.

Once you identified an idea of how something could go wrong it’s not sufficient to just note down ‘Hackers get in and steal our data!’. Though this is a good starting point to discuss further, try to specify three aspects for each risk:

  1. Cause: There are inconsistencies between layers of the application, leaving an exploitable gap
  2. Event: Someone with malicious intent gains access to the inner workings of our system
  3. Result: Data is stolen, removed or worse. We may face lawsuits, lose users and loss of business.

Exploring these three aspects to a risk gives you and the team a better sense of the gravity, possibility and consequences of the problem. You’ll be better informed as well as better prepared to take countermeasures.

Have an open discussion about Risks

Working together, you have now identified several risks that are actually tied to quality aspects that are important. They are well understood and discussed by the team. Nobody should later blush and say ‘oh, I didn’t know that was important’.

For each risk, people who can make an educated guess can specify a likelihood & impact quantifier, others can leave it to ‘?’. These values can help you prioritise, group or use them as a basis for your compliance process.

What’s next?

If you don’t do anything else with RiskStorming at all, having done Phase 1 and this phase with your team, understanding the lessons and taking heed to what you discussed, you’ve taken a huge step forward to building a successful product.

All Skyrim Enemies Lord of the Rings music 'Bridge of Khazad Dum' - YouTube
Some risks are scarier than others. Don’t have a wizard? Do RiskStorming Phase 3!

Now that you’ve identified the most probable and dangerous threats on your journey, you’re better prepared and equiped to deal with them. Phase 3 offers your team a whole lot of prompts and ideas to deal with them. Figure out what works for you, discuss and plan.

 

Read all posts made in this category here: RiskStormingOnline Knowledge Center

How to Define Quality?

Beren Van Daele, The Core of Things

‘We need to Improve our Quality!’, ‘We have bad Quality!’, ‘We sell a tool that measures Quality!’, ‘I’ll add some tests to improve Quality!’

These are just a fraction of sentences I’ve picked up over the years that make NO SENSE.

Quality is conflict

Quality is a word that creates divisiveness. It creates conflict and controversy. However, not all conflict is bad and this particular one is looming over our heads whether we face it or stick our head in the sand. You will pay a price.
Freedom, is a very similar word that lets us dream of utopic scenarios, yet suffers the same boundaries as Quality does. Both concepts find their limitations as soon as you bounce into someone else. When we permit ourselves a certain luxuary, it will more than likely impact someone negatively. Quality works similar.

Freedom, to me, is walking on a mountain without a care and nobody around me. How do I feel about someone claiming that piece of land for themselves and building a hotel there?

Quality, to me, is a smooth experience with no hassles, no passwords or extra verification. How do I feel about someone stealing, selling or even just viewing my data?

Trade-offs and Feelings

Both situations impact me directly, one way or another. A hotel can offer me comfort, lunch and ease-of-access. It also implies many more people can enjoy nature. It’s a trade-off. There is no right or wrong. There is only right or wrong in context. Additionally, what you base yourself on is a personal feeling. This feeling is influenced by your own personal values, principles, maybe even some data, but it remains your personal feeling.

Cycling Lac d’Iseo

Defining Quality

This is how I tackle Quality discussions. It is a step-by-step backbone that I will almost always follow. How each of these steps take form, how they evolve, who I include are very situational. However, this backbone is usually a firm pillar to lean on. This is also why my product RiskStormingOnline.com follows these steps and enables you to achieve the same in an interactive and structured manner.

  1. Gather stakeholders
  2. Understand your context deeply
  3. Break up ‘Quality’ into more defined Quality Aspects
  4. Visualise Quality Aspects as Journeys

Gathering Stakeholders

People who matter. These can be users, teammembers, people in Marketing, Salespeople, people who wish to make profit from the product, external experts,…
Pay special attention to diversity and empathy.

Clippy Didn't Just Annoy You — He Changed the World

A diverse set of people with different skills, backgrounds, goals and dreams, demograpics, mindset,… will help you tackle your biases, blind spots and other points of failure. History provides us with many product failures due to biases. Example: Clippy, the ‘friendly’ digital assistant in Microsoft Office. During market focus groups, women who found the animations creepy and leering, were ignored.

Empathy is a seriously underestimated skill during these discussions. The ability to pick up when someone else would like to pitch in or feeling when someone else might have a better supported opinion than you might have is an amazing help. When discussions about Quality, or any discussions actually, become a shouting match, my advice is to walk out. Yet even then empathy can help diffuse the situation with calm explanations and reasoning.
Additionally, empathy enables someone to take the point of view from people outside of the discussion. ‘Oh, I don’t think Laura from opperations would like this.’

Understand your context deeply

Context is another word that is difficult to grasp due to its many, many facets. In Tech environments context usually means ‘everything around our product, project and happy team’. Lots of people are happy to work within the bubble of the project and leave the ‘working with the outside’ to managers, product owners, marketing or sales departments and do their own thing without truely understanding the ‘why’.

We’re not just building a product, we’re solving a problem. Someone, somewhere will be impacted by what we do. We need to understand our goals, failures and how we impact them personally within our context. If we don’t, we’re part of the problem.

This coin has a second side though, people on the outside can’t just treat the project as a black box and expect ‘quality’ to come out. This is one of the conflicts we need to resolve: Leaving our comfort zones and finding common ground, common language.

Our new product: RiskStormingOnline.com

Breaking up Quality into Quality Aspects

In my experience as a Product Owner and working with other Product Owners I noticed a huge flaw. We are obsessed with functionality and have little patience for other quality aspects. Many people around us know this, but feel that their voice isn’t strong enough or that it isn’t their role to provide direction.
This results in a backlog filled with Functionality stories, with the occasional security, performance story or requirement.

This puts all discussions about important stuff that isn’t functionality on at least the second place. That is, until problems arrive… and they do.

With RiskStormingOnline.com, we provide a simple tool to select and prioritise Quality Aspects. We offer 25 Quality Aspects that are designed to initiate discussion. Functionality is only 1 out of these 25 cards.
We introduce a limit of 6 cards to be chosen to encourage stakeholders to make these trade-offs.

Phase 1 of RiskStormingOnline.com

Visualise Quality Aspects as Journeys

The important thing about this phase is the discussion, the exploring of different points of view, elaborating on their importance and making decisions on what really matters.

I like to think of these Quality Aspects as journeys. The easiest one is probably Functionality, since we’re usually quite familiar with this one. We often break this Quality Aspect down to a Minimum Viable Product. Maybe we have ‘a finalised product’ in mind. What we can usually agree with, is that we’re not at our destination yet. We still have steps to take. This next step is often the topmost user story of our backlog.

We need to keep in mind a similar way of thinking for the other Quality Aspects. What does the final destination of Security look like? What is the next step we can take?
Very often, we don’t really know what the next step is, what the destination is or sometimes even how to begin the journey.

At this point, you might think that this isn’t helpful?
On the contrary. Knowing what you don’t know is often more important that what you do know. It gives you the opportunity to learn, adapt and evolve.

Knowledge is power, France is bacon

– Lard_Baron

What’s next?

You’ve just had a discussion on what is actually important to the success of your project and your product. People’s ideas were challenged and are hopefully more aligned. Questions were answered, misconceptions straightend. But…

There’s still more questions to ask!

Lucky you. You’re on a journey, you’re learning and you’ve found people who accompany you.
Adventurers of old applaud you! Yet they warn you:

Fellowship of the Ring (Group) | Middle-Earth Films Wiki | Fandom
Adventurers from Lord of The Rings applaud you

No journey is without Risks.
Our next step will explore possible risks from as many angles as we can.

A well-informed and prepared (wo)men is worth two.

 

Read all posts made in this category here: RiskStormingOnline Knowledge Center

Launching Your Own Digital Product

Beren Van Daele, Product Ownership

At the start of my Product Owning journey a mentor gave me his number One tip:

Treat the product as if the budget came out of your own pocket.

I thought I understood him back them. Turns out I was wrong.

22550410_1153441251456855_4267502031610269933_o

TestSpere used in RiskStorming

What does it take?

After months of hard work, we’re launching a new product in a couple of days.
It’s an online product that helps team collaborate on their understanding of quality, identify risks and how they can work together to achieve said quality and battle the risks, together.

Sure, we started coding some moths ago, but it actually started in 2017 when I launched a card deck called TestSphere with Ministry of Testing. Months after that, RiskStorming became a workshop which used the cards. Two years later after the workshop improved and improved over time AND gathered more ‘believers’ we decided to build it into an online tool.
It seemed logical to do so when the whole world suddenly went remote for obvious reasons.

Things working for us:

  • The workshop format was mature already
  • The userbase was modest but vocal and enthusiastic
  • The product adds immediate and clear value

To be invested:

  • 20.000 EUR and counting. (1 fulltime dev + several part-time freelancers)
  • Countless hours of my own time & headspace

Possible reasons of failure

  • Confirmation Bias bubble: People may be vocal, but this doesn’t translate to sales
  • User feedback wasn’t reflecting needs of the larger group of possible users

How do I feel about this?

A strange mixture of pride, fear, anxiety and excitement. It is my own budget, my idea-baby and my personal feeling of success/failure on the line.
When you have this much of your own skin in the game, your project fills you with both happiness and dread, depending on the day or even the hour. Sometimes I have stress-induced headaches spanning several days. Sometimes I feel like I can take on the world.

It’s a strange but exciting journey. Even in the worst case scenario, I may lose some material and selfconfidence, but I’ll have learned to be a more emphatic leader and human being.

If you’re in product ownership, management, leadership, I dare you to fund your own project. You may curse me for it afterwards, but you’ll have grown considerably.


The Product?

Demo

Early version of RiskStormingOnline

If you want to know more about what I think was worth the investment and effort, read on.

RiskStormingOnline is for you if:

  • You believe your team can’t define Quality as a group.
  • You suspect there are highly important risks that are being ignored
  • There is no clear strategy or plan to build in Quality from the start

We launch on the 3rd of September 2020.
Interested? Sign up here