Bad Testing is Worse than No Testing

Thus far I have written about how great test and quality work is. This is not the whole picture – far too often, quality work is not a pleasant experience. It is soul crushing to do a job that you know will end up having no impact at all. 

Sometimes Bad Testing even has a negative effect on quality. Allow me to list 3 Quality Antipatterns that will leave you worse off than when you started:

Bad Testability

Once I was in a team where the usual process went as follows:

  1. Developer tests the happy path, and it works. 
  2. Tester does some deeper testing, and it works. 
  3. Acceptance test is performed, and it works. 
  4. Feature immediately breaks in production. Every time. 

The reason? The environment and data used for development, test and acceptance differs significantly from the production environment. So whatever the team does to ensure quality is meaningless. The Bad Testability Antipattern creates a false sense of security.

Non-maintainable autotests

Speaking of a false sense of security, have you ever heard someone brag about their “100% Test Coverage”? This kind of talk smells of a dangerous mindset: 

  • that all tests are automatable 
  • that testability is static and not an issue
  • that there is a finite amount of test that can be performed
  • that we don’t need or want fallible humans to complicate our work

Autotests that are poorly designed are hard to maintain. Now you have ended up with a bigger workload, but no improvement in quality. Some autotests are permanently broken, and the team doesn’t question that. But worse, what about the autotests that run for years and never fail? What do they test in the first place?

Autotest is one of many testing tools, and it must be used with purpose and intent. 

Rebound effect

This antipattern is borrowed from the field of climate and energy efficiency. “The money I save by driving electric I spend on a trip to Thailand” leaves you with a negative climate rec´bound effect from the change. Do you recognize these thought patterns?

  • “I’m too stressed to write clean code. Let’s just commit this and hope QA catches and fixes it”
  • “I don’t understand this story, and also there’s a scary QA person who’s gonna evaluate my work and find out that I’m an impostor”

Thus it is very important to evaluate your quality interventions and look out for regressions in other places that may have been caused by a well-meaning change.

We all want to create value in our day job. Can you think of other quality antipatterns?

This Post Has 4 Comments

  1. Ben F

    Excellent article Emma, really hits home that we need to be smart when thinking about quality. I love the rebound effect especially, that’s new to me.

    Possibly another example is running types of tests that don’t matter. For example, running deep, lengthy user experience tests on an application that’s only going to be used a couple of times by IT systems administrators. You may find lots of user experience bugs, but IT systems administrators may not care about these as much as functionality or security, so it’s mostly time wasted in trying to improve user experience quality.

    1. Emma Lilliestam

      Good one! Digging deep and thoroughly in the wrong place.
      I just googled it, it’s called the Streetlight Effect.

  2. Barry

    Bad tests allow improvement. It means people are trying. No tests mean you don’t have a culture of testing. This is much harder to fix.

    1. Emma Lilliestam

      Thank you for your view. In my experience, it’s a lot harder to change an existing culture than to add one for a scope that people are aware of is missing. I can imagine other contexts, but I haven’t experienced them myself. From what I’ve seen, bad testing is a symptom of not wanting to see or act on relevant test findings, and lack of testing is a symptom of lack of know-how.

Leave a Reply