Chapter 2 - Do we need QA?

Why we don’t need QA

Note

In this chapter we use the term “QA” to mean Manual Testing. This does not include UAT which is considered distinct from QA.

Many organizations who have instigated some level of regimented software testing will have a split between automated tests and manual tests operated by a QA (Quality Assurance) department.

This seems like a sensible solution:

  • Create automated (unit) tests to ensure discrete, independent components of the system operate correctly.
  • Create integration tests to ensure components work together as expected and deliver final products to QA staff who pull all the levers and push all the buttons in an effort to prove the tests wrong.

In theory this should work well, but in practice it can have a significant negative effect on the overall process of creating software, for a range of reasons.

Manual testing doesn’t scale

A traditional approach to QA testing is often characterized by a “script” which a human will follow in order to make a determination that a software system is “working correctly”. This usually consists of a set of steps that someone will follow each and every time an application enters its “testing” phase. This can often be a good way to identify software faults but is incredibly labor intensive and does not scale well. As features and/or complexity in the software increases the task of the tester(s) gets bigger and bigger, but there is little or no capability to reuse the previous work done testing. Every time a new feature is added, no matter how small, the entire application must be re-tested to ensure there are no regression faults that have been introduced. This is of course also true of automated tests but clearly the difference is in the time is takes. Automated tests typically run in a matter of seconds whereas manual testing can consume hours if not days.

Manual testing requires humans, who make mistakes

Any system that involves humans will be prone to error and both automated and manual testing systems will suffer from this, but manual tests are particularly problematic in this regard. The onerous task of having to retest the same product features over and over again can cause mild psychosis in even the most cognitively hardy tester. Over time the repeated testing of the same feature that always seems to yield a “success” result can lead to complacency. The tester will often skip over or rush through components of the application that consistently pass their tests even though the chances of a particular addition/change affecting that feature is not known to the tester. This means that no two tests are ever the same resulting in an overall drop in confidence surrounding the entire test process. Additionally some testers will be more adept than others at both locating bugs and communicating them effectively back to the engineers. This further encumbers the process with unwanted barriers and delays. Finally whatever script is used by the tester is just as prone to error by omission as an automated test. That is, there is no guarantee that the script followed by the tester covers all areas of the application and crucially may miss those areas in which bugs are present.

Engineers won’t test if they don’t have to

Although some environments may proactively require engineers to write automated tests to “cover” all the code they write, environments without such stringent measures run a risk of engineer complacency. Rather than creating an automated test and engineer might be tempted to assume “QA will pick up any problems”. This may be true, but the time lost through iterating in and out of QA as bugs are discovered can quickly derail a project time line.

Why we DO need UAT

Despite the premise of this discussion being in support of automated testing, a comprehensive software development methodology would be incomplete with only automated tests. The positive impact of humans in the process should not be overlooked and there are several reasons why the role of humans cannot be easily duplicated or automated.

The key distinction here is that QA is not the same as UAT (User Acceptance Testing) which can, and should be adopted for testing usability. That is, does the function do what the user expected it would, is it easy to access & understand etc. These are more tests of the way the software is defined rather than testing that the code works.

Importantly however the UAT role does not necessarily have to be filled by internal staff. A staged release cycle where external beta testers are engaged can often yield the same (or greater) value in terms of identifying potential usability problems.

Key point:

User Acceptance Testing is important but should not be used to identify bugs and can be significantly streamlined through the introduction of staged releases.

The whole is greater than the sum

Engineers are often surprised by how end users interact with the software. They will regularly do things that weren’t predicted or documented and this can often be an area that demonstrates an abundance of hitherto undiscovered bugs. The most effective way to identify these areas remains exposure to real users, however this does not necessarily mean a return to the QA/Manual Test cycle in the development process. A crowd sourced approach to this problem is by far the most effective solution both from a cost and coverage perspective. This is discussed more in Scalable Manual Testing.

Engineers are not always right

An engineer working in isolation may be able to create amazingly complex solutions that have a series of automated tests which ensure that what they created works perfectly in every conceivable situation, and yet this solution can still be completely wrong. A function that produces result “A” with an accompanying test that asserts that the function produces result “A” is pointless if the function was supposed to produce result “B”.

Software requirements are notoriously vague and tend to be in a constant state of flux. Humans will always be required to ensure that the original requirement was not only satisfied, but correctly interpreted by the implementing engineer(s).

Sometimes it’s just not testable

Some things are just difficult to test no matter how much forethought is given to the task. Scale/Load testing and concurrency testing are two classic examples of this. Often problems only arise when a system is put under significant load, and/or put under significant concurrent load. These can both be extremely difficult to test for in an automated manner and often require dedicated efforts which may fall outside the regular testing regime. While automated testing of these scenarios can be done there is always a balance between the amount of time spent creating the test and the reduction in risk that having the test provides. As per earlier points however, building things that are by their very nature testable may impact this.

Scalable Manual Testing

As previously highlighted manual testing (traditional QA) doesn’t scale well as system complexity increases, however the role of humans in the testing process nonetheless remains important.

In situations where automated tests are difficult to create or where such testing is unlikely to uncover rare and or unexpected code paths that lead to bugs, one of the most powerful approaches to testing is to crowd source testing from your current user base. Large companies like Google and Facebook use this technique regularly and it involves segmenting your user base either automatically or manually.

Manual user segmentation

This is most often via the creation of an Early Access or BETA program to which end users voluntarily join. Members of these programs are given early access to versions of the software that has not yet completely passed muster and may still contain bugs, however there is a clear understanding of this from the users which mitigates the risk of a poor impression being formed about the software in its target market.

Automated user segmentation

Often used in conjunction with manual segmentation, automated segmentation exists where newer versions of software is released to a portion of the user base without them being enrolled into a specific early access program. A common approach is to stage this in several iterations:

  • Release the new version exclusively to early access members
  • Release the new version to a portion of the wider user base with their consent
  • Release the new version to a portion of the wider user base without their consent
  • Continue to expand the exposure until it encompasses the entire user base

If done in a controlled manner this can be a very safe and very effective way to elicit both reports of bugs and crucially feedback as to usability and/or product-market fit for the new version.