Chapter 4 - How to Write Tests¶

When approaching software testing many developers start by creating tests which assert the boundary conditions of their code. That is, send some data to a section of their code and expect some data back. This is what might classically be called a “black box” test, or often referred to as an “integration test”. The key assumption is that the test does not know (or care) about the inner workings of the code but simply cares whether the actual output produced matches the expected output given known inputs.

This type of testing is often thought of as “unit testing” because the developer is testing a single “unit” (e.g. a method) of code, but in reality they are often testing several units of code because of the dependent nature of the application.

Although this approach may seem sensible, it turns out it can be extremely problematic for a few good reasons:

Related to Rule #1 is the concept that the only reason for a test failure should be a failure in the code being tested. If a test relies on non-deterministic components in order to succeed then failures in these components will cause a failure in the test which does not accurately reflect the validity of the code under test.

Common examples of this are:

• Depending on external systems which may fail for reasons that are unrelated to the code under test

  • Mitigation: Mock dependencies

  
  

• Tests involving asynchronous processes that take variable amounts of time to complete may lead to timeout failures

  • Mitigation: Where possible test the components of the asynchronous process independently

  
  

• Test code that retains state which makes the order in which tests are run meaningful and that may cause failures due to unexpected state when subsequent tests begin.

  • Mitigation: Either avoid storing state external to tests, or ensure external state is cleared after each test.

When creating unit tests it is often the case that the test will simply call a method with prescribed inputs and expect certain outputs. This is really a black box test because the test code does not know anything about the internals of the method, and this is actually fine but the problem can arise when dependencies, or other methods, are required by the method under test but not visible to the test code.

Consider the following example:

public class Foo {

    public int methodA(int value) {
        return methodB(value) + 10;
    }

    public int methodB(int value) {
        return value + 5;
    }
}

In this case a test written to test “methodA” will have the effect of calling “methodB”. While this is not wrong per se, it is not ideal for two reasons:

1. The code in methodB is (presumably) already tested in its own test so executing it a second time in the test for methodA is unnecessary
2. If tests were written for methodA and methodB and a failure occurred in methodB, both tests would fail. In this trivial example this may not be a problem but in more complex applications creates a situation in which the cause of a failure may be unclear, that in turn leads to developers spending more time tracking down the source of the failure.

The preferred way to test this would be to mock methodB in the above code and simply assert that methodA called the mock. More details on creating mock objects are covered below.

Conversely consider the following variation where methodB is marked as private:

public class Foo2 {

    public int methodA(int value) {
        return methodB(value) + 10;
    }

    private int methodB(int value) {
        return value + 5;
    }
}

In this example “methodB” is not callable by the test and hence will not typically have its own test so the test for “methodA” that calls “methodB” could be considered ok. Although there is still the problem that the test is testing two logically independent units of work the limitations/features of the language mean it may not be feasible to create independent tests [1].

One of the common mistakes made when engineers start down a path of testing is to invoke conditions within their code specifically intended for testing, commonly referred to a “Test Mode”. Running an application in Test Mode means that the expectation of the executor is that the application will not perform “real” operations, but rather will behave in a predictable, consistent manner for the purposes of testing.

This in itself is not a problem and having applications that run in Test Mode is not necessarily a bad thing, but the most common way this is implemented is through conditional statements within the production code.

Consider the following:

public class Foo {
	public float methodA(float value) {
		float result;

		if(Globals.TEST_MODE) {
			result = value;
		}
		else {
			result = methodB(value) + 10.0f;
		}

		return result;
	}
	public float methodB(float value) {
		return 10.0f / value;
	}
}

In this example a global (static) variable called “TEST_MODE” is used to determine the result of “methodA”. If the application is in “TEST_MODE” then “methodB” will not be called. While this satisfies our objective of not combining more than one unit, it presents several other problems:

In an effort to increase the code coverage engineers may be tempted to write integration (black box) tests simply to lift the coverage percent. This implies the implementation of a single test that may actually be testing a fairly trivial case but which executes a large section of code. Depending on the coverage system used this will often report that the code has been covered by tests but this may not actually be the case. It’s true that the code was executed during the test but that doesn’t mean that the behavior of the code was “tested”.

The simplest mitigation to this is to separate unit tests from integration tests and ensure that code coverage is not recorded for the integration tests.

Even in the hardiest testing system bugs may still exist in production code and when a bug is reported the first instinct of the engineer is often the locate the cause of the problem. Once located the next most compelling temptation is to fix the problem, particularly if the fix is trivial. This temptation should be avoided wherever possible in preference to first creating a test that exhibits the problem.

The only thing worse that a customer experiencing a bug is a customer experiencing a bug that was fixed but somehow was reintroduced in a subsequent release. The temptation to simply fix problems as they arise exposes the system to the potential for bugs to reemerge without anyone knowing.

A common argument in favor of integration tests is that without an integration test we don’t truly know if the system is behaving as expected. This can be indicative of a misunderstanding of the purpose of the software test.

If there are concerns about whether the environment in which the software is to be run is correct then there should be dedicated “environment tests” which verify the environment independent of the software tests. Merging these two distinct activities can lead to software tests that fail because of environment conditions which doesn’t necessarily mean the “software” is faulty.

When creating unit tests it may be necessary to maintain state outside the test, or use components that are shared by tests and which themselves maintain state. Although it may be preferable to avoid doing this, in some case it cannot be avoided. In these cases it is important to assert pre conditions before, and post conditions after the test is run.

Test should be both idempotent and isolated. Changing the order in which tests are run or re-running the same tests multiple times should not affect the outcome. If state is maintained outside the test and not restored to its original state it may affect subsequent tests leading to either false positive or false negative results.

If an integration test is deemed to be the simplest or best solution for a given software test (this should be a rare occurrence!) then it’s important to ensure that any external systems are checked before the test is run and reverted after the test is complete. Problems can often arise when a test persists state in an external system which then affects subsequent tests.

For example:

A test that writes files to a file system should assert both that the files written during the test do not exist before the test is run, and that they are removed before the test completes.

A mock object is an object that implements the same signature as the object it is mocking, but does not possess any of the actual implementation. That is, as far as the application is concerned it “looks” like the real thing, but it’s actually a fake.

The behavior of the mock object is orchestrated as part of the test so that it produces predictable outputs.

Consider the following example:

public void doSomething(String data,
                        DatabaseWriter writer,
                        Emailer emailer) throws Exception {

    if(data != null) {

        try {
            writer.writeValueToDatabase(data);
        }
        catch (Exception e) {
            emailer.sendErrorEmail(e);
        }
    }
    else {
        writer.logErrorToDatabase("No data");
    }
}

If we assume all the prerequisites to this method are met (i.e. all the parameters passed to the method are valid), there are 5 possible outcomes:

1. “data” is null in which case we expect an error is written to the database
2. “data” is null but the write operation to the database for the error fails in which case we expect the method to throw an Exception.
3. “data” is NOT null but the database write operation fails in which case expect an email to be sent
4. “data” is NOT null, the database write operation fails AND the email fails in which case we expect the method to throw an Exception.
5. “data” is NOT null and the database write operation succeeds in which case we expect “data” is written to the database

Ideally we should provide tests for all of these possible scenarios [2] but for this to work we’d need to somehow orchestrate the states in the external systems (database and email server) to simulate these scenarios. That is, for each of the above variations we’d need create a state in which:

1. Database is available and working as expected
2. Database is not available
3. Database is not available, but email server is
4. Database is not available and email server is not available
5. Database is available and working as expected

Creating a test environment to simulate these conditions presents several challenges:

• It is difficult to automate the enabling/disabling of external resources like databases/email servers in order to simulate a failure
• Tests are naturally fragile as external systems may go down for reasons unrelated to the test
• External systems that maintain state (e.g. databases) may need to be setup in a predictable state before each test. Depending on how the database is accessed this often means the creation of custom queries and/or API endpoints specifically for use in tests which are then subject to errors/failure as database schemas change etc.

A far simpler approach is to mock these external systems and configure the mocks with the predictable state you need.

While there are many ways to do this, using a dedicated mocking framework is often the easiest as they will provide several key benefits over a roll-your-own solution:

1. Mocks of real objects are easily created without needing to implement all required methods [3]
2. Test conditions are easily orchestrated by setting expectations on mocks
3. Mocks allow assertions of methods called and method parameters as well as methods NOT called.

The following is an example of a test for the above code using Mockito, a Java based mocking framework:

DatabaseWriter writer = Mockito.mock(DatabaseWriter.class);
Emailer emailer = Mockito.mock(Emailer.class);
DatabaseError error = Mockito.mock(DatabaseError.class);
String data = "foobar";

// Orchestrate
Mockito.doThrow(error).when(writer).writeValueToDatabase(data);

// Execute
MyClass instance = new MyClass();
instance.doSomething(data, writer, emailer);

// Assert
Mockito.verify(writer, Mockito.times(1)).writeValueToDatabase(data);
Mockito.verify(emailer, Mockito.times(1)).sendErrorEmail(error);

In this example the “MyClass” instance (the class with the “doSomething” method) is provided with mock implementations of the DatabaseWriter and the Emailer that have been orchestrated to create the conditions needed for the test.

After the code is executed we verify that the methods we expected to be called were in-fact called with the parameters we expect.

Importantly this test does not actually require a database or an email server at all and can be run in complete isolation to these external systems.

As may now be apparent from the previous section on mocks, use of a Dependency Injection design pattern is critical to successful implementation of mocked tests in a compiled language like Java.

If the application under test has been created in such a way that all object dependencies are injected it becomes far simpler to inject mocks into these objects when running tests. This has the crucial benefit of not requiring changes to production code solely to facilitate the creation of tests for that code.

When method calls are idempotent (they can be called many times without changing the outcome) it is possible that multiple calls will be made without the caller knowing. Additionally code that executes within a method but which does not significantly alter the outcome of the method may be executed without the caller’s knowledge. Some may claim this is correct and that the called shouldn’t need to know what goes on inside the method, and to a large extent they’d be right but there is critical information being lost here that may lead to bugs of a particularly insidious nature.

Imagine that we have a method that, among other things, updates a row in the database:

public void updateData(
        String data,
        DatabaseWriter writer) {

    writer.update(data);
    writer.update(data);
    writer.flush();
}

A traditional black box test for this might look something like this:

String data = "foobar";

MyClass instance = new MyClass();
DatabaseWriter writer = new DatabaseWriter();

instance.updateData(data, writer);

DatabaseReader reader = new DatabaseReader();

String dataAfter = reader.readData();

Assert.assertEquals(data, dataAfter);

In this test we want to make sure that after we call the update method the database has been correctly updated with the new data.

This test passes and we are happy the code is working, but there’s an obvious problem. The “update” call on the DatabaseWriter was executed twice but because this call is idempotent the test does not know. Although the outcome of this is somewhat meaningless from a behavioral standpoint, the actual implementation has a bug which could prove disastrous. If this method is called several times (or millions/billions of times) the the overall performance of the system will suffer and this may be an extremely difficult bug to track down.

Conversely consider the mocked variation:

public void duplicateCheck () {
    String data = "foobar";

    MyClass instance = new MyClass();
    DatabaseWriter writer = Mockito.mock(DatabaseWriter.class);

    instance.updateData(data, writer);

    Mockito.verify(writer, Mockito.times(1)).update(data);
}

The first thing to note is that the test has considerably less code which is quite common when using mocks but the more important issue is that not only does this verify that the database “would have” been written to, but crucially that it would only happen once. This is something not typically caught by a more coarse black box test.

Similarly the execution of unexpected code can also be detected. In the previous example the call to “flush()” may not be required by the method but may in fact be a cause of significant performance delay. Again this may not be detected by the black box test as it does not have a meaningful impact on the outcome of the method, but would be detected by the mocked example as a matter of course.

There is of course a natural criticism of this approach in that asserting that (for example) the database “would have been” written to is not the same as asserting that it actually was written to but this is a case of appropriate separation of tests. We assume that the DatabaseWriter class in the above example is tested elsewhere and those tests verify that it does what it is supposed to do. Even in those tests we may not be testing whether the database was actually written to but simply following execution path up to the point that we no longer have control over the outcome. We don’t for example need to test that MySQL knows how to deal with an “UPDATE” SQL statement, we assume this works.

As highlighted at the beginning there are several classifications of tests and even with a comprehensive automated testing policy in place there are certainly conditions in which production systems will fail with little or no warning. This however is still not a valid case for arguing the need for integration tests.

Here’s a simple example of why:

Consider a method of a class that writes a file to the local disk. The integration test approach would have the class write the file then the test would assert that the file was actually written. But in this case we are actually testing several things:

1. Our code correctly calls the appropriate file system operations to write a file
2. The programming language knows how to call the appropriate operating system commands to write to disk.
3. The operating system knows how to correctly access the disk hardware to write data
4. The disk is operating correctly and is able to have data written to it.

Of all of these things we only truly have control over the first one. Or to put it another way, “our code changes can not affect anything beyond our code”. So the question becomes, “why do we need to test this?”, to which the answer is “we don’t”.

Now the important implication here is that we are never actually testing that files are written to disk and there could indeed be a failure at some point that is outside our code but which affects the behavior of the method such that it fails. The key point is that tests are required to ensure external systems are available and working correctly but these tests should not be the same as the tests which verify that code is working as expected.

Indeed testing all these components in the one place can create problems for testers and engineers:

• Over time tests become fragile. When a test includes dependencies on external systems changes in state or availability in these external systems will often cause tests to fail when in fact there is nothing wrong with the code being tested. Code that is heavily dependent on many external systems will be sensitive to changes in any external system and may result in being in a fail state more often than not. This leads to the problem of “Test Nonchalance”. A situation in which team members are not particularly worried by tests failing because they “are always failing” and it’s rarely due to the code.
• Verifying the state of external systems after a test can be problematic (or impossible) as already discussed.
• Performing integration tests on production systems can be risky, and testing on non-production systems is meaningless. If we want to make sure that the external systems are behaving correctly in production, but we don’t test the production systems because we are concerned about polluting production data then one could say the test is meaningless.

It turns out that the answer to the question, “when is integration testing required?” is surprisingly, “not very often at all”. Code is tested in unit tests that utilize mocked versions of external dependencies. External systems are tested independently and monitored with appropriate hardware and/or device monitoring systems.

There are however some cases where integration tests are useful. Consider the case where a database schema change is made which introduces a new field, or removes an existing field from a table in the database. This change causes a catastrophic failure in the code because the unit tests that cover the code did not completely test that data can be written to the database however the schema changes made imply that the code written will no longer function. The database server is up and running and is accessible over the network. In this case the production system will fail and yet all the tests are passing. So what went wrong?

There are broadly two approaches to answering this:

1. The system needs some integration tests to verify that data is getting all the way to the database
2. The system currently has a gap that is not tested, specifically the relationship between the code and the data model. This gap needs to be plugged.

With all the information presented it is probably no surprise that the preferred option is the second one. In this case we have a disconnect between what the code expects the database to look like and what it actually looks like. The correct fix for this is to create a set of tests that assert that the expectations of the code are correct, and a separate set of tests that assert the database represents these expectations. Creation of an intermediate representation of the database to which both tests refer would one possible way to do this (although there may be many).