Preface

Software testing is a topic which has undergone much debate over the course of software development’s recent history. In the most broad terms people fall somewhere along a spectrum of those who believe it is, for the most part, a “nice to have”, and those who cannot live without the confidence and satisfaction a robust testing regime can offer despite its perceived overhead.

The following guide will attempt to outline some of the key reasons testing should be considered a critical component of any mature software development process as well as some basic guidelines for the creation of a testing approach which maximizes release confidence while minimizing the impact on development times.

Language Use

The majority of code examples in this guide use Java as the default programming language however the concepts presented are generally language non-specific. Java has been chosen because although the syntax is more verbose than some other languages, its familiarity means the examples can be understood by a wide audience.

Definitions

Note

Not all types of tests are covered in this list, merely the ones referred to in this guide.

For the purposes of avoiding ambiguity or confusion we are going to start with defining some terms. Some definitions may not correspond exactly to what you’re used to, but these are how we use the terms in this guide.

Unit Test

Although the term “unit test” is in the common vernacular for software engineers it is actually a very imprecise and subjective term. Wikipedia suggests that a unit is “often an entire interface, such as a class, but could be an individual method” [1]. This ambiguity can lead to misunderstanding when talking about testing so for the purposes of this discussion we will define a unit test as:

“A unit test is a one which tests a single public method or function of a class or module”

The two key words here are “single” and “public”. A public method which internally calls private or protected methods in the same class could be considered a single unit, whereas public methods that call other public methods would not.

Black Box Test

This is a test in which the mechanisms involved in processing data are unknown to the test and the test simply provides pre-conditions and asserts post-conditions. That is, the test provides data and expects certain results without knowing how those results are derived.

Integration Test

For the purposes of this discussion we will define an Integration Test as one in which many parts of the system are tested at once. Importantly an integration test will test a complete end-to-end lifecycle of an application or a portion thereof as opposed to testing selected components independently. Note that an integration test may be done with mocked components and the mere term Integration Test does not necessarily mean real systems are used, although often this is the case. During this discussion when the term Integration Test is used it will typically refer to tests that are performed with real systems as opposed to mocks.

Manual Test

A manual test is simply one that is conducted by a human via using the software directly without any intermediate testing tools. This is often done by Quality Assurance staff who would typically follow a prescribed script of actions and expected results.

Load/Scale Test

Load tests are tests done specifically to impose stress on the application by simulating heavy load conditions in addition to highly concurrent load conditions (several simultaneous actions). This can be done to identify failures but is often done to simply gauge scalability. That is, “how is performance affected with increases in load?” as opposed to simply “does the application function under load?”.

Infrastructure Test

Infrastructure tests are those which test or verify the state of the infrastructure underlying the software. This typically includes servers and networks associated with providing application uptime. Automated tests that verify database operations are working, for example, may not signal a failure if/when a database server ceases to be visible to the application due to a failure in a network device. Infrastructure Tests would also encompass monitoring and alerting systems designed to provide “early warning” of imminent or recent failures.

QA Test

QA (Quality Assurance) tests, for the purposes of this guide, are equivalent to Manual Tests. The term Manual Test is used to avoid any confusion between QA Testing and UAT (User Acceptance Testing)

UAT (User Acceptance Testing)

User Acceptance Testing is the process by which (human) testers will assess the implementation of the software to determine if it accurately reflects the expectations and/or specifications originally provided to the engineers.

Although this is an activity which is distinct from Quality Assurance, which is typically intended to assure the “quality” of the software (i.e. look for bugs), UA testing is often performed by the same staff who are responsible for QA. This distinction is nevertheless an important one and is discussed in further detail later in the guide.




[1]http://en.wikipedia.org/wiki/Unit_testing