Basic Concepts

Mutation Operators

PIT applies a configurable set of mutation operators (or mutators) to the byte code generated by compiling your code.

For example the CONDITIONALS_BOUNDARY_MUTATOR would modify the byte code generated by the statement

if ( i >= 0 ) {
    return "foo";
} else {
    return "bar";
}

To be equivalent to

if ( i > 0 ) {
    return "foo";
} else {
    return "bar";
}

PIT defines a number of these operations that will mutate the bytecode in various ways including removing methods calls, inverting logic statements, altering return values and more.

In order to do this PIT requires that the following debug information is present in the bytecode

  • Line numbers
  • Source file name

Most build systems enable this information by default.

Mutants

By applying the mutation operators PIT will generate a number (potentially a very large number) of mutants. These are Java classes which contain a mutation (or fault) which should make them behave differently from the unmutated class.

PIT will then run your tests using this mutant instead of the umutated class. An effective set of tests should fail in the presence of the mutant.

Equivalent Mutations

Things are not quite this simple in practice as not all mutations will behave differently than the unmutated class. These mutants are referred to as equivalent mutations.

There are various reasons why a mutation might be equivalent including

  • The resulting mutant behaves in exactly the same way as the original

For example, the following two statements are logically equivalent.

int i = 2;
if ( i >= 1 ) {
    return "foo";
}

//...
int i = 2;
if ( i > 1 ) {
    return "foo";
}
  • The resulting mutant behaves differently but in a way that is outside the scope of testing.

A common example are mutations to code related to logging or debug. Most teams are not interested in testing these. PIT avoids generating this type of equivalent mutation by not generating mutations for lines that contain a call to common logging frameworks (this list of frameworks is configurable, so mutation of logging statements can be enabled by configuring a list containing only a non-existent class).

Running the tests

PIT runs your unit tests against the mutated code automatically. Before running the tests PIT performs a traditional line coverage analysis for the tests, then uses this data along with the timings of the tests to pick a set of test cases targeted at the mutated code.

This approach makes PIT much faster than previous mutation testing systems such as Jester and Jumble, and enables PIT to test entire code bases, rather than single classes at a time.

For each mutation PIT will report one of the following outcomes

  • Killed
  • Lived
  • No coverage
  • Non viable
  • Timed Out
  • Memory error
  • Run error

Killed and Lived are self explanatory.

No coverage is the same as Lived except there were no tests that exercised the line of code where the mutation was created.

A mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop.

A non viable mutation is one that could not be loaded by the JVM as the bytecode was in some way invalid. PIT tries to minimise the number of non-viable mutations that it creates.

A memory error might occur as a result of a mutation that increases the amount of memory used by the system, or may be the result of the additional memory overhead required to repeatedly run your tests in the presence of mutations. If you see a large number of memory errors consider configuring more heap and permgen space for the tests.

A run error means something went wrong when trying to test the mutation. Certain types of non viable mutation can currently result in an run error. If you see a large number of run errors this is probably be an indication that something went wrong.

Under normal circumstances you should see no non viable mutations or run errors.