Basic Concepts

Mutation Operators

PIT applies a configurable set of mutation operators (or mutators) to the byte code generated by compiling your code.

For example the CONDITIONALS_BOUNDARY_MUTATOR would modify the byte code generated by the statement

if ( i >= 0 ) {
    return "foo";
} else {
    return "bar";
}

To be equivalent to

if ( i > 0 ) {
    return "foo";
} else {
    return "bar";
}

PIT defines a number of these operations that will mutate the bytecode in various ways including removing methods calls, inverting logic statements, altering return values and more.

In order to do this PIT requires that the following debug information is present in the bytecode

  • Line numbers
  • Source file name

Most build systems enable this information by default.

Mutants

By applying the mutation operators PIT will generate a number (potentially a very large number) of mutants. These are Java classes which contain a mutation (or fault) which should make them behave differently from the unmutated class.

PIT will then run your tests using this mutant instead of the umutated class. An effective set of tests should fail in the presence of the mutant.

Equivalent Mutations

Things are not quite this simple in practice as not all mutations will behave differently than the unmutated class. These mutants are referred to as equivalent mutations.

There are various reasons why a mutation might be equivalent including

  • The resulting mutant behaves in exactly the same way as the original

For example, the following two statements are logically equivalent.

int i = 2;
if ( i >= 1 ) {
    return "foo";
}

//...
int i = 2;
if ( i > 1 ) {
    return "foo";
}
  • The resulting mutant behaves differently but in a way that is outside the scope of testing.

A common example are mutations to code related to logging or debug. Most teams are not interested in testing these. PIT avoids generating this type of equivalent mutation by not generating mutations for lines that contain a call to common logging frameworks (this list of frameworks is configurable, so mutation of logging statements can be enabled by configuring a list containing only a non-existent class).

Running the tests

PIT runs your unit tests against the mutated code automatically. Before running the tests PIT performs a traditional line coverage analysis for the tests, then uses this data along with the timings of the tests to pick a set of test cases targeted at the mutated code.

This approach makes PIT much faster than previous mutation testing systems such as Jester and Jumble, and enables PIT to test entire code bases, rather than single classes at a time.

For each mutation PIT will report one of the following outcomes

  • Killed
  • Lived
  • No coverage
  • Non viable
  • Timed Out
  • Memory error
  • Run error

Killed and Lived are self explanatory.

No coverage is the same as Lived except there were no tests that exercised the line of code where the mutation was created.

A mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop.

A non viable mutation is one that could not be loaded by the JVM as the bytecode was in some way invalid. PIT tries to minimise the number of non-viable mutations that it creates.

A memory error might occur as a result of a mutation that increases the amount of memory used by the system, or may be the result of the additional memory overhead required to repeatedly run your tests in the presence of mutations. If you see a large number of memory errors consider configuring more heap and permgen space for the tests.

A run error means something went wrong when trying to test the mutation. Certain types of non viable mutation can currently result in an run error. If you see a large number of run errors this is probably be an indication that something went wrong.

Under normal circumstances you should see no non viable mutations or run errors.

Incremental analysis

PIT contains an experimental feature to enable it use on very large codebases - incremental analysis.

If this option is activated, PIT will track changes to code and tests and the results from previous runs. It will then use this information to avoid re-running analysis when the results can be logically inferred.

A number of optimisations are possible.

  1. If an infinite loop was detected in the last run, and the class has not changed then it can be assumed that this mutation still results in an infinite loop.
  2. If a mutation was killed in the last run and neither the class under test or the killing test has changed, then it can be assumed that this mutation is still killed
  3. If a mutation survived in the last run and no new tests cover it, and none of the covering tests have changed, then it must still survive
  4. If a mutation was previously killed, but the class or killing test has changed then it is likely that the last killing test will still kill it and it should therefore be prioritised above others
  5. If a number of mutations for a class previously survived, but the class has changed then it is likely that these mutations will still survive. If they are enabled simultaneously and can not be killed as a single meta mutant then the mutations need not be analysed individually.

With the exception of 4), all these optimisations introduce a degree of potential error into the analysis.The main issue is that a class’ behaviour is not defined only by its bytecode, but also by its dependencies (i.e the classes it interacts with and the graph of classes that they interact with). PIT will only consider the strongest of these dependencies - changes to super classes and outer classes, when deciding if a class has changed.

So the incremental feature is based on the assumption that it will be relatively rare for changes in the dependencies of a class to change the status of a mutation.

Although this assumption seems reasonable, it is currently unproven.

Optimisation 5) carries the additional risk that the mutations within the meta mutant might cancel each other out, leaving the behaviour of the class unchanged. Again, it seems likely that this would be rare, but this has not been quanitified.

Incremental analysis is currently controlled by two parameters

historyInputLocation historyOutputLocation

These point to the locations from which to read and write mutation analysis results. This can be the same location, if different locations are used you will need to implement some mechanism to swap the values between runs as PIT does itself does not currently provide a mechanism.