PIT applies a configurable set of mutation operators (or mutators) to the byte code generated by compiling your code.
For example the CONDITIONALS_BOUNDARY_MUTATOR would modify the byte code generated by the statement
if ( i >= 0 ) {
return "foo";
} else {
return "bar";
}
To be equivalent to
if ( i > 0 ) {
return "foo";
} else {
return "bar";
}
PIT defines a number of these operations that will mutate the bytecode in various ways including removing methods calls, inverting logic statements, altering return values and more.
In order to do this PIT requires that the following debug information is present in the bytecode
Most build systems enable this information by default.
By applying the mutation operators PIT will generate a number (potentially a very large number) of mutants. These are Java classes which contain a mutation (or fault) which should make them behave differently from the unmutated class.
PIT will then run your tests using this mutant instead of the unmutated class. An effective set of tests should fail in the presence of the mutant.
Things are not quite this simple in practice as not all mutations will behave differently than the unmutated class. These mutants are referred to as equivalent mutations.
There are various reasons why a mutation might be equivalent including
For example, the following two statements are logically equivalent.
int i = 2;
if ( i >= 1 ) {
return "foo";
}
//...
int i = 2;
if ( i > 1 ) {
return "foo";
}
A common example are mutations to code related to logging or debug. Most teams are not interested
in testing these. PIT avoids generating this type of equivalent mutation by not generating mutations
for lines that contain a call to common logging frameworks (this list of frameworks is configurable,
to enable mutation of logging statements disable the feature FLOGCALL).
PIT runs your unit tests against the mutated code automatically. Before running the tests PIT performs a traditional line coverage analysis for the tests, then uses this data along with the timings of the tests to pick a set of test cases targeted at the mutated code.
This approach makes PIT much faster than previous mutation testing systems such as Jester and Jumble, and enables PIT to test entire code bases, rather than single classes at a time.
For each mutation PIT will report one of the following outcomes
Killed means a test caught the mutation successfully.
Lived means the mutation was not detected by the covering test.
No coverage is the same as Lived except there were no tests that exercised the line of code where the mutation was created.
A mutation may time out if it causes an infinite loop, such as removing the increment from a counter in a for loop.
A non viable mutation is one that could not be loaded by the JVM as the bytecode was in some way invalid. PIT tries to minimise the number of non-viable mutations that it creates.
A memory error might occur as a result of a mutation that increases the amount of memory used by the system, or may be the result of the additional memory overhead required to repeatedly run your tests in the presence of mutations. If you see a large number of memory errors consider configuring more heap and permgen space for the tests.
A run error means something went wrong when trying to test the mutation. Certain types of non viable mutation can currently result in an run error. If you see a large number of run errors this is probably be an indication that something went wrong.
Under normal circumstances you should see no non viable mutations or run errors.