Mutation Testing 

Publikované: 1. júl 2016 14:02

We always try to find new ways how to make our code better. Now we learned about a technique for improvement of test quality. It’s called Mutation testing. Below you can read what it is and see some examples of how to use it. Enjoy.

Successful and properly working applications cannot be developed without proper testing. As developers we learned to accept it and started using different test libraries on a daily basis. JUnit and TestNG are few of them that directly come to my mind.  

One of the statistics that came out of the tests we do at work is test coverage. This percentage should reflect how much of your code base is covered by the tests you write. Although it gives some informational value, most of the time you cannot rely on it, since it does not reflect if the code is tested properly and with correct assertions. This is the part where mutation testing comes into play. 

 

What is mutation testing? 

The basic idea behind this technique is the improvement of your existing tests quality. It will not help you write new tests or test the functionality of a system which has zero existing tests.  

Mutation testing involves modifying a program in small ways where each mutated version is called a mutant. Think of a mutant as an additional class with a single modification compared to the original one. Changes which are created are based on pre-defined mutation operators that mimic typical programming errors. Easy example of mutation could be switching one mathematic operator, for example < into another > , or even removing some method call altogether.  

Tests are executed against this mutated version of the source code class, while expecting your tests to catch e.g. „kill“ the „mutant“.  Killing the mutant consists of your tests failing when executed against mutated version of your code. This indicates that you correctly asserted/covered your code for this particular change. If, however, your tests pass even after fundamental logic of your application was changed, it indicates your test cases are not done well enough. 

 

Origins 

Mutation testing was originally proposed by Richard J.Lipton in 1971, however it was not well received due to the performance impact it had. Nowadays it starts gaining popularity again due to the evolution of computers and their capacities. 

 

Goal of this technique 

The purpose is to locate weaknesses of the tests you wrote. The more mutants a test suite kills, the more we can be confident that our code is more „production-ready“ and covered against potential bugs.  

In general, this technique could be used with existing test frameworks like Mockito, EasyMock, PowerMock and others. 

 

Example 

Luckily, you do not have to write these mutated versions of your source code manually. There are tools which will mutate the actual byte code into different versions and execute your unit tests against them automatically. One of these tools is called PIT (Parallel Isolated Test), with which we will demonstrate the following functionality.  

PIT works with different build managers like Maven or Ant. We will look at the Maven integration below: 

<plugin> 
   <groupId>org.pitest</groupId> 
   <artifactId>pitest-maven</artifactId> 
   <version>1.1.10</version> 
   <configuration> 
      <reportsDirectory>C:UsersuserDesktoppit</reportsDirectory> 
      <targetClasses> 
         <param>com.your.package.root.you.want.to.mutate</param> 
      </targetClasses> 
      <targetTests> 
         <param>com.test.you.want.to.execute</param> 
      </targetTests> 
      <avoidCallsTo> 
         <avoidCallsTo>java.util.logging</avoidCallsTo> 
         <avoidCallsTo>org.apache.log4j</avoidCallsTo> 
         <avoidCallsTo>org.slf4j</avoidCallsTo> 
         <avoidCallsTo>org.apache.commons.logging</avoidCallsTo> 
      </avoidCallsTo> 
   </configuration> 
</plugin> 

Example configuration of the plugin

 

There are numerous other configuration options, where most commonly you will probably use the following: 

• mutators – list of mutation operators to apply (list – http://pitest.org/quickstart/mutators/) 
• excludedMethods 
• excludedClasses 
• timeoutConstant – constant amount of additional time to allow a test to run for (after the application of the timeoutFactor) before considering it to be stuck in an infinite loop. 

For further information about plugin configuration you could visit PIT documentation at http://pitest.org/quickstart/maven/. 

After the plugin is configured to fit our needs, we will execute it with command mvn -DwithHistory org.pitest:pitest-maven:mutationCoverage 

The reports produced by PIT are in an easy to read format combining line coverage and mutation coverage information. Within the report you will see which mutants were killed and which passed through your test. 

 

 Example snippet taken from the coverage report of Wicket Core 

Light green shows line coverage, dark green shows mutation coverage. 

Light pink show lack of line coverage, dark pink shows lack of mutation coverage. 

 

Depending on the number of test cases and classes we will now calculate the estimated execution time of the mutation tests: 

Total time execution for all tests ( 100 classes, 10 test cases for each class) = 100 x 10 x 2ms = 2seconds. 

10 mutants for each class would result in 1000 mutants in total. 

Brute force testing will result in 1000 x 2s = 33 minutes 20 seconds. That is due to the fact, that each mutation will have an impact on the system. So all the tests will be executed for each mutation. Therefore it will take 2 seconds for each of the thousand mutations to finish. 

PIT offers smart testing functionality based on code coverage which will result in 1000 x 10 x 2ms = 20 seconds, because we do not have to run all tests upon each mutation. Tests which needs to be executed are known based on code coverage and history. In this case each class is relevant for 10 test cases, so we only need to execute those 10 for that particular mutation, and not all tests in our system. 

To achieve best performance you should write small and fast test methods. Tests that should probably be avoided for mutation testing are the integration tests, which initialize database connection and call different web services. 

If the duration of these tests is higher than you want, there is always an option to integrate the mutation tests within nightly continuous integration builds. 

 

Example of some pre-definned mutators 

Return Values Mutator – The return values mutator mutates the return values of method calls. Depending on the return type of the method, another mutation is used. For example, if return type is Object, then mutator will replace non-null return values with null and throw a java.lang.RuntimeException if the unmutated method returns null. 

Math Mutator – The math mutator replaces binary arithmetic operations for either integer or floating-point arithmetic with another operation. The replacements will be selected according to the table below: 

Original conditional	Mutated conditional
+	–
–	+
*	/
/	*
%	*
&	|
|	&
^	&
<<	>>
>>	<<
>>>	<<

 

Void Method Call Mutator – The void method call mutator removes method calls to void methods.   

public void someVoidMethod(int i) { 
   // does something 
} 
 
public int foo() { 
   int i = 5; 
   someVoidMethod(i); 
   return i; 
} 

will mutate into 

public void someVoidMethod(int i) { 
   // does something 
} 
 
public int foo() { 
   int i = 5; 
   return i; 
}

 

So why should you want to use this technique? 

If the above mentioned examples did not convince you enough, think of how your customer will be happy if no production bugs will occur in his system. Think of how happy you will be with maintaining the code and making changes to it, knowing that all possible cases are covered. If you or your team need desperate help with improving quality of your code and tests, I think this is exactly something you should look into.  

 

Used resources 

• What the Heck Is Mutation Testing?, http://www.codeaffine.com/2015/10/05/what-the-heck-is-mutation-testing/ 
• PIT tool, http://pitest.org/