What is Mutation Testing?

1. Mutations are automatically seeded into your code
2. Tests are run
3. If your tests
   1. fail -> the mutation is killed
   2. pass -> the mutation lived

The quality of your test suite can be measured by the percentage of mutations lived.

In other words, your application code is automatically modified. When the production code changes then it should produce different results and therefore cause your unit tests to fail. If this does not happen, then it may indicate a problem with the quality of the test suite.

Why Mutation Test Your Code?

Test coverage only measures the percentage of code that is executed by your tests. It tells you nothing about the quality of your assertions in your unit tests. For example, you could have 100% code coverage with 0 assert statements (meaning, your tests always pass, no matter what). Therefore, code coverage only tells you how much of your code is not tested. In contrast, mutation testing makes sure that your unit tests are actually able to detect faults in your production code.

How does it work on the JVM?

Pitest mutates your project’s bytecode in memory (note that it does not recompile, meaning it’s very fast) and executes your whole test suite against every mutation. Pitest integrates well with modern build tools like Gradle and Maven. In the following example, we’ll be using Gradle.

How to enable Pitest in Gradle?

We’re using Gradle Kotlin DSL, so our build.gradle.kts looks like this:

import info.solidsoft.gradle.pitest.PitestTask

buildscript {
    repositories {
        mavenCentral()
    }
    configurations.maybeCreate("pitest")
    dependencies {
        classpath("info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.4.7")
        "pitest"("org.pitest:pitest-junit5-plugin:0.12")
    }
}

plugins {
    val kotlinVersion = "1.3.70"
    kotlin("jvm") version kotlinVersion
    id("info.solidsoft.pitest") version "1.4.7"
}

group = "com.producement"
version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
}

dependencies {
    implementation(kotlin("stdlib-jdk8"))

    testImplementation("org.junit.jupiter:junit-jupiter-api:5.6.0")
    testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:5.6.0")
    testImplementation("org.assertj:assertj-core:3.15.0")
}

tasks {
    withType<Test> {
        useJUnitPlatform {
            includeEngines("junit-jupiter")
            excludeEngines("junit-vintage")
        }
    }

    named("build") {
        dependsOn("pitest")
    }

    withType<PitestTask> {
        testPlugin.set("junit5")
        threads.set(1)
        outputFormats.set(setOf("XML", "HTML"))
        mutators.set(setOf("STRONGER", "DEFAULTS"))
        avoidCallsTo.set(setOf("kotlin.jvm.internal", "kotlinx.coroutines"))
    }
}

We’re using Kotlin, JUnit5 and AssertJ in this example.
Now, let’s write an example unit test:

class ExampleTest {

    private val example = Example()

    @Test
    fun exampleTest() {
        example.add(id = "123", name = "Erko")

        val name = example.getName(id = "123")

        assertThat(name).isNotNull()
    }
}

And the production code to make this pass:

class Example {

    private val idToName: MutableMap<String, String> = HashMap()

    fun add(id: String, name: String) {
        idToName[id] = name
    }

    fun getName(id: String): String? {
        return idToName[id]
    }
}

We have correctly followed the TDD Red-Green-Refactor cycle, we have a passing unit test and 100% code coverage. However, there’s a slight problem with our unit test. Can you spot the problem?

Now let’s run Pitest to see if it’s able to detect the issue: ./gradlew pitest:

>> Generated 1 mutations Killed 0 (0%)
>> Ran 1 tests (1 tests per mutation) 

The test report reveals the issue:

When the return value of the getName() method was replaced with an empty string "", the test still passed. This tells us that our assertion in the test assertThat(name).isNotNull() is not specific enough. A proper assert statement would be  assertThat(name).isEqualTo("Erko"). And this does indeed pass our mutation test:

>> Generated 1 mutations Killed 1 (100%)
>> Ran 1 tests (1 tests per mutation)

Summary

Mutation testing really helps us with identifying weak tests (those that never kill mutants). You can read more about Mutation Testing from the wiki: https://en.wikipedia.org/wiki/Mutation_testing