Humbug for Mutation Testing

> I was running 'Humbug' by Pádraic Brady for the first time I was interested in how good bad it would show my tests to be
First off you need to be able to run your phpunit tests without parameters as

phpunit

rather than

phpunit Tests/

I had been running like that rather than specifying the test suite in my phpunit.xml

./Tests/

Before I added in the testsuite to the phpunit.xml I was getting the heading from humbug and then it was stopping and echoing out the help output from phpunit
Then you need to check it runs ok and everything passes otherwise humbug stops

Ivor:tools Ivor$ phpunit
...............................................................  63 / 248 ( 25%)
..............S................................................ 126 / 248 ( 50%)
.................S................S............................ 189 / 248 ( 76%)
.............................................S.............     248 / 248 (100%)
Time: 34.08 seconds, Memory: 51.27Mb
OK, but incomplete, skipped, or risky tests!
Tests: 248, Assertions: 5606, Skipped: 4.

Now trying humbug it may take some time ...
Humbug on the above set of tests took 1.43 hours and there was a lot of output that made understanding it difficult. The output was quite useful though in that it showed weaknesses in my tests the --incremental flag is designed to speed up the test somewhat although in my circumstances it didn't make a large difference.
To be honest at least at first you probably want to try this out on something smaller with just a few tests at first the output is easier to understand when your doing it on a smaller set of tests. You get an output file with diffs so you can have a look through and see how Humbug has twisted your code about. The one produced for the unit tests above was about 6000 lines.
Then your left viewing the issues do they matter or are they false warnings.

1) \Humbug\Mutator\ReturnValue\This
Diff on \adminLessonAccess::__construct() in /classes/adminLessonAccess.php:
--- Original
+++ New
@@ @@
         $this->db =3D dbConnection::get();
-        return $this;
+        return null;
     }
     function listLessonsForAdminAccess(){

So my tests did not cover the constructor returning null rather than $this.

2) \Humbug\Mutator\Boolean\TrueValue
Diff on \adminLessonAccess::deleteAllExpiredIds() in
/classes/adminLessonAccess.php:
--- Original
+++ New
@@ @@
         }
-        return True;
+        return false;
     }

Looking at the tests, my test covered that the things were deleted but not that the result should be true. Which isn't the end of the world but its a simple line to add an assertion that that should turn True;
I think its a really useful tool in blowing a hole through how solid you thought your unit tests were according to code coverage stats and also in helping you improve existing unit tests and create better new ones.

Version

I was using Humbug version 1.0.0-alpha1-18-gd102496 as a phar my humbug.dist.json looks like this

{
    "source": {
        "directories": [
            "../classes"
        ],
        "excludes": [
            "../classes/PHPExcel"
        ]
    },
    "timeout": 10,
    "logs": {
        "text": "humbuglog.txt"
    }
}

Why do mutation testing at all

Pádraic probably answers that a lot better than I can in his article (currently down) if that link doesn't work try this.
Put simply code coverage stats can be pretty overrated. They are good at showing how much of your code has tests but don't answer the other questions.

• How good are those tests,
• do they really test the code properly
• are they just thin coverage filler written to achieve a coverage statistic.
• are the tests strict or fluffy
• are the tests redundant

These are some of the things you can get a better idea of with mutation testing.