Generating code coverage reports using Istanbul

Introduction

This is the second article in a series of three in which I will explain how I have setup unit testing for a JavaScript module. It documents how I have added code coverage reports using Istanbul. In the next article I will explain how to integrate with a continuous integration (CI) server.

Series overview

This series consists of the following articles:

Part 1: JavaScript unit testing with Mocha and Grunt
Part 2: Generating code coverage reports using Istanbul
Part 3: Running unit tests on a continuous integration service like Travis CI or Wercker

The demo project on GitHub

The files of the demo project are available on GitHub. For every step in getting the project setup there is separate branch in the repository. Whenever there is a branch with the current state of the code in the article I will make a mention of it in the text.

Setting up Istanbul

For the code coverage reports I am using Istanbul. It is the code coverage tool of choice for The Intern and Karma so it seemed like a safe pick. To add Istanbul to the project the following command needs to be executed:

npm install --save-dev istanbul

Adding Grunt tasks

With the package installed it is time to create the Grunt task to run Istanbul. In the Gruntfile.js the following needs to be added:


// Require Istanbul, this way we can use it in our Grunt task
var istanbul = require('istanbul');

// This is the reason we needed a modified version of grunt-mocha. The
// modification calls this method and passes on the information we need for
// Istanbul to do its work
grunt.event.on('coverage', function (coverage) {
    // Write the data we received to the coverage property of the coverage task
    grunt.config('coverage.coverage', coverage);
});

// Here we define the coverage task, it will have two targets: instrument and report
grunt.registerMultiTask('coverage', 'Generates coverage reports for JS using Istanbul', function () {
    switch(this.target) {
    case 'instrument':
        // In the target configuration it is possible to exclude certain files like
        // third party libraries
        var ignore = this.data.ignore || [];
        // Create a new instrumenter
        var instrumenter = new istanbul.Instrumenter();
        // In the target configuration you need to specify the files to cover, here
        // we will loop over all the files
        this.files.forEach(function (file) {
            // 1: Get the filename for the current file
            // 2: Read the file from disk, even if it might be a file we instructed
            //    Istanbul to ignore. It will still get written to the output folder
            var filename = file.src[0],                     /* [1] */
                instrumented = grunt.file.read(filename);   /* [2] */
            // Only instrument this file if it is not in ignored list
            if (!grunt.file.isMatch(ignore, filename)) {
                // Instruct the instrumenter to work its magic on the file
                instrumented = instrumenter.instrumentSync(instrumented, filename);
            }
            // Write the file to its destination
            grunt.file.write(file.dest, instrumented);
        });
        break;
    case 'report':
        // We need config property coverage.coverage to be present, if it is not
        // present this will fail the task
        this.requiresConfig('coverage.coverage');

        // 1: In the target configuration you can set the reporters to use when
        //    generating the report.
        // 2: In the target configuration you can set the folder in which the
        //    report(s) will be stored.
        var Report = istanbul.Report,
            Collector = istanbul.Collector,
            reporters = this.data.reports,    /* [1] */
            dest = this.data.dest,            /* [2] */
            collector = new Collector();

        // Fetch the coverage object we saved earlier
        collector.add(grunt.config('coverage.coverage'));

        // Iterate over all reporters
        reporters.forEach(function (reporter) {
            // Create a report at the specified location for the current reports
            Report.create(reporter, {
                dir: dest + '/' + reporter
            }).writeReport(collector, true);
        });
        break;
    default:
        // The target is neither instrument nor report, display a friendly warning message
        grunt.warn('The target "' + this.target + '" is an invalid target. Valid targets are "instrument" and "report"');
        break;
    }
});

Registering the Grunt tasks for Istanbul

Registering the tasks is only half the work. Next we need to add the configuration for these tasks, this we will do in grunt/coverage.js (again, the file needs to have the same name as the task we registered) and it looks like this:


module.exports = {
    // This is the property we use for the report task to get its data from. The
    // coverage data which will be stored in this property comes from the modified
    // grunt-mocha task
    coverage: null,

    // Configure the instrument task. It will tell Istanbul which files to prepare
    // 1: The files which Istanbul should not cover, e.g.: third party libraries.
    //    These files will be copied as-is to the destination folder.
    // 2: The file mask for the files to cover.
    // 3: The folder where to look for the files.
    // 4: The folder where the instrumented files and ignored files should be
    //    copied to. Don't make this the same as the folder at [3] or your
    //    original files will be overwritten
    instrument: {
        ignore: [],                   /* [1] */
        files: [
            {
                src: '**/*.js',       /* [2] */
                expand: true,
                cwd: 'lib',           /* [3] */
                dest: 'test/src'      /* [4] */
            }
        ]
    },

    // Configure the report task
    // 5: Specify the formats to use for the report. 'html' will result in a HTML
    //    file which links to the reports for the instrumented files. This will
    //    allow you to see which branches weren't executed etc. The 'text-summary'
    //    reporter will write a small stats table to the terminal giving some
    //    immediate visual feedback after running the Grunt task.
    // 6: This is the folder where the reports will be written to. Each report
    //    format, in our case just HTML, will be written to a subfolder named
    //    after the format.
    report: {
        reports: ['html', 'text-summary'],    /* [5] */
        dest: 'coverage'                      /* [6] */
    }
};

Configuration of the coverage task

Adding a new test task target

There is one last thing to do before we can start generating code coverage reports. Generarting the code coverage reports has to become part of the unit test flow. To do this it is necessary to change the test task that was registered in part 1. The expanded task will look like this, it replaces the original registration:


// Here we define the test task, it has two targets: testonly and testcover
grunt.registerMultiTask('test', 'Run JS Unit tests', function () {
    // Get the options for the current target
    var options = this.options();
    // In the options for the task you can configure which spec files should be
    // run. We use this to create a list of file names which we can insert into
    // the {{ tests }} placeholder in our HTML template
    var tests = grunt.file.expand(options.files).map(function(file) {
        return '../' + file;
    });

    // build the template by replacing the placeholders for their actual values
    var template = grunt.file.read(options.template).replace('{{ tests }}', JSON.stringify(tests)).replace('{{ baseUrl }}', JSON.stringify(options.baseUrl));

    // write template to tests directory
    grunt.file.write(options.runner, template);

    // Check the current target for the task
    switch(this.target) {
    case 'testcover':
        // Tell grunt to run the following three tasks:
        // 1. coverage:intrument to created the instrumented files to use in the
        //    unit tests.
        // 2. mocha:testcover to perform the unit test. Because the instrumented
        //    versions of our files are in a different folder we need a separate
        //    target to instruct Mocha where to look for them.
        // 3. coverage:report to generate the actual code coverage report(s).
        grunt.task.run('coverage:instrument', 'mocha:test', 'coverage:report');
        break;
    case 'testonly':
        // Run Mocha the way we did in part one
        grunt.task.run('mocha:test');
        break;
    default:
        // The target is neither testcover nor testonly, display a friendly warning message
        grunt.warn('The target "' + this.target + '" is an invalid target. Valid targets are "testcover" and "testonly"');
        break;
    }
});

The expanded Grunt test task

Now that the task has a new target it is needed to add a configuration for it. The grunt/test.js needs to be updated to this:


module.exports = {
    // Configure the test task with the following options:
    // 1: The name of the HTML template file, this is the file with the
    //    placeholders.
    // 2: This is the filename that is used to write the modified template to, it
    //    is the file that we will run with Mocha.
    // 3: The pattern for the spec files to include in the test runner, you can
    //    use a glob pattern for this.
    options: {
        template: 'test/index.template.html', /* [1] */
        runner: 'test/index.html',            /* [2] */
        files: 'test/spec/**/*.js'            /* [3] */
    },

    // Configure the testonly target
    // 4: This is the path where the JavaScript files to test are located, it is
    //    relative to the /test folder.
    testonly: {
        options: {
            baseUrl: '../lib'                   /* [4] */
        }
    },

    // Configure the testcover target
    // 5: This is the path where the instrumented versions of the JavaScript files
    //    are placed. It is relative to the /test folder
    testcover: {
        options: {
            baseUrl: 'src'                      /* [5] */
        }
    }
};

The expanded test task configuration

When running the test:testcover task we need to change the path of the baseUrl which is placed in the require config in the index.html file. This path needs to be same as the path used in the coverage:instrument task. Be aware that in grunt/coverage.js the path is relative to the project root and in grunt/test.js the path is relative to the /test folder.

Generating the code coverage reports

Now that everything is registered and configured it is possible to generate a code coverage report. All that is required to generate the reports is to run this command from the terminal in the root folder:

grunt test:testcover

This will generate the following summary in the terminal:


=============================== Coverage summary ===============================
Statements   : 100% ( 5/5 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 3/3 )
Lines        : 100% ( 5/5 )
================================================================================

The summary report generated by Istanbul

Yeey! Every single line of the Example.js module was covered with the unit test. A job well done. For a more indept report we can turn to the /cover/html folder where Istanbul generated an index.html file.

The HTML report Istanbul generated for the demo project can also be found here. From this page you can navigate to the individual JavaScript modules and see which lines of code were touched and which branches didn't get hit.

Wrapping up

This concludes the second part of the series. The project will now generate code coverage reports whenever the unit tests are run. This will give a clear indication of how thoroughly we've tested the JavaScript modules and if everything still works according to plan. In the next part I will explain how to tie all this together with a continuous integration server. This will make sure the unit tests are run whenever changes to the repository get pushed to the Git server.