Code coverage

Code coverage is not the mark of good test coverage - in fact you can have fantastic code coverage, yet terrible test coverage.

However it can be a helpful to understand what is covered and what is not.

Generating a code coverage report is easy, albeit potentially time consuming.

Understanding code coverage in PHPUnit

PHPUnit supports generating code coverage reports and has good documentation on generating code coverage analysis.

To keep it simple there are really two command line arguments that you will need to know about:

HTML code coverage analysis

The following will generate an HTML code coverage analysis that you can explore in your browser - just use the wwwroot in your path.

test/phpunit/phpunit.php run --coverage-html /var/www/html/coverage_report

vendor/bin/phpunit --coverage-html /var/www/html/coverage_report

Clover code coverage analysis

The following generates a code coverage analysis report in the Clover XML format.

test/phpunit/phpunit.php run --coverage-clover /tmp/coverage_report.xml

vendor/bin/phpunit --coverage-clover /tmp/coverage_report.xml

The reason to mention this is PHPStorm (and probably all other popular IDEs) has the ability to import these files and reflect coverage within your IDE.

Whitelists and Totara

In order to produce a code coverage analysis you need to provide a whitelist configuration.

Failure to do so will result in the following error (aka warning):

Defining a whitelist is easily done via a small modification to the phpunit.xml that is generated after you have initialised PHPUnit.

Make sure you init phpunit first!

php test/phpunit/phpunit.php init

php admin/tool/phpunit/cli/init.php

Add the following to your phpunit.xml to whitelist a group of files within Totara.

    <filter>
        <whitelist processUncoveredFilesFromWhitelist="false" addUncoveredFilesFromWhitelist="false">
            <directory suffix=".php">totara/core/classes/local</directory>
            <directory suffix=".php">totara/core/classes/task</directory>
            <file>totara/core/classes/visibility_adviser.php</file>
        </whitelist>
    </filter>

This should be added after the <php> tag.

Once the whitelist is in place you are ready to go.

If you wanted to run code coverage for the whole of Totara the following is the filter block we would recommend using:

    <filter>
        <whitelist processUncoveredFilesFromWhitelist="false" addUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./admin</directory>
            <directory suffix=".php">auth</directory>
            <directory suffix=".php">availability</directory>
            <directory suffix=".php">backup</directory>
            <directory suffix=".php">badges</directory>
            <directory suffix=".php">blocks</directory>
            <directory suffix=".php">blog</directory>
            <directory suffix=".php">cache</directory>
            <directory suffix=".php">calendar</directory>
            <directory suffix=".php">cohort</directory>
            <directory suffix=".php">comment</directory>
            <directory suffix=".php">completion</directory>
            <directory suffix=".php">course</directory>
            <directory suffix=".php">dataformat</directory>
            <directory suffix=".php">elementlibrary</directory>
            <directory suffix=".php">enrol</directory>
            <directory suffix=".php">error</directory>
            <directory suffix=".php">files</directory>
            <directory suffix=".php">filter</directory>
            <directory suffix=".php">grade</directory>
            <directory suffix=".php">group</directory>
            <directory suffix=".php">iplookup</directory>
            <directory suffix=".php">lib</directory>
            <directory suffix=".php">local</directory>
            <directory suffix=".php">login</directory>
            <directory suffix=".php">media</directory>
            <directory suffix=".php">message</directory>
            <directory suffix=".php">mnet</directory>
            <directory suffix=".php">mod</directory>
            <directory suffix=".php">my</directory>
            <directory suffix=".php">notes</directory>
            <directory suffix=".php">plagiarism</directory>
            <directory suffix=".php">portfolio</directory>
            <directory suffix=".php">question</directory>
            <directory suffix=".php">rating</directory>
            <directory suffix=".php">report</directory>
            <directory suffix=".php">repository</directory>
            <directory suffix=".php">rss</directory>
            <directory suffix=".php">search</directory>
            <directory suffix=".php">tag</directory>
            <directory suffix=".php">totara</directory>
            <directory suffix=".php">user</directory>
            <directory suffix=".php">webservice</directory>
        </whitelist>
    </filter>

In general the more you whitelist and test the longer the execution will take.

Extensions

Generating a code coverage analysis requires more than just PHP and PHPUnit.

The three most commonly talked about are:

• PHPDBG is a SAPI module bundled with PHP since 5.6
• XDebug is a PHP debugging extension that supports code coverage analysis
• pcov is a PHP 7 code coverage extension

All allow us to generate a code coverage analysis, albeit with different results. Xdebug is commonly installed by developers here as not only does it facilitate code coverage but it integrates well with many IDEs and is a full-blown debugging extension with many great offerings. It is however much slower when generating a code coverage analysis, by an order of magnitude.

PHPDBG is very poorly documented, and provides some really interesting functionality if you are keen to wade through some deep waters. However it is reasonably simple to use and pretty quick. pcov requires a bit more to get running because we use PHPUnit 7, however it appears to perform best of all.

Here are some sample timings from running code coverage on a visibility refactor patch using PHPDBG, XDebug and pcov:

Notice the difference in time whitelist has as well.

In addition preparing the actual coverage analysis report takes time, and each extension has its own timings.

They tend to be representative of the test execution time, increasing it by around 25%.

Code coverage with XDebug

1. Make sure XDebug is installed and enabled.
   php -m | grep xdebug
   If that returns xdebug then it's ready to go, otherwise you will need to install and enable it.
2. Ensure you have modified phpunit.xml to add your whitelist.
3. Totara 13 and above

   php test/phpunit/phpunit.php run --filter=totara_core --coverage-clover /tmp/coverage.xml

   Totara 12 and below

   vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml
   
   

When run directly like this XDebug code coverage can take a very long time.

One method of speeding up the processing is to delegate whitelist functionality to XDebug. This vastly improves code coverage analysis.

To do this:

1. Prepare a filter file for XDebug.

   Totara 13 and above

   php test/phpunit/phpunit.php run --dump-xdebug-filter /tmp/xdebug_filter.php

   Totara 12 and below

   vendor/bin/phpunit --dump-xdebug-filter /tmp/xdebug_filter.php

2. Execute coverage with the filter file included.

   Totara 13 and above

   php test/phpunit/phpunit.php run --prepend /tmp/xdebug_filter.php --filter=totara_core --coverage-clover /tmp/coverage.xml

   Totara 12 and below

   vendor/bin/phpunit --prepend /tmp/xdebug_filter.php --filter=totara_core --coverage-clover /tmp/coverage.xml

Code coverage with PHPDBG

This is done on the command line:

phpdbg -qrr -d memory_limit=-1 vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

Code coverage with pcov

pcov is an alternative to XDebug and PHPDBG, with one big advantage being its speed.

It's designed for PHPUnit 8, however there is a composer package that helps by enabling compatibility with PHPUnit 7 (which we use).

To get it running:

1. Disable XDebug if you have it enabled presently.
2. pecl install pcov.

3. If you are running coverage on Totara 12 or below you will need to install pcov/clobber. If testing 13 or above skip this step.

   composer require pcov/clobber
   vendor/bin/pcov clobber

4. Ensure PHP has enough memory in php.ini. Give it heaps, as code coverage is memory intensive.

   memory_limit=8G

5. Configure pcov (within php.ini):

   [pcov]
   extension="/path/to/pcov.so"
   pcov.enabled=1
   pcov.exclude='~(vendor|tests|node_modules|.git|client|.scannerwork)~'
   pcov.initial.memory=1073741824
   pcov.initial.files=30000
   
   

6. Run the tests:

   Totara 13 and above

   cd test/phpunit
   php -d pcov.directory=`pwd` vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

   Totara 12 and below

   cd /path/to/code
   php -d pcov.directory=`pwd` vendor/bin/phpunit --filter=totara_core --coverage-clover /tmp/coverage.xml

Alternative installation methods exist if you are using Fedora, Ubuntu or Debian. See pcov install.md.

PHPStorm and XDebug/pcov

PHPStorm has the ability to run code coverage anaylsis in product using XDebug or pcov. The analysis is automatically loaded into the platform and displayed both in the Coverage tool, and in-editor when looking at files. Their documentation on code coverage explains how to set it up.

A few pointers and tips to help however:

• You still need to define that whitelist
• As a product we support several different versions of PH - if you're using php-fpm and have them installed you can configure PHPStorm so that you can choose which version you run tests and coverage on
• When looking at a testcase file you can click on the little arrows next to test case classes and test case methods and choose to run specific tests

PHPStorm and PHPDBG

PHPStorm does not have the ability to use PHPDBG for code coverage.

However you can import clover format code coverage analysis reports into PHPStorm.

To do so:

1. Go to Tools > Show code coverage data.
2. Click the + icon (top left).
3. Locate your clover report and click Open.
4. Make sure your file is checked.
5. Click Show selected.

Practical coverage within Totara

Further reading

• Unit testing in Totara
• External: PHPUnit documentation on code coverage analysis
• External: XDebug code coverage analysis
• External: PHPDBG documentation
• External: pcov repository and documentation