Code Coverage
November 2, 2024
NOTE: I originally wrote this post to show how to fix incorrect 0% code coverage results. I was wrongly convinced that the coverageFailOnMinimum setting flag was the culprit, but it turned out to be SBT shell issues with Scoverage. Before publishing the post I rewrote it to be about basic Scoverage usage. Enjoy!
Scoverage is a code coverage tool in Scala. It generates coverage reports to show you what branches and statements have been executed in your tests. Using the coverage flag will gather data about what code pathways were traversed in your codebase.
sbt clean coverage test
Clean removes any artifacts in the build directory (target), Coverage enables the coverage flag for the compiler, and test runs the test suites. The data collected during the tests isn't friendly for human consumption and needs to be formatted in a more palatable way. To format the data, you need to generate a coverage report:
sbt coverageReport
You should see the code coverage report summary in the terminal:
[info] Waiting for measurement data to sync... [info] Reading scoverage instrumentation […/target/coverage/scoverage-data/scoverage.coverage] [info] Reading scoverage measurements... [info] Generating scoverage reports... [info] Written Cobertura report […/target/coverage/coverage-report/cobertura.xml] [info] Written HTML coverage report […/target/coverage/scoverage-report/index.html] [info] Statement coverage.: 41.36% [info] Branch coverage....: 50.93% [info] All done. Coverage was stmt=[41.36%] branch=[50.93%] [success] Total time: 0 s, completed Nov 2, 2024, 8:51:35 a.m.
Awesome it generated a report! If you have limited, but not zero test coverage in your project, the summary should tell you the statement and branch coverage of your code base. If you have tests and you see 0%, make sure you are NOT running SBT shell. I find the coverage and coverageReports commands generate reliable results when not using SBT shell, eg. execute them direct from the terminal command line.
Minimum Coverage
If your code base has decent code coverage, consider setting the coverageFailOnMinimum flag along with minimum coverage requirements. This prevents the coverageReport from succeeding if coverage drops below certain thresholds. In your build.sbt file, add the following line somewhere at the bottom of the file, you can move it elsewhere in the build file if you want:
coverageFailOnMinimum := true
Just adding this line wont do much without minimum values. Below are the default recommended values from Scoverage documentation. Set the values to whatever works best for your project at this time.
coverageMinimumStmtTotal := 90 coverageMinimumBranchTotal := 90 coverageMinimumStmtPerPackage := 90 coverageMinimumBranchPerPackage := 85 coverageMinimumStmtPerFile := 85 coverageMinimumBranchPerFile := 80
Re-run the previous commands used earlier: sbt clean coverage test and sbt coverageReport. If you set really aggressive minimum values that are higher than your current code coverage, you'll see an exception in your terminal.
[info] Coverage reports completed [error] Coverage is below minimum [50.23% < 90.00%]: Branch:Total [error] java.lang.RuntimeException: Coverage minimum was not reached
This is helpful if your team is still adopting test driven development, and they have a bad habit of adding code without tests — you'll know who broke the build.
HTML Report
If you need to consume the coverage reports in more detail, the HTML report allows you to browse line by line what code is covered. In your build.sbt file, near the previous line you added, add the following:
coverageOutputHTML := true
If you'd like, you can also set the output directory for the coverage reports by adding:
coverageDataDir := target.value / "coverage"
This will place all of the coverage information in the directory target/coverage. The default location buries the reports in target/<scala-version>.
scoverage-report directory contains the HTML output of the generated report: target/coverage/scoverage-report
Re-run the previous commands used earlier: sbt clean coverage test and sbt coverageReport. Open the directory target/coverage, there will be three new directories: coverage-report, scoverage-data and scoverage-report. scoverage-report contains the HTML output of the generated report. The index.html report file shows a coverage breakdown of the files found in the project.
Similar to other code coverage reporting tools in different languages, Scoverage allows you to click on individual files to see the line by line breakdown of what statements and branches are covered.
As of Scala 3.3.4 most major language features are supported in code coverage data. For-comprehensions were recently fixed in Scala 3.3.3 so almost everything between Scala 2 and 3 should give similar code coverage results now.
Dependencies
The following dependencies were used in this post.
Attributions
- The main post image was generated with Leonardo.AI, using the prompt "An ornate, intricately detailed logo in the style of vintage gilt tarot cards, evoking a sense of mysticism and mystique, with a Steampunk twist, featuring a bold, cursive script with subtle, distressed textures, reading "Scoverage" in a flowing, art nouveau-inspired font, surrounded by delicate, filigree-like patterns and ornate, gothic-inspired flourishes, set against a rich, dark background with subtle, warm golden undertones, reminiscent of antique, worn leather, with hints of copper and brass accents, and subtle, eerie flickers of candlelight casting an air of mystery and otherworldliness."