Visual regression tests#

We’re using BackstopJS to run visual regression tests, shortly known as VRT.

Hint

Read Leornado Giroto’s Visual Regression Testing blog post if you want to get a nice introduction.

Setup VRT#

  1. Install BackstopJS in your repository

  2. Create a config by running backstop init, or copy an existing one (e.g. website or products)

  3. Create the required visual regression test targets in the Makefile

  4. Add the shared VRT CI file CI file to your GitLab CI pipeline

VRT workflow#

To run visual regression tests, you can use one of the obvious VRT test targets. The workflow is usually like this:

  1. The tests are run by running make test-vrt

  2. In case there’s an error the report is opened by running make vrt-report

  3. Each failed test case will be inspected

For each failed test case you’ve to make one decision:

  • Unwanted respectively unexpected changes:
    Fix the error and run the VRT tests again by running make test-vrt.
  • Wanted respectively expected changes:
    Approve it by either running make vrt-approve or using the Interactive VRT report.

Hint

Please note make vrt-approve will always approve all failed tests. Thus using the Interactive VRT report to approve tests individually, or fixing all unwanted / unexpected changes first is highly recommended.

Interactive VRT report#

BackstopJS also supports interactive web reporting, which allows you to approve tests directly in the browser, instead via CLI.

To approve tests interactively, you must run the interactive server in a separate terminal first:

make vrt-remote

The interactive server is automatically detected when you open a new report, for example by running make vrt-report. Each failed test should then show an Approve button in its header.

OS differences in VRT#

There are OS differences, for example font renderings between Linux and macOS.

Since most of our clients are macOS, the Visual Regression Test job of the shared VRT CI file is also running on a Mac.

Important

Approving new reference files should therefor only happen on macOS.

Alternatively you can also trigger the CI/CD pipeline, wait until the visual regression tests failed, download the artifact and replace the corresponding files in bitmaps_reference/ directory with the ones of the bitmest_test/YYYYMMDD-hhmmss/ artifact directory.