Visual regression tests
=======================

We're using `BackstopJS <https://github.com/garris/BackstopJS>`_ to run visual regression tests, shortly known as `VRT`.

.. hint::

    Read `Leornado Giroto's Visual Regression Testing <https://medium.com/loftbr/visual-regression-testing-eb74050f3366>`_ blog post if you want to get a nice introduction.

Setup VRT
---------

1. Install `BackstopJS`_ in your repository
2. Create a config by running ``backstop init``, or copy an existing one (e.g. `website <https://git.confirm.ch/confirm/marketing/website/-/blob/master/backstop.js>`_ or `products <https://git.confirm.ch/confirm/marketing/products/-/blob/master/backstop.js>`_)
3. Create the required visual regression :ref:`test targets <Test targets>` in the :ref:`Makefile`
4. Add the `shared VRT CI file <https://git.confirm.ch/confirm/gitlab-ci/-/tree/master/vrt>`_ CI file to your :ref:`GitLab CI` pipeline

VRT workflow
------------

To run visual regression tests, you can use one of the obvious VRT :ref:`test targets <Test targets>`. 
The workflow is usually like this:

1. The tests are run by running ``make test-vrt``
2. In case there's an error the report is opened by running ``make vrt-report``
3. Each failed test case will be inspected

For each failed test case you've to make one decision:

- | **Unwanted respectively unexpected changes**:
  | Fix the error and run the VRT tests again by running ``make test-vrt``.
- | **Wanted respectively expected changes**:
  | Approve it by either running ``make vrt-approve`` or using the `Interactive VRT report`_.

.. hint::

    Please note ``make vrt-approve`` will always approve all failed tests.
    Thus using the `Interactive VRT report`_ to approve tests individually, or fixing all unwanted / unexpected changes first is highly recommended.

Interactive VRT report
----------------------

`BackstopJS`_ also supports `interactive web reporting <https://github.com/garris/BackstopJS#interactive-web-reporting>`_, which allows you to approve tests directly in the browser, instead via CLI.

To approve tests interactively, you must run the interactive server in a separate terminal first:

.. code-block:: bash

    make vrt-remote

The interactive server is automatically detected when you open a new report, for example by running ``make vrt-report``. Each failed test should then show an ``Approve`` button in its header.

OS differences in VRT
---------------------

There are OS differences, for example font renderings between Linux and macOS.

Since most of our clients are macOS, the ``Visual Regression Test`` job of the `shared VRT CI file`_ is also running on a Mac.

.. important::

    Approving new reference files should therefor only happen on macOS. 

    Alternatively you can also trigger the CI/CD pipeline, wait until the visual regression tests failed, download the artifact and replace the corresponding files in ``bitmaps_reference/`` directory with the ones of the ``bitmest_test/YYYYMMDD-hhmmss/`` artifact directory.