Faster Behat Tests

The template's behat tests job runs one scenario. Almost all of its time is spent starting the container and Postgres, so there is nothing here to speed up. This page is for a project whose own suite has grown to hundreds of scenarios.

It is guidance, not a template default. Sharding a small suite only adds runner cost, and GitLab's parallel: count is fixed in the YAML, so it can't size itself to the suite.

Measure first

Two numbers tell you which fix you need:

• The reset's share of the run. Time the @BeforeScenario hook in api/features/bootstrap/DoctrineContext.php and compare it with the whole suite. In that project, rebuilding the schema was 66% of the run time, spread evenly across scenarios.
• The per-file distribution. run_test_behat writes JUnit XML to api/build/logs/behat/junit/. Each feature file is a <testsuite> and each scenario a <testcase>, both with a time attribute. If a few files take most of the time, fix those files instead.

Reset the database without rebuilding the schema

The template's DoctrineContext drops and recreates the whole schema before every scenario. That project measured four ways to reset:

Truncate mode is a trap: the purger runs one statement per table, and each is its own transaction. Delete mode deletes in Doctrine's commit order, so foreign keys need no special handling. ORMPurger comes from doctrine/data-fixtures, which the template already installs through doctrine/doctrine-fixtures-bundle.

Build the schema once, for the first scenario, then purge:

<?php

declare(strict_types=1);

namespace App\Features\Bootstrap;

use Behat\Behat\Context\Context;
use Doctrine\Common\DataFixtures\Purger\ORMPurger;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\Tools\SchemaTool;
use Doctrine\Persistence\ManagerRegistry;

class DoctrineContext implements Context
{
    private static bool $schemaCreated = false;    private EntityManagerInterface $manager;

    public function __construct(ManagerRegistry $doctrine)
    {
        $this->manager = $doctrine->getManager();
    }

    /**
     * @BeforeScenario
     */
    public function resetDatabase(): void
    {
        if (!self::$schemaCreated) {            $classes = $this->manager->getMetadataFactory()->getAllMetadata();
            $schemaTool = new SchemaTool($this->manager);
            $schemaTool->dropSchema($classes);
            $schemaTool->createSchema($classes);
            self::$schemaCreated = true;
        } else {
            (new ORMPurger($this->manager))->purge(); // PURGE_MODE_DELETE is the default            $this->restartSequences();        }
        $this->manager->clear();
    }

    private function restartSequences(): void
    {
        $connection = $this->manager->getConnection();
        $sequences = $connection->fetchFirstColumn(
            "SELECT quote_ident(sequence_schema) || '.' || quote_ident(sequence_name)
             FROM information_schema.sequences WHERE sequence_schema = current_schema()"
        );
        foreach ($sequences as $sequence) {
            $connection->executeStatement('ALTER SEQUENCE '.$sequence.' RESTART');
        }
    }
}

The static flag lives for the whole Behat process, so each run (and each shard) builds the schema exactly once.

That project also tried DAMA\DoctrineTestBundle, which wraps each scenario in a rolled-back transaction. It was a new dependency for about 3% more, and it doesn't roll back sequences.

The sequence trap

A delete leaves sequences where they were, so a purged database hands out new ids. Any feature that asserts an IRI such as /_api/…/1 then fails, depending on which scenarios ran before it.

• A sequence that Doctrine creates for the SEQUENCE id strategy is a standalone sequence that no column owns. TRUNCATE … RESTART IDENTITY only restarts sequences owned by the truncated tables' columns, so it leaves these alone and the ids drift. That project hit exactly this.
• Only an explicit ALTER SEQUENCE … RESTART for each sequence, as in restartSequences() above, keeps ids stable with either reset.

CWA's own entities use UUIDs, and the template's entities do too, so a project with no integer ids has no sequences to restart.

Shard the job by scenario

Once the reset is cheap, the fixed cost of each job dominates. At that point, split the suite across parallel jobs with GitLab's parallel: N. Each copy of the job gets its own postgres service, and CI_NODE_INDEX (from 1) and CI_NODE_TOTAL tell it which share to run:

behat tests:
  parallel: 4  script:
    - setup_test_db_environment
    - run_test_behat

run_test_behat passes Behat no paths, so change it to pass this shard's share. Behat takes several paths and runs only those, and a path can name one scenario as features/some.feature:LINE. Your shard script picks this job's list, and run_test_behat adds it to the behat command.

• Shard by scenario, not by feature file. In that project one file held 100 of the 211 scenarios, so no split by file could balance it. Weight each scenario by how many it runs (a Scenario Outline counts each Examples row), and assign the heaviest first, each to the lightest shard so far.
• An empty shard must exit 0 without starting Behat. Behat with no paths runs the whole suite, so a shard that got nothing would run everything:

  [ -z "$SHARD_PATHS" ] && { echo "No scenarios for shard $CI_NODE_INDEX"; exit 0; }
  

• Choose N from the fixed cost per job. It was about 81s there, so extra shards soon cost more runner minutes than they save. N=4 was the sweet spot. N=8 doubled the runner minutes to save about 55s.
• Share one vendor cache across shards. The template's behat tests job runs composer install and has no cache. If you add one, key it on api/composer.lock. A key that includes the job name or branch makes every shard of a new branch start cold.

Each shard writes its own JUnit file, and GitLab merges the reports of parallel jobs in the test report.

Prove the order doesn't matter

A reset that leaves anything behind makes a scenario depend on the ones before it. Sharding changes that order, so check it before you trust the shards.

Behat's --order=random and --order=reverse only reorder whole feature files. The scenarios inside a file keep their order. Build the orders yourself from the list of scenarios instead, pass each as paths, and run the full suite:

• in its default order,
• fully reversed,
• under several random seeds.

Results in that project

The runner minutes roughly halved as well.

See CI/CD for the rest of the pipeline.