Concurrent tests in GitHub Actions

Running builds and tests are probably the most common use-cases for GitHub Actions. Modern test frameworks and build systems have built-in support for sharding tests and remote execution. Sharding is essentially running your tests or builds in parallel, not only on different threads or cores but distributing them across different machines altogether.

Concurrent tests for popular test frameworks

This guide details how to set up concurrent tests for some of the popular test frameworks. We will explore how to set up test sharding in GitHub Actions for Jest, Playwright and Pytest.

Jest

Jest is a popular JavaScript testing framework. It provides native support for test sharding using the --shard option to run your tests in parallel simultaneously across multiple machines. The option takes an argument in the form of shardIdx/shardCount, where shardIdx is a number representing index of the shard and shardCount is the total number of shards.

A simple benchmark on a dummy test suite run showed that sharding improves the run time from 3 minutes to 30 seconds.

Here's how you can set it up in GitHub Actions:

1name: CI
2on: push
3
4jobs:
5  test:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        shardCount: [10]
13        shardIdx: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-node@v4
18      - run: npm ci
19
20      - name: Run Jest tests
21        run: npx jest --shard=${{ matrix.shardIdx }}/${{ matrix.shardCount }}

Playwright

Playwright is a popular end-to-end automation and testing framework for web applications. Native support for test sharding is available using the --shard option. Just like we saw with Jest earlier, the --shard option takes an argument in the form of shardIdx/shardCount, where shardIdx is the index of the shard and shardCount is the total number of shards.

Sharding the tests improve the time from around 5 minutes 26 seconds to 1 minute 25 seconds for a dummy test suite run.

The setup is very similar to that of Jest:

1name: CI
2on: push
3
4jobs:
5  tests:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        shardCount: [10]
13        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-node@v4
18      - run: npm ci
19
20      - name: Install Playwright browsers
21        run: npx playwright install --with-deps
22
23      - name: Run Playwright tests
24        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardCount }}

Pytest

Pytest is a widely used testing framework for Python. Although Pytest doesn't natively support test sharding across machines, the third-party plugin pytest-split can distribute tests based on duration or name.

The performance gain after sharding was really significant since Pytest doesn't run tests in parallel by default. The dummy test runs showed a 10x improvement in test run times – from 500 to 50 seconds.

Setting up the workflow involves installing the pytest-split package and running pytest with the --splits and --group options:

1name: CI
2on: push
3
4jobs:
5  test:
6    name: Tests
7    runs-on: warp-ubuntu-latest-x64-4x
8
9    strategy:
10      fail-fast: false
11      matrix:
12        splitCount: [10]
13        group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
14
15    steps:
16      - uses: actions/checkout@v4
17      - uses: actions/setup-python@v5
18      - run: pip install pytest pytest-split
19
20      - name: Run pytest
21        run: pytest --splits ${{ matrix.splitCount }} --group ${{ matrix.group }}

Comparison and Notes

All of the test suites used, workflows and their runs can be found in our concurrent-tests repository. Here is a comparison the test run performance before and after sharding for the three test frameworks:

Limitations

While GitHub Actions' matrix strategy is really handy for parallelizing our tests and builds, a constraint that can't be ignored for performance is the imposed limit on number of concurrent jobs. GitHub has a limit of 20 concurrent jobs per account. You can increase this limit to 40 on a pro account or 60 on a team account but it is still very limiting for large teams.

Further improvements

Overall CI workflow times with GitHub actions can be reduced by parallelizing tests. Run times can be further improved by making the tests within each shard run faster by using the native parallelization features of the test frameworks or by using third-party packages.

WarpBuild provides blazing fast runners, optimized for CPU, network, and disk performance that are 30% faster at half the cost. At WarpBuild, there are no limits on the number of concurrent jobs. You can run as many jobs as you want in parallel and minimize the wait times on GitHub Actions workflows. It takes only ~2 minutes to get started.