Warm Pool

A warm pool is a set of machines kept booted and ready ahead of demand, so that a queued job attaches to one of them and skips the wait for a cold boot. The startup work is paid in advance by the pool rather than at the moment someone pushes a commit.

Every pool is the same trade written at a different size. Capacity sits idle so that arriving work finds a machine already there, and the standing cost of that idle capacity buys back the seconds a job would otherwise spend waiting for hardware to exist.

Definition

Three properties make a pool warm. Machines are provisioned before any job asks for them. Each machine is held somewhere along the startup path rather than at the beginning of it. And the pool is drawn down by arriving work and refilled behind it, on a loop that runs continuously.

The word warm describes how far along that path a machine already sits. Startup is a sequence, and a pool can stop at any point in it. Where the pool stops decides both what a job still waits for and what the idle capacity bills for.

Reading the table downward, each row moves work out of the job's critical path and into a standing cost. Reading it upward, each row gives back some of that standing cost and hands the delay to whoever pushed the commit.

What the standing cost looks like

The arithmetic of a pool is short:

standing cost = pool depth x hourly rate of whatever is held x hours held

The rate in that expression depends entirely on the readiness stage. A pool of prebuilt images holds storage, so it bills for storage. A pool of machines booted and then stopped holds an attached disk while the compute is off, so it bills mostly for that disk. A pool of running idle machines holds compute, which is the most expensive row in the table and also the only one where a job waits for nothing at all.

Depth, drain, and refill

Depth is sized against concurrent demand rather than daily volume. Ten thousand jobs a day that never overlap need a depth of one. A single pull request whose test job fans out to six shards on the same machine type needs a depth of six, because those six jobs enter the queue in the same second.

Refill is the second number, and it is the one that usually gets missed. A pool that refills one machine a minute, drained by a burst of six, is back to full six minutes later. Whether that matters depends on how far apart the bursts arrive. Take a fleet with these assumptions, stated so they can be replaced with real ones:

• Every pull request push queues six jobs on the same machine type.
• The pool depth is six.
• Refill produces about one prepared machine a minute and runs continuously.

One push every ten minutes is comfortably served. The burst empties the pool, refill restores all six within six minutes, and the next burst finds a full pool. Two pushes five minutes apart are served differently: the second burst arrives with five machines restored, five of its six jobs attach immediately, and the sixth falls back to cold provisioning. Average demand of 0.6 machines a minute sits under a refill rate of 1.0 a minute, so the fleet keeps up over an hour while individual jobs still miss during the bursts. Depth absorbs the shape of a burst, and refill rate decides whether the pool survives a sequence of them.

A pool of machines and a warm cache are different things

Both are described as warm, and they hold different objects. A warm cache holds data, such as package archives, compiler output, or container layers, and it removes work from inside a job. A warm pool holds machines, and it removes waiting from before the job starts. A job can attach to a prepared machine in a second and then spend four minutes restoring dependencies, which is a full pool serving a cold cache.

What happens when the pool runs dry

A well built pool degrades rather than failing. When nothing is ready, the request falls through to cold provisioning, a fresh machine is created, and the job runs on it. The pool therefore changes when a job starts rather than whether it starts.

That fallback is what makes an undersized pool hard to notice. Nothing errors, no step fails, and the only visible symptom is a Set up job step that takes far longer on some runs than on others. Sorting a week of Set up job durations for one workflow shows the two clusters plainly when a pool is undersized: a tight group at the warm attach time and a second group at the full provisioning time.

Workflow authors have no depth control over pools operated by someone else. GitHub-hosted runners are served from pools GitHub runs, and the workflow file has no key that reaches them. A pool's depth and refill rate are set by whoever operates the fleet the label routes to, not by the workflow that queues a job onto it.

Example

This workflow fans a test suite across six shards. Six jobs enter the queue together every time a pull request is pushed:

name: test

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: [self-hosted, linux, x64, test-pool]
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4, 5, 6]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/6

Nothing in the file mentions a pool. The job asks for a label, GitHub routes it to a runner advertising every label in the list, and the pool sits behind that label as an implementation detail of whoever supplies the machines. Adding, resizing, or removing the pool changes no line of this workflow.

Running the pool at a depth of six with a refill rate of about one machine a minute, and pushing twice within five minutes, produces this sequence:

Eleven of the twelve jobs started on prepared machines and one paid the full provisioning path. That single slow shard sets the wall clock for the whole matrix, because a fan-out finishes when its slowest leg finishes. Raising the depth to twelve covers back to back pushes, and raising the refill rate covers a longer sequence of them.

Implementations differ in which readiness stage they hold and in how the pool is sized. One documented shape for runners in your own cloud account holds each machine booted and then shut down, with the count configured per runner type, described in the standby disks documentation.

Related Terms

• What a cold start is on a GitHub Actions job: the parts of the startup delay a pool removes, and the parts that stay behind inside the job.
• Standby disks for bring-your-own-cloud runners: how a pool of prepared machines is configured for a runner type and what it costs on your own cloud account.
• Removing cold starts from GitHub Actions jobs: how to measure each part of the setup stack from the job log before choosing a fix.
• WarpBuild pricing: per minute rates by runner type.