Long Running Jobs on GitHub Actions

A GitHub Actions job that runs for hours ends in one of three ways: the timeout-minutes value cancels it, a platform execution limit cancels it, or one step hangs and consumes the rest of the job's budget before the later steps get to run. The fix is to stop shipping the work as a single job and split it into stages that checkpoint their state and hand it forward, so a failure at hour four costs one stage rather than the whole run.

This guide names the three ceilings and how to tell which one you hit, gives the workflow YAML for a checkpointed pipeline, and prices one long job against the staged version of the same work.

Diagnosis

Start by separating the three ceilings, because the remedy differs for each and the log looks similar for two of them.

Values below were checked on 2026-08-13 against GitHub's limits reference, the workflow syntax reference, and the automatic token authentication guide.

The timeout you did not set

jobs.<job_id>.timeout-minutes defaults to 360. A job that legitimately needs seven hours is cancelled at minute 360 with no configuration mistake anywhere in the file. Read the run duration first, because a job that ends at exactly 360 minutes ended on the default, and the answer on how to set a timeout on a GitHub Actions job covers the job and step keys that replace it.

The platform limit under the timeout

The lowest applicable ceiling wins, so raising timeout-minutes past the runner's execution limit changes nothing. Which limit applies depends on the pool. GitHub's own fleet stops a job at 6 hours. Registered runners get 5 days, and WarpBuild registers itself as a self-hosted runner in the Default runner group (id 1) of your organization, per the public repositories documentation, so a WarpBuild job runs under the 5 day ceiling rather than the 6 hour one.

That headroom is worth less than it sounds. Five days on warp-ubuntu-latest-x64-16x at $0.032 per minute is 7,200 minutes and $230.40 for one job nobody is watching, which is why every stage in the configuration below carries its own timeout-minutes.

The GITHUB_TOKEN ceiling is the one that surprises people on long jobs. The token issued for a run expires after at most 24 hours, so a job planning to push a tag at hour 30 loses its credentials before the execution limit ever fires. The job timeout glossary entry lists that ceiling next to the others.

The step that hangs

A job with a job level timeout and no step level timeouts spends its entire remaining budget inside whichever step blocks. A network fetch against a dead mirror, an integration suite waiting on a service that never binds, or a prompt on stdin all present the same way: one step in the timing view holds the tail of the run and the steps after it never start.

Two checks separate a long job from a stuck one. Compare the step timings across the last five runs and look for the step whose duration jumped rather than grew. Then confirm the clock was running at all, because a job waiting for a runner has not started executing and no timeout applies to it; the common issues documentation covers runner group access, bot permissions, and workflow restrictions, which are the usual reasons a job sits unclaimed.

Fix

Split the work at boundaries where the finished state is cheap to hand forward, and give every stage a timeout it can actually hit.

Three transports move state between stages, and the right one depends on the size of what survives.

• Artifacts. actions/upload-artifact and actions/download-artifact work from any runner and any platform. Upload and download time bills as runner minutes, so this fits reports, binaries, and test fixtures rather than a 30 GB build tree.
• Cache actions. Keyed restores fit dependency directories that change on a lockfile bump. Cache storage bills $0.20 per GB-month and cache operations $0.0001 each, from the pricing page.
• Snapshot runners. A snapshot runner captures the whole runner disk mid-workflow and later jobs boot from that image, which is the transport that fits an entire working tree. Restores bill $0.04 per job and storage $0.025 per snapshot-hour. Booting from a snapshot takes 45 to 60 seconds, snapshots are deleted after 15 days, and the feature is supported on WarpBuild Cloud Ubuntu runners only, per the snapshot runners documentation. The /tmp directory is cleaned on reboot and a snapshot boot is a reboot, so the work directory belongs under $HOME.

Four rules make the split hold up in practice.

Cut where the state is small or the disk is large. A boundary that produces a 40 MB artifact wants upload-artifact. A boundary that leaves a compiled tree, a populated Docker image store, and a warm dataset on disk wants a snapshot, since none of that fits a path: list.

Make every stage idempotent and guard it with a checkpoint marker. Each stage writes $HOME/.checkpoints/<stage>.done containing github.sha, and the next stage fails fast when the marker is missing or holds a different commit. A re-run keeps the same GITHUB_SHA and GITHUB_REF as the original event (re-running workflows and jobs, checked on 2026-08-13), so a marker keyed on the commit still validates on the retry and rejects yesterday's alias.

Size the timeout per stage from its own p95. A single 360 minute value spread across four stages tells you nothing about which one hung.

Keep the stage count low. Every stage pays a boot, a restore, and a queue wait. Four stages out of a five hour job is a reasonable ratio; twenty is a pipeline that spends its savings on overhead.

Work that runs long because it is repeating itself needs a different fix. Boot the machine that already holds the previous result, following incremental builds on GitHub Actions, and start from the guide to speeding up GitHub Actions for the ordinary caching, sizing, and parallelism moves before splitting anything.

Configuration

A nightly simulation pipeline, roughly 320 minutes of work, cut into four stages. Stage one boots from the base image and saves an alias; each later stage boots from the alias the previous stage wrote and saves its own.

name: nightly-simulation

on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch:

jobs:
  compile:
    runs-on: warp-ubuntu-latest-x64-16x;snapshot.enabled=true
    timeout-minutes: 100
    steps:
      - uses: actions/checkout@v5
      - name: Build the simulator
        run: make -j16 -C $HOME/work simulator
      - name: Write checkpoint
        run: |
          mkdir -p $HOME/.checkpoints
          echo "${{ github.sha }}" > $HOME/.checkpoints/compile.done
      - name: Cleanup credentials
        run: rm -rf $HOME/.ssh $HOME/.aws
      - uses: WarpBuilds/snapshot-save@v1
        with:
          alias: nightly-compile
          fail-on-error: true
          wait-timeout-minutes: 60

  dataset:
    needs: compile
    runs-on: warp-ubuntu-latest-x64-16x;snapshot.key=nightly-compile
    timeout-minutes: 105
    steps:
      - name: Verify checkpoint
        run: |
          test "$(cat $HOME/.checkpoints/compile.done)" = "${{ github.sha }}"
      - name: Generate the dataset
        run: $HOME/work/bin/gen-dataset --out $HOME/work/data
      - name: Write checkpoint
        run: echo "${{ github.sha }}" > $HOME/.checkpoints/dataset.done
      - uses: WarpBuilds/snapshot-save@v1
        with:
          alias: nightly-dataset
          fail-on-error: true
          wait-timeout-minutes: 60

  simulate:
    needs: dataset
    runs-on: warp-ubuntu-latest-x64-16x;snapshot.key=nightly-dataset
    timeout-minutes: 120
    steps:
      - name: Verify checkpoint
        run: |
          test "$(cat $HOME/.checkpoints/dataset.done)" = "${{ github.sha }}"
      - name: Run the simulation
        run: $HOME/work/bin/simulate --data $HOME/work/data --out $HOME/work/runs
      - name: Write checkpoint
        run: echo "${{ github.sha }}" > $HOME/.checkpoints/simulate.done
      - uses: WarpBuilds/snapshot-save@v1
        with:
          alias: nightly-simulate
          fail-on-error: true
          wait-timeout-minutes: 60

  report:
    needs: simulate
    runs-on: warp-ubuntu-latest-x64-16x;snapshot.key=nightly-simulate
    timeout-minutes: 95
    steps:
      - name: Verify checkpoint
        run: |
          test "$(cat $HOME/.checkpoints/simulate.done)" = "${{ github.sha }}"
      - name: Summarize
        run: $HOME/work/bin/report --runs $HOME/work/runs --out report.html
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: simulation-report
          path: report.html

Four details carry the behavior. snapshot.enabled=true boots from the base image and turns the feature on, while snapshot.key=<alias> turns it on and boots from that alias when one exists, which is the pairing the snapshot documentation describes. fail-on-error: true turns a capture that did not finish into a failed stage, so a later stage never boots from a stale alias behind a green predecessor. The work directory sits under $HOME rather than /tmp. And the credential cleanup runs before the first capture, since anything on disk at capture time is in the image every later stage boots.

When stage three fails, gh run rerun RUN_ID --failed restarts it from the nightly-dataset alias with the same commit SHA. A run stays re-runnable for 30 days and accepts at most 50 re-runs.

This shape is Linux only. Snapshots are supported on WarpBuild Cloud Ubuntu runners only, so a macOS or Windows pipeline hands state forward through artifacts and caches instead. Snapshot runners sit alongside remote Docker builders, CI observability, an MCP server, and the Action Debugger in the product surface, and every stage above stays a plain GitHub Actions job.

Cost or Time Model

Assumptions, stated so you can substitute your own timings:

• 320 minutes of work per nightly run, 30 runs a month, on warp-ubuntu-latest-x64-16x at $0.032 per minute from the pricing page.
• Six of the 30 runs fail. In the single job shape a failure lands on average at minute 250 and the retry repeats all 320 minutes. In the staged shape the failure lands 50 minutes into stage four, and the retry repeats only that stage.
• The staged shape splits into four stages averaging 80 minutes, each paying a one minute boot, and three snapshot restores per run.
• Three aliases stay live continuously at $0.025 per snapshot-hour, which is 3 x 730 hours = 2,190 snapshot-hours a month.

The staged pipeline costs $24.03 more a month at this failure rate, and that is the honest reading of the trade. What it buys is recovery time. A failed night runs 570 minutes end to end in the single job shape and 374 in the staged one, because the retry is one 81 minute stage rather than a fresh 320 minute run, so the result lands 196 minutes earlier and no stage sits anywhere near a platform ceiling.

The break-even moves with the failure count. Each failure costs the single job 250 extra minutes and the staged pipeline 50, a difference of $6.40. The staged pipeline carries $62.62 a month of fixed overhead, being $7.87 of boots and restores (126 boot-minutes at $0.032 plus 96 restores at $0.04) plus $54.75 of storage, so it turns cheaper past roughly 10 failed runs a month. Below that, split for the ceilings and the feedback loop rather than for the invoice.

Against GitHub-hosted list prices

warp-ubuntu-latest-x64-16x (16 vCPU, 64 GB) costs $0.032 per minute against $0.042 per minute for the 16-core Linux larger runner (16 vCPU, 64 GB): 24 percent lower list price (GitHub Actions minute multipliers, GitHub list price checked on 2026-08-13). The same 11,100 minutes of the single job shape bill $466.20 at that rate against $355.20 here. Neither column changes the ceilings, and the 6 hour execution limit on GitHub's fleet leaves a 320 minute job with 40 minutes of headroom before a slow night becomes a cancelled one.

FAQ

What is the longest a GitHub Actions job can run?

Six hours on a GitHub-hosted runner and five days on a self-hosted runner, with the whole workflow run capped at 35 days, per GitHub's limits reference checked on 2026-08-13. A job also stops at its own timeout-minutes value, which defaults to 360 minutes, so the default cancels a job on a self-hosted runner long before the platform ceiling does.

How do I resume a GitHub Actions job that failed after four hours?

There is no resume for a single job. Split the work into stages chained with needs, have each stage write a checkpoint the next one boots from, and re-run the failed stage with gh run rerun RUN_ID --failed. A re-run keeps the same GITHUB_SHA and GITHUB_REF as the original event, so a checkpoint keyed on the commit still validates.

Does splitting a long job into stages make it cheaper?

Not on its own. Each stage adds a boot and a restore, so a four-stage split of a 320 minute job bills about 4 extra minutes per run plus $0.12 in snapshot restores. The saving comes from failures, since a failed stage repeats 81 minutes instead of 320, and the split pays for itself past roughly 10 failed runs a month at the rates in the model above.