Can I Alert When a Runner Runs Out of Memory?

Answer

Yes. Resource threshold alerts are part of the observability feature: the Recommendations view filters and highlights every runner instance whose max memory utilization reaches 80 percent and labels it High Memory Usage, per the observability documentation. The signal worth alerting on is sustained pressure across a week of runs rather than one spike, because a single job that touched 81 percent once proves nothing and a job whose p75 sits at 80 percent is running out of room on every pull request.

Four thresholds ship with the Recommendations view, and memory is one of them.

Memory utilization here is the maximum memory usage percentage recorded for the instance while the job ran, collected by an agent on the runner over OpenTelemetry on port 33931. Two collection details change what you will see. Observability records metrics and logs only for jobs longer than about one minute, so a fleet of 30 second jobs shows gaps, and the numbers cover the whole machine rather than one process, so a test runner and a database container in the same job land in the same reading.

GitHub Actions itself reports none of this. The job object in the workflow jobs REST API carries timestamps, steps, and a conclusion, with no utilization field, which is why an exhausted runner usually surfaces as a step that died with no stack trace. A process killed by the Linux out-of-memory killer dies on SIGKILL, and the shell reports that as exit code 137 (signal(7)).

Detail

What the built-in alert covers

The Recommendations view is a filter over the fleet rather than a pager. It applies the fixed thresholds above, flags the jobs that cross them, and names the current label next to a recommended label, leaving the change to you. To turn that into a notification, poll the same analysis from the API. It is served at https://api.warpbuild.com/api/v1 and authenticated with an API key carrying the ci scope as a bearer token.

curl -sS -H "Authorization: Bearer $WARPBUILD_API_KEY" \
  "https://api.warpbuild.com/api/v1/org_metrics/job_runner_recommendations?per_page=100"

Each recommendation carries type, current_label, recommended_label, already_at_max_size, and a resources array naming the metric that triggered it, so filtering on "memory" gives you exactly the jobs this question is about. A weekly workflow that fails when that list is non-empty is enough of an alert for most teams.

name: runner-memory-watch
on:
  schedule:
    - cron: "0 13 * * 1"
  workflow_dispatch:

jobs:
  check-memory-recommendations:
    runs-on: warp-ubuntu-latest-x64-2x
    steps:
      - name: Pull runner recommendations
        env:
          WARPBUILD_API_KEY: ${{ secrets.WARPBUILD_API_KEY }}
        run: |
          curl -sS -H "Authorization: Bearer $WARPBUILD_API_KEY" \
            "https://api.warpbuild.com/api/v1/org_metrics/job_runner_recommendations?per_page=100" \
            > recommendations.json

      - name: Report memory bound jobs
        run: |
          jq -r '.recommendations[]
            | select(.recommendation.type == "upgrade")
            | select(.recommendation.resources | index("memory"))
            | "\(.repository) \(.workflow_name) \(.job_name): \(.recommendation.current_label) -> \(.recommendation.recommended_label)"' \
            recommendations.json | tee memory-bound.txt
          test ! -s memory-bound.txt

That watch job runs on warp-ubuntu-latest-x64-2x at $0.004 per minute from the cloud runners catalog, so a weekly run that finishes inside a billed minute costs under $0.02 a month. Swap the final test for a Slack step if you would rather be told than blocked.

Read the percentile before you pick a number

The threshold that matters for your own alert comes out of history, not out of a guess. Pull the Jobs report for the last seven days and read memory_p75 and memory_p90 per repository, workflow, and job name, then compare the pair. The shapes below are illustrative rather than measurements, and they cover the four readings you will actually meet.

Row one is the alert case: the machine is short of memory on most runs, and the next failure is a matter of when the dependency graph grows. Row two is the false positive that wastes an afternoon if you alert on p90 alone, because one linker or one bundler pass touching 92 percent while the rest of the job sits near 31 percent is a step problem. Set the alert on p75 at 80 percent, keep p90 as a warning, and re-read both after any change.

What moving up a size costs

Memory and vCPU scale together in this catalog, so the fix for row one is the next label up and the price of the fix is exact. Take that job at 600 runs a month with a 12.0 minute duration on warp-ubuntu-latest-x64-8x, which is 8 vCPU and 32 GB at $0.016 per minute: 7,200 minutes, or $115.20 a month. warp-ubuntu-latest-x64-16x carries 16 vCPU and 64 GB at $0.032 per minute. If the extra memory removes the swapping and the duration falls to 9.0 minutes, that is 5,400 minutes at $0.032, or $172.80, an increase of $57.60 a month for a job that stops failing. If the duration does not move at all, the same 7,200 minutes cost $230.40, and the upgrade is buying reliability rather than time.

Compare that rate against the nearest GitHub-hosted shape before deciding it is expensive. warp-ubuntu-latest-x64-16x at $0.032 per minute sits against the 16-core Linux larger runner of the same 16 vCPU, 64 GB shape at $0.042 per minute: 24 percent lower list price (GitHub Actions per-minute rates, checked on 2026-08-13). Every rate on this page comes from the pricing page and the cloud runners catalog, checked on the same date.

One check before you spend anything. A runtime that caps its own heap will die at the same point on a 64 GB machine, so confirm the ceiling is the machine rather than a flag. Node processes stop at --max-old-space-size, and JVM builds stop at the -Xmx value in org.gradle.jvmargs or MAVEN_OPTS. The out of memory guide separates those two cases, and the right-sizing guide covers the full ladder in both directions.

Coverage and what the alerting costs

The same agent collects utilization on every runner WarpBuild operates, including BYOC instances in your own cloud account, as long as port 33931 is open on egress. The memory alert uses that observability data. The observability solution page covers the reports and the API in full, and alerting on GitHub Actions regressions covers the duration and failure-rate signals that sit next to this one.

Observability, the Recommendations view, and the reports API carry no separate charge.

Related Questions

What memory threshold fires the alert on a WarpBuild runner?

Max memory utilization at or above 80 percent, flagged as High Memory Usage in the Recommendations view. The same 80 percent rule covers max sustained CPU, max filesystem utilization, and max disk I/O against supported throughput, and all four are listed in the observability documentation. Jobs shorter than about one minute report no metrics at all, so they never trigger any of the four.

How do I set my own memory alert threshold for a GitHub Actions job?

Read a week of runs first. Pull memory_p75 and memory_p90 per job from the Jobs report, alert when p75 crosses 80 percent, and treat a high p90 next to a low p75 as one heavy step rather than an undersized machine. The observability solution page shows the report fields and the API call that returns them, and alerting on GitHub Actions regressions covers wiring the result into a schedule.

What does moving the job up one runner size cost?

Memory and vCPU scale together, so the next label doubles both the RAM and the rate. warp-ubuntu-latest-x64-8x is 8 vCPU and 32 GB at $0.016 per minute and warp-ubuntu-latest-x64-16x is 16 vCPU and 64 GB at $0.032 per minute, both from the cloud runners catalog and the pricing page. At 600 runs a month of 12.0 minutes, that step adds $115.20 if the duration holds flat and $57.60 if the duration falls to 9.0 minutes. The right-sizing guide works through the break-even in both directions.