Spot instances and interruptions

What are spot instances?#

Spot instances are a purchasing option that allows you to take advantage of unused EC2 capacity in the AWS cloud.

This can result in significant cost savings, often up to 90% off the on-demand price.

However, spot instances are not guaranteed to be available at all times and can be terminated by AWS if the capacity is needed for on-demand instances.

Usage#

jobs:
  ci:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64
  deploy:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/spot=false

AWS_PROFILE=runs-on-admin roc interrupt "$JOB_URL" --wait

Default behavior#

RunsOn always tries Spot first and falls back to on-demand if Spot capacity is unavailable. On Flex, the fallback adds a 2–3s launch delay. Fleet retries the affected runners on-demand about 5 seconds later. Starting with v3.5.0, only those runners fall back, as on Flex: the next runners of that runner fleet ask for Spot again. In v3.4.x, the whole runner fleet launched on-demand for the next 5 minutes.

Spot interruption is also rare for short jobs: AWS does not bill any Spot instance interrupted within its first hour, and the request is rejected up front if capacity is critical. That makes Spot a safe default for almost all CI work under one hour without critical side effects. See AWS billing for interrupted Spot instances for the exact rules.

Disable Spot when:

• the workflow cannot be interrupted (deployment, migration, release signing, long-running jobs with poor checkpointing);
• you pin a specific instance type that often hits capacity ceilings and want to skip the Spot retry delay.

Spot allocation strategies#

Supported allocation strategies on RunsOn include:

• spot=price-capacity-optimized or spot=pco: This strategy balances between price and capacity to optimize cost while minimizing the risk of interruption.
• spot=lowest-price or spot=lp: This strategy focuses on obtaining the lowest possible price, which may increase the risk of interruption.
• spot=capacity-optimized or spot=co: This strategy prioritizes the allocation of instances from the pools with the most available capacity, reducing the likelihood of interruption.
• spot=capacity-optimized-prioritized or spot=cop (v3.3.0+): Uses exact instance types in family order as a best-effort preference, while optimizing for capacity first.

To prefer AMD over Intel, use exact types such as family=c8a.large+c8i.large with spot=cop. Family names such as family=c8a+c8i and wildcards are not supported with this strategy. AWS may still choose Intel when its capacity assessment outweighs your preference; neither a strict fallback order nor a particular distribution is guaranteed.

For on-demand (spot=false), RunsOn automatically prioritizes exact types in list order. Starting with v3.3.0, Flex also applies that order when it falls back after an empty Spot response. The default Spot strategy remains price-capacity-optimized, which does not honor list order. See Instance type priority for Flex and Fleet configuration examples.

For more details on each strategy, refer to the official AWS allocation strategy documentation ↗. For guidance on which strategy to pick when trading cost against stability, see Cost control.

Spread across Availability Zones (beta)#

AWS Spot allocation strategies choose, across every subnet of the stack, the pool they expect to interrupt least. One zone often holds that pool for long stretches, so most Spot runners can land in a single Availability Zone. RunsOn leaves that choice to AWS by default, because it is what keeps interruptions low.

If spreading runners across zones matters more to you, for example to limit what one zone’s outage or NAT gateway affects, Terraform stacks can opt in starting with v3.5.0. Set this environment variable on the control plane with extra_env_vars (Flex, Fleet):

extra_env_vars = {
  RUNS_ON_SPOT_FORCE_ZONE_SPREAD = "true"
}

Each Spot request then asks for capacity in one of the stack’s subnets, and the allocation strategy still picks the instance type within that zone:

• Flex picks the subnet from the job ID. If that zone has no Spot capacity, Flex retries Spot in every subnet, then falls back to on-demand.
• Fleet rotates the subnet from one launch request to the next. Runners launched in the same request share a zone, so the spread comes from successive requests. If that zone has no Spot capacity, Fleet retries those runners about 5 seconds later on Spot in every subnet, and that runner fleet offers every subnet for the next 5 minutes.

Before you enable it:

• Runners go to zones AWS did not rank first, so Spot interruptions can become more frequent. The option suits runners that allow several instance types, where every zone has a good pool.
• Spot prices differ between zones, so the average price per runner can go up or down.
• It is in beta: the variable name and the behavior may change. CloudFormation stacks cannot set it yet.

Runners with a single subnet to launch in are unaffected.

Automatic Spot protection and recovery#

On Flex, RunsOn will automatically force spot to false in the following conditions:

• when RunsOn provisions new capacity for an automatic Spot recovery attempt;
• whenever a Spot instance quota is exhausted — for the next 5 minutes, Flex launches on-demand the runners whose instance families all fall under that quota, and emits an email alert so you can request an EC2 Spot quota increase. AWS sets Spot quotas per group of families (Standard A, C, D, H, I, M, R, T, Z; G and VT; P; and so on), so since v3.5.0 runners from other groups keep using Spot. Before v3.5.0, every runner fell back to on-demand;
• if the spot circuit breaker (available since v2.6.7) is active, in which case Flex falls back to on-demand for the recovery duration configured on the circuit breaker.

The spot circuit breaker trips to on-demand once spot interruptions exceed a threshold within a time window. Its value uses the format COUNT/WINDOW_MINUTES/RECOVERY_MINUTES:

• COUNT: number of interruptions before the breaker trips;
• WINDOW_MINUTES: time window, in minutes, over which interruptions are counted;
• RECOVERY_MINUTES: time, in minutes, before spot is tried again.

Raising COUNT makes the breaker less sensitive, tolerating more interruptions before switching to on-demand.

Fleet uses the same circuit-breaker format and default: spot_circuit_breaker = "2/15/30"; set it to false to disable it. It tracks interruption notices across Fleet targets and sends new launches to on-demand capacity for the recovery period after the threshold is reached.

Separately, when Spot has no capacity for some Fleet runners, those runners launch on-demand. This does not count toward the circuit breaker, and still applies when the circuit breaker is set to false. Starting with v3.5.0, the rest of the runner fleet keeps asking for Spot; in v3.4.x, the runner fleet launched on-demand for the next 5 minutes.

Fleet also recovers attempts 1 and 2 when the instance that ran a job was interrupted and the workflow run finishes with the failure conclusion. It starts GitHub’s rerun failed jobs flow after snoozing Spot launches for that Fleet target. Fleet checks the run every 2 minutes during the first hour after the interrupted job finished, so a run that fails in that hour is rerun within about 2 minutes. After that hour, the checks slow down gradually to every 30 minutes. Attempt 3 is terminal. This requires Actions read and write on the Fleet GitHub credential. For a GitHub App, grant Actions Read and write; an enterprise PAT also needs repo in addition to manage_runners:enterprise.

Recovery capacity is best effort. Flex and Fleet launch new recovery capacity as on-demand, but GitHub may assign the job to an existing matching Spot runner.

Upgrading Fleet from v3.2 to v3.3, or rolling back across that boundary, can skip an automatic rerun that was already pending. Rerun that workflow manually if this happens.

Set retry: false on a Fleet runner to opt that runner out of creating recovery markers and initiating the rerun. It does not prevent a person or another system from rerunning the workflow, and it does not disable interruption tracking for the circuit breaker. Use a separate on-demand runner fleet when a workload must never start on Spot.

Merge queues#

A Spot interruption fails the affected job so GitHub includes it in the automatic retry. RunsOn also adds an error annotation whose title is exactly EC2 Spot interruption. The runs-on/spot-retry-gate reusable workflow uses that annotation to keep a transient interruption from ejecting a pull request from a merge queue. The workflow supports GitHub.com; GitHub Enterprise Server does not support its $/ self-repository action reference.

Require only merge-gate / pass in the branch ruleset. Do not make the individual Spot jobs required checks. A retryable interrupted attempt reports a successful merge-gate / interrupted check instead, so the required check remains pending while RunsOn starts the retry.

Enable automatic Spot retry for every RunsOn job in the workflow, and ensure RunsOn can request workflow reruns. If RunsOn cannot start the retry, the required merge-gate / pass check remains pending until the ruleset timeout.

name: CI

on:
  pull_request:
  merge_group:

permissions:
  contents: read

jobs:
  test:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64
    steps:
      - uses: actions/checkout@v7
      - run: make test

  merge-gate:
    if: ${{ always() }}
    needs: [test]
    permissions:
      actions: read
      checks: read
    uses: runs-on/spot-retry-gate/.github/workflows/merge-queue.yml@v1
    with:
      job_results: ${{ toJSON(needs) }}

List every job that contributes to the aggregate result in needs, and use one non-matrix gate job. When all dependencies succeed, the gate publishes the normal check without querying GitHub. Attempts 1 and 2 publish interrupted for a confirmed Spot interruption. Attempt 3 and later publish the normal aggregate result, so the required check cannot remain pending after RunsOn reaches its two-rerun limit.

Keep the workflow running until the gate finishes. Do not cancel it: RunsOn waits for the failed workflow attempt to complete before requesting the retry. Set the merge queue’s status-check timeout long enough to cover that delay and the retry. The gate fails closed if an annotation lookup required for a failed dependency cannot complete. GitHub carries unrelated successful jobs into the new attempt without executing them again. Failed jobs and their dependent jobs run again, including the successful gate job.

Set the optional check_name input to change the normal pass suffix. The names interrupted and Detect Spot interruption are reserved.

The gate recognizes only interruptions carrying the exact annotation. If the runner is terminated before its post-job hook reports it, the gate fails the normal check even when RunsOn has enough durable instance evidence to retry the job. That evidence is not available to a workflow with GitHub-only permissions.

Flex vs Fleet#

Same Spot-first default; the difference is where the override lives.