quickstart →

Disaster recovery

What a RunsOn stack loses when an Availability Zone or a Region fails, the recovery objectives we design for, and how to fail over to another Region.

A RunsOn stack holds very little state. The source of truth for your jobs is GitHub, the runners are ephemeral, and the whole stack is one CloudFormation template or Terraform module. So the recovery strategy is rebuild from code: a stack in another Region is created in minutes and starts taking jobs as soon as its GitHub App is installed.

What a stack holds#

DataWhereIf it is lost
Queued and running jobsGitHubNothing is lost. The stack rediscovers queued jobs from GitHub. A job that was running on a lost runner fails and can be rerun.
Runner configuration (.github/runs-on.yml, pools, images)Your repositoriesNothing is lost.
GitHub App credentials, stack configurationAWS Secrets Manager, in the stack’s RegionA new stack registers a new GitHub App from its setup page.
Job history (dashboards, cost reports)DynamoDB, with point-in-time recoveryA new stack starts with an empty history. It is not needed to run jobs.
Caches, sticky disks, the ephemeral registryS3, EBS snapshots and ECR, in the stack’s RegionThe first jobs in the new Region run with cold caches.
Custom AMIsEC2, per RegionCopy them to the recovery Region ahead of time, or rebuild them there. The public RunsOn images exist in every supported Region.

Recovery objectives#

These are the objectives RunsOn is designed and tested for. They are not a service level agreement: the stack runs in your AWS account, on your AWS quotas.

FailureRecovery time objectiveRecovery point objectiveHow
A Spot interruptionMinutesNo data lossAutomatic. The failed job is rerun once its workflow run finishes.
One Availability Zone15 minutesNo data lossAutomatic. The stack spans two Availability Zones: ECS restarts the control plane in the other one, and new runners launch where capacity exists. Jobs that were running in the lost zone fail and need a rerun.
One Region1 hourNo data loss for jobs and configuration. Caches and job history start empty.Manual. Create a stack in another Region and move the GitHub App installation, as described below.

Fail over to another Region#

  1. Create a stack in the recovery Region, with the same parameters and the same Environment as the failed one. Keep the parameters in version control, or use the Terraform module, so that this step is one command.
  2. Open the RunsOnEntryPoint URL of the new stack and register its GitHub App. Install it on the same repositories.
  3. In your GitHub organization settings, suspend the GitHub App of the failed stack. GitHub then delivers jobs to the new stack only.
  4. Rerun the jobs that failed during the outage. The new stack finds jobs that are still queued within a few minutes; rerun any that stay queued.

Workflows that pin a Region with the region label must be changed, or use the recovery Region’s name, before they run on the new stack.

To fail back, unsuspend the first app once its Region is healthy, then suspend or uninstall the recovery one and delete its stack.

What to prepare#

  • Service quotas (EC2 On-Demand and Spot vCPUs) in the recovery Region. A new Region starts with default quotas.
  • Custom AMIs and any private networking (VPC peering, firewall rules, static egress IPs that third parties allow-list) in the recovery Region.
  • Your stack parameters and license key, stored outside the primary Region.

How we test it#

RunsOn’s own CI creates and destroys complete stacks on every change to the installation templates. We also run a Region recovery exercise at least once a year on our reference stack: create a stack in a second Region from the published template, register its GitHub App, and run a job there.

The last exercise was on 19 November 2026, from us-east-1 to us-west-2, with release v3.6.2:

StepMeasured
Create the stack from the published template5 min 32 s
Register and install the GitHub AppManual, a few minutes
First job: queued to running on a new runner in us-west-222 s

One thing the exercise showed: a job only runs on the new stack when its env label matches the stack’s Environment, which is why step 1 keeps the same value.