Disaster recovery
What a RunsOn stack loses when an Availability Zone or a Region fails, the recovery objectives we design for, and how to fail over to another Region.
A RunsOn stack holds very little state. The source of truth for your jobs is GitHub, the runners are ephemeral, and the whole stack is one CloudFormation template or Terraform module. So the recovery strategy is rebuild from code: a stack in another Region is created in minutes and starts taking jobs as soon as its GitHub App is installed.
What a stack holds#
| Data | Where | If it is lost |
|---|---|---|
| Queued and running jobs | GitHub | Nothing is lost. The stack rediscovers queued jobs from GitHub. A job that was running on a lost runner fails and can be rerun. |
Runner configuration (.github/runs-on.yml, pools, images) | Your repositories | Nothing is lost. |
| GitHub App credentials, stack configuration | AWS Secrets Manager, in the stack’s Region | A new stack registers a new GitHub App from its setup page. |
| Job history (dashboards, cost reports) | DynamoDB, with point-in-time recovery | A new stack starts with an empty history. It is not needed to run jobs. |
| Caches, sticky disks, the ephemeral registry | S3, EBS snapshots and ECR, in the stack’s Region | The first jobs in the new Region run with cold caches. |
| Custom AMIs | EC2, per Region | Copy them to the recovery Region ahead of time, or rebuild them there. The public RunsOn images exist in every supported Region. |
Recovery objectives#
These are the objectives RunsOn is designed and tested for. They are not a service level agreement: the stack runs in your AWS account, on your AWS quotas.
| Failure | Recovery time objective | Recovery point objective | How |
|---|---|---|---|
| A Spot interruption | Minutes | No data loss | Automatic. The failed job is rerun once its workflow run finishes. |
| One Availability Zone | 15 minutes | No data loss | Automatic. The stack spans two Availability Zones: ECS restarts the control plane in the other one, and new runners launch where capacity exists. Jobs that were running in the lost zone fail and need a rerun. |
| One Region | 1 hour | No data loss for jobs and configuration. Caches and job history start empty. | Manual. Create a stack in another Region and move the GitHub App installation, as described below. |
Fail over to another Region#
- Create a stack in the recovery Region, with the same parameters and the same
Environmentas the failed one. Keep the parameters in version control, or use the Terraform module, so that this step is one command. - Open the
RunsOnEntryPointURL of the new stack and register its GitHub App. Install it on the same repositories. - In your GitHub organization settings, suspend the GitHub App of the failed stack. GitHub then delivers jobs to the new stack only.
- Rerun the jobs that failed during the outage. The new stack finds jobs that are still queued within a few minutes; rerun any that stay queued.
Workflows that pin a Region with the region label must be changed, or use the recovery Region’s name, before they run on the new stack.
To fail back, unsuspend the first app once its Region is healthy, then suspend or uninstall the recovery one and delete its stack.
What to prepare#
- Service quotas (EC2 On-Demand and Spot vCPUs) in the recovery Region. A new Region starts with default quotas.
- Custom AMIs and any private networking (VPC peering, firewall rules, static egress IPs that third parties allow-list) in the recovery Region.
- Your stack parameters and license key, stored outside the primary Region.
How we test it#
RunsOn’s own CI creates and destroys complete stacks on every change to the installation templates. We also run a Region recovery exercise at least once a year on our reference stack: create a stack in a second Region from the published template, register its GitHub App, and run a job there.
The last exercise was on 19 November 2026, from us-east-1 to us-west-2, with release v3.6.2:
| Step | Measured |
|---|---|
| Create the stack from the published template | 5 min 32 s |
| Register and install the GitHub App | Manual, a few minutes |
First job: queued to running on a new runner in us-west-2 | 22 s |
One thing the exercise showed: a job only runs on the new stack when its env label matches the stack’s Environment, which is why step 1 keeps the same value.