Upgrade to RunsOn v3.2

RunsOn v3.2 adds native sticky disks, scoped Magic Cache credentials, unified telemetry, and stronger Fleet Spot recovery. It also changes a few deployment contracts. Treat this as a planned minor-version upgrade, not a blind module bump.

Before you update#

1. Review the Terraform plan or CloudFormation change set.
2. Update custom dashboards, alerts, and log queries for the new telemetry contract.
3. Make the required input changes below before selecting the published v3.2 release.
4. Run representative cache, Docker, Linux, Windows, and Spot workflows in a non-production stack.

Use the normal upgrade procedure for the deployment itself.

Required deployment changes#

Terraform IAM policy inputs are lists#

Flex replaces both singular policy inputs. Fleet replaces the runner policy input. Wrap a single ARN in a list; use the list directly when your policy ARNs are computed.

# Flex before v3.2
app_custom_policy_arn    = aws_iam_policy.app.arn
runner_custom_policy_arn = aws_iam_policy.runner.arn

# Flex v3.2
app_custom_policy_arns    = [aws_iam_policy.app.arn]
runner_custom_policy_arns = [aws_iam_policy.runner.arn]

# Fleet v3.2
runner_custom_policy_arns = [aws_iam_policy.runner.arn]

CloudFormation keeps the existing singular AppCustomPolicy and RunnerCustomPolicy parameters.

Restore the generic Environment tag explicitly#

Terraform no longer adds the generic Environment resource tag. The environment input still controls RunsOn’s job-targeting contract. Add the tag yourself when cost allocation, IAM, or inventory rules need it:

tags = {
  Environment = var.environment
}

RunsOn still adds runs-on-stack-name for resource discovery.

Replace ECR pull-through ROOT rules#

ecr_repository_prefix = "ROOT" is rejected in v3.2 because it would give runners access to every ECR repository in the account. Create or reference a named rule instead, such as docker-hub, and pass it through ecr_pull_through_cache_rules.

The prefixed rule creates its own cache namespace. The first pulls populate it again; remove the old root-level cache repositories only after workflows use the new rule. See pull-through registry caching for the runner setup.

Choose a cost-report cadence#

For Flex Terraform, replace the former boolean enable_cost_reports value with "no", "daily", "weekly", or "monthly". Use "daily" for the previous true behavior and "no" for false. CloudFormation uses the same values through CostReportsEnabled.

CloudFormation checks#

Before updating a Flex CloudFormation stack, correct these values in the stack parameters:

• Stack name: v3.2 adds a ${stack-name}-cache-broker Lambda. Stack names longer than 51 characters cannot accommodate it. CloudFormation stacks cannot be renamed, so create a new, shorter stack and use a blue-green migration.
• OtelExporterEndpoint: leave it empty or use a value containing a non-whitespace character.
• OtelExporterHeaders: leave it empty or supply comma-separated key=value pairs with a non-empty key and value.

Always select Replace current template when following the Flex upgrade instructions.

Roll out cache and sticky-disk isolation#

Magic Cache isolation#

Magic Cache isolation is opt-in:

It gives Magic Cache clients repository-and-branch-scoped credentials and moves their objects from cache/* to scoped-cache/*. Each new scope starts cold; previous objects expire normally.

Fleet runners backed by capacity also receive one cold classic Magic Cache after the upgrade, even when isolation stays disabled. v3.2 corrects their previous empty repository prefix and repopulates the matching cache/v1/<owner>/<repo>/... namespace automatically.

Sticky-disk isolation#

Sticky-disk isolation is also opt-in:

It removes legacy EBS volume and snapshot permissions from runner roles. New sticky= disks keep working because the control plane performs their EBS operations. The legacy runs-on/snapshot@v1 action stops working.

Do not enable this setting until every workflow on the stack uses the supported sticky-disk workflow instead of runs-on/snapshot@v1.

Recommended sequence#

1. Correct the policy, tag, ECR, cost-report, and CloudFormation changes.
2. Upgrade with both isolation flags set to false.
3. Update and validate custom telemetry queries.
4. Replace all legacy snapshot-action workflows with sticky-disk workflows.
5. Enable Magic Cache isolation on a non-production stack and validate cache restores.
6. Validate sticky disks, then enable sticky-disk isolation.

Flex: require runner extras#

Flex Terraform adds mandatory_extras. Use it when every runner must enable a capability, regardless of workflow labels or repository configuration:

mandatory_extras = ["s3-cache", "otel"]

This input belongs to the Flex Terraform module. Fleet continues to declare extras in its runner catalog.

Fleet: capacity, recovery, and telemetry#

Set a sustained capacity ceiling#

Set fleets.<name>.max_runners in the Fleet catalog. It defaults to 1000; set it to 0 to drain the fleet without deleting its configuration.

max_runners is the fleet’s sustained concurrency ceiling. It is not a launch rate: max_launch_batch_size controls how quickly Fleet approaches that ceiling.

Configure Spot recovery#

Fleet’s spot_circuit_breaker uses COUNT/WINDOW_MINUTES/RECOVERY_MINUTES. An empty value uses the built-in 2/15/30 policy: after two interruptions in 15 minutes, Fleet launches On-Demand capacity for 30 minutes. Set it to "false" to disable the breaker.

When a first-attempt job fails after a Spot interruption, Fleet can rerun the failed jobs on On-Demand capacity. Existing GitHub Apps need Actions: read and write permission. Enterprise PAT deployments need the repo scope in addition to their normal runner-management scope. Set retry: false on runners.<name> to opt that runner out of automatic recovery reruns; interruption detection still informs the circuit breaker.

Export Fleet telemetry#

Fleet now accepts the same root-module OTLP settings as Flex:

• otel_exporter_endpoint
• otel_exporter_headers
• otel_exporter_temporality ("cumulative" or "delta")
• otel_logs_enabled
• otel_traces_enabled

Configure an endpoint and keep logs enabled when using the supplied operator dashboard. See OpenTelemetry for the signal contract.

Update custom telemetry queries#

The supplied dashboards use the v3.2 contract. Update custom dashboards, alerts, recording rules, and OTLP/log queries before relying on them:

runs_on_jobs_queue_fetched and runs_on_scaling_boost_active are removed. CloudWatch no longer writes the RunsOn/minutes metric; count job_launched or job_summary logs, or use runs_on_jobs_total over OTLP.

Older in-flight Fleet claims can replay one lifecycle event during the first maintenance pass. Structured logs include deterministic event_id values for deduplication; OTLP counters can show a one-time rollout bump.

Verify the rollout#

After deployment, confirm that:

• Terraform plans or CloudFormation change sets contain only expected resources.
• Magic Cache and Docker workflows restore the expected scope.
• Sticky-disk workflows no longer use runs-on/snapshot@v1 before its isolation flag is enabled.
• Fleet capacity respects max_runners, and a controlled Spot test follows the expected recovery path.
• Custom telemetry queries use the new service names, metrics, and log events.