quickstart →

v3.5.0

View on GitHub Upgrade guide

Spotlight

  • Linux jobs now run under /home/runner/work, like GitHub-hosted runners, so artifacts with absolute paths (Vitest blob reports, coverage) work across both. /home/runner/_work still points there. Caches keyed on the absolute workspace path miss once after the upgrade. Fixes runs-on#554.
  • Every job's traces and logs are captured on the runner, even without extras=otel. Replay them into any OTLP backend with roc otel replay JOB_URL. Jobs also get TRACEPARENT and a local OTLP endpoint, so instrumented tools join the job's trace with no setup; your own OTEL_* env still wins. This new local listener (127.0.0.1:4318) may conflict with an OpenTelemetry setup in your custom AMIs, so review it before upgrading.
  • Magic Cache is faster: small entries take one S3 request instead of three, the first request waits about 2 s for credentials instead of 6, and saves no longer stall in log-heavy builds. "Complete runner" shows the job's cache traffic, and sccache's GitHub cache mode (SCCACHE_GHA_ENABLED=true) now saves instead of running read-only.
  • Restore-only sticky caches: set sticky_save: false in runs-on/action (for now runs-on/action@dist/main) to use a sticky disk without saving it, for example to save caches only from main.

CloudFormation

  • New GithubEnterpriseUrl parameter for GHE.com and GitHub Enterprise Server. It cannot be combined with EnableWAF=true.
  • Control-plane containers run with a read-only root filesystem (Security Hub ECS.5). Fixes runs-on#550.
  • Installs no longer roll back when GitHub rate-limits the runner release lookup.
  • The Flex cost-allocation-tag alert works again (missing IAM permission).

Terraform

  • Container Insights is now off by default on the control-plane cluster (saves about $9/month per stack). Set container_insights_enabled = true to keep it. Thanks @DeeRakki.
  • Fleet gets a daily AWS budget of 10 USD by default (app_budget_daily_usd, as on Flex). Set 0 to disable it. The deploying role needs AWS Budgets permissions, and the cost allocation tag must be active in AWS Billing.
  • New messaging_kms_key_arn input encrypts the queues and alerts topic with your own KMS key. Its key policy must allow EventBridge, and AWS Budgets when a budget is set.
  • Opt-in Spot spread across Availability Zones (beta): RUNS_ON_SPOT_FORCE_ZONE_SPREAD = "true" in extra_env_vars. Expect more interruptions unless the runner allows several instance types. Addresses runs-on#561.
  • Upgrades from v3.0.x and v3.1.x no longer delete and recreate renamed resources (custom runner policy, alerts topic policy, ECR repository).
  • Same read-only root filesystem, install and cost-alert fixes as CloudFormation.

Other fixes

  • Flex and Fleet are more resilient to bursts, restarts and Spot shortages:
    • A job whose runner was taken by a sibling job gets a new runner within seconds instead of occasionally minutes.
    • Flex relaunches within seconds when GitHub removes a runner before assigning it a job.
    • Flex finds jobs with a lost queued webhook within minutes in busy repositories, instead of up to hours in large organizations.
    • An exhausted Spot quota or a Spot capacity miss now sends only the affected runners to on-demand, not every job for 5 minutes.
    • Fleet reruns Spot-interrupted runs within minutes of them finishing, instead of up to 30 minutes.
    • Fleet restarts no longer lose in-flight launches, and the control plane retries network errors at startup instead of crashing.
  • Faster jobs on local NVMe: the disks are no longer zeroed in the background, and on Ubuntu 26.04 Docker images now live on NVMe (or in memory with extras=tmpfs). Images baked into a custom Docker 29 AMI are pulled again when needed.
  • extras=tmpfs runners of any size get zram swap, and their tmpfs can grow to 150% of RAM.
  • The Flex setup status page checks webhooks, the control plane, the event queue and your vCPU quotas, and links to logs and roc commands.
  • "runner spec not found" errors list the config files RunsOn read, and config edits apply on the next job.
  • Jobs without S3 access no longer spend 60 s in "Complete runner".
  • Built-in image AMI IDs are published to SSM at /<stack-name>/images/<image-id>, for EC2 Image Builder.
  • New extras=diagnostics ships the GitHub runner's _diag logs, to debug slow runner sessions.
  • OpenTelemetry:
    • New runs_on_runner_vcpus and runs_on_jobs_in_progress gauges; runners export memory in bytes and a job_key attribute shared by matrix jobs.
    • The SigNoz dashboard is split into Operations, Runners and Troubleshooting.
    • Delta temporality now works, and filesystem usage covers the instance store and Docker data roots. Fixes runs-on#559.
    • Percent-encoded OTLP headers (Authorization=Basic%20…, as vendors document them) are now decoded.
  • Security updates for Go dependencies.

Release resources