v3.2.0

Spotlight

Sticky disks: native-speed persistent build caches

Jobs can now attach dedicated EBS cache volumes through the sticky= runner label. RunsOn restores the latest compatible snapshot, mounts the volume directly on the runner, cleanly unmounts it after the job, and snapshots it for the next run.

Files remain in their native format, with no archive upload or extraction step.

• Syntax: sticky=[<name>:]<size>[:<type>][:<throughput>][:<iops>]
• Examples: sticky=20gb and sticky=docker:20gb:gp3:750mbs:6000iops
• Linux mount: /mnt/runs-on/stickydisk
• Windows mount: C:\runs-on\stickydisk
• Default volume settings: gp3, 3,000 IOPS, and 400 MiB/s
• Snapshot initialization defaults to 200 MiB/s and contributes to job cost estimates.

Snapshots are isolated by repository, lineage name, Git ref, operating system, and architecture. Branches restore their latest snapshot first, then fall back to the default branch. Pull requests can restore from the default branch without writing back to it. RunsOn preserves the last clean generation when a job is interrupted or a disk cannot be unmounted cleanly.

runs-on/action@v2 maps common tool caches directly onto sticky disks. Built-in modes cover Go, Node/npm, Yarn, pnpm, Ruby/Bundler, Rust/Cargo, Python/pip, uv, Poetry, APT, Gradle, Maven, Playwright, Git checkouts, and BuildKit layers. Custom paths are also supported on Linux and Windows.

Git caching maintains bare mirrors on the sticky disk and serves supported GitHub fetches through a local smart-HTTP proxy. BuildKit caching includes the matching RunsOn ECR Docker Hub mirror configuration when available.

The action reports cache hits and disk pressure. Critically full caches reset automatically so jobs run cold instead of failing with no space left on device.

Unified cost and operational telemetry

Flex and Fleet now emit one operator telemetry contract.

New telemetry covers:

• Per-job EC2 and live-EBS cost estimates
• Fleet pool and queue costs with explicit ownership attribution
• Retained sticky-snapshot cost estimates
• Queue and end-to-end durations
• Launch attempts and provisioning failures
• Runner and Fleet capacity
• Provisioning backlog
• Spot interruptions
• Explicit histogram buckets

The supplied CloudWatch dashboard has been updated. A new SigNoz dashboard shows job volume, failure rate, queue latency, estimated spend, estimate coverage, repository and workflow cost breakdowns, Fleet pool costs, sticky-snapshot costs, and Spot interruptions.

Full diagnostic archives now include each attempted runner’s metrics.jsonl, preserving CPU, memory, disk, network, tmpfs, and host-device evidence.

Custom telemetry consumers must account for these contract changes:

• Flex service name: runs-on-server → runs-on-flex
• runs_on_pool_instances_total → runs_on_pool_instances
• runs_on_provisioning_registration_retries_total → runs_on_provisioning_retries_total, with a provisioning_stage attribute
• runs_on_launch_failures_total → runs_on_provisioning_failures_total
• Removed metrics: runs_on_jobs_queue_fetched and runs_on_scaling_boost_active
• Structured log metric_type=snapshot → metric_type=operator_snapshot
• Removed metric_type=job_event; use job_launched and job_summary
• Removed CloudWatch writes to RunsOn/minutes; use structured logs or runs_on_jobs_total over OTLP

Older in-flight Fleet claims can replay one lifecycle event during the first maintenance pass after upgrading. Structured logs contain deterministic event_id values for deduplication. OTLP counters can show a one-time rollout bump.

Fleet Spot-interruption recovery

Fleet records durable Spot-interruption evidence, feeds interruptions into the shared Spot circuit breaker, and can automatically rerun failed first-attempt jobs on On-Demand capacity.

The default policy is 2/15/30: after two interruption failures within 15 minutes, matching work moves away from Spot for a 30-minute recovery period. Fleet’s spot_circuit_breaker input uses the same COUNT/WINDOW_MINUTES/RECOVERY_MINUTES format as Flex.

Existing GitHub Apps must grant Actions: read and write for automatic reruns. Without this permission, Fleet continues normal execution but skips recovery reruns with a warning.

Fleet launch, completion, registration, and repair processing is also hardened for bursts, ambiguous EC2 launch responses, stale claims, permanent GitHub JIT failures, and failed cleanup. Lifecycle callbacks can run concurrently while preserving ordering where required.

Safe cache cleanup with roc

roc cleanup JOB_URL plans and deletes re-creatable cache data for a job’s Git ref, including classic Magic Cache objects, isolated cache objects, and matching sticky-disk snapshots.

Use --dry-run to inspect the plan, --yes to apply it, and --include-default-branch to reset the default-branch fallback too. The command validates stack ownership and uses authenticated GitHub metadata before presenting its deletion plan.

Better troubleshooting with roc logs --full

roc logs JOB_URL --full now includes the diagnostics resolver response, job and run server logs, CloudTrail events, EC2 console output, agent logs, and uploaded runner metrics for every attempted instance.

A redacted stack_settings block records non-sensitive behavior switches covering cache and sticky-disk isolation, storage, BuildKit and registry mirrors, runner limits, scheduling, networking, runtime sizing, telemetry, and integrations.

Secrets, exporter endpoints and headers, policy contents and ARNs, credentials, resource identifiers, and tag values remain excluded.