Observability

RunsOn exposes several independent monitoring surfaces — some always on per job, some opt-in per job, some wired at the stack level. They are easy to confuse because OpenTelemetry, CloudWatch, the GitHub job log, and the roc CLI can all show similar-looking data. This page lays them out one by one, says where each one ships data, and is explicit about which ones Fleet supports today.

Usage#

Flex#

Always on. A normal RunsOn job already writes inline ASCII charts and instance metadata into the job log. No label, no action required.

jobs:
  build:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64

Per-job CloudWatch metrics. Add runs-on/action@v2 with the metric groups you want; the action configures the CloudWatch agent and renders post-step charts. CloudWatch ingestion is the more expensive backend — prefer the OTLP path below when you have an OTLP backend.

- uses: runs-on/action@v2
  with:
    metrics: cpu,memory,disk

Per-job OTLP export. When the Flex stack is configured with OtelExporterEndpoint, opt a job into runner-side OTLP with extras=otel. The runner then ships bootstrap logs, host metrics, and agent traces to your OTLP backend.

jobs:
  observe:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel

roc logs is unrelated to any of the above — it pulls instance and control-plane logs from CloudWatch for a specific job URL, for ad-hoc debugging.

AWS_PROFILE=runs-on-admin roc logs "$JOB_URL" --include=console --watch

Fleet#

Fleet supports the shared per-job surfaces: inline runner metrics, runner metadata, and runs-on/action@v2 CloudWatch metrics. In v3.2, its control plane also exports OTLP metrics, logs, and traces, and creates a Fleet CloudWatch dashboard. Use OpenTelemetry or the SigNoz guide for operator telemetry.

Fleet does not expose the Flex runner-side extras=otel label. Its OTLP settings observe fleetd and the Fleet job lifecycle, not individual runner host metrics.

Per-job surfaces (always on)#

Built-in runner metrics (metrics.jsonl)#

The local collector runs on every RunsOn runner. It samples host metrics during the job, writes them to metrics.jsonl, renders ASCII charts inside the Complete runner step, and uploads the raw file to the RunsOn S3 bucket for later inspection.

You get this without enabling anything. See OpenTelemetry for the per-job paths and example chart output.

Runner metadata in the job log#

The Set up job GitHub UI step shows EC2 instance type, lifecycle (spot vs on-demand), AMI, availability zone, pool name, and boot timings. Useful for spot-checking what RunsOn actually launched. Also always on.

Opt-in per-job surfaces#

runs-on/action@v2 CloudWatch metrics#

Add the action with the metric groups you want — cpu, network, memory, disk, io. The action configures the CloudWatch agent on the runner, then a post step queries CloudWatch and renders charts inside the Post Run runs-on/action@v2 step. See the action README ↗ for the full input list.

CloudWatch metric ingestion is billed per metric per month, which makes this the more expensive surface for high-cardinality runners. It is still useful when you don’t have an OTLP backend, or when you want the action’s other features (cost summary, sccache, etc.).

This path does not work for container-based jobs and does not write metrics.jsonl.

extras=otel runner-side OTLP export#

Add extras=otel to a job label (or to the pool runner spec) when you want the runner to ship its bootstrap logs, host metrics, and agent traces to your OTLP backend. The stack must already be configured with OtelExporterEndpoint.

This is the path forward for long-term observability — OTLP is cheaper per byte than CloudWatch and works with any OTLP-compatible backend (Grafana Cloud, SigNoz, Datadog, Honeycomb, etc.). See OpenTelemetry for the full emitted-signal inventory.

Stack-level surfaces (operator-facing)#

Control-plane OTLP#

When an OTLP endpoint is configured, Flex and Fleet export their own structured logs, metrics, and traces. The shared operator metrics cover job counters, queue duration, runner and capacity state, provisioning health, cost-estimate coverage, and Spot interruptions. Flex also emits one span per GitHub workflow step automatically; workflow YAML does not need extras=otel to get those spans.

Use the Terraform otel_exporter_*, otel_logs_enabled, and otel_traces_enabled inputs for either product. Flex CloudFormation installs use OtelExporterEndpoint, OtelExporterHeaders, and OtelExporterTemporality.

CloudWatch stack dashboards#

Both products create a CloudWatch dashboard automatically. The Flex dashboard covers webhook ingress, queues, rate limits, launches, and Spot protection. The Fleet dashboard covers job volume, queue and job duration, estimated job cost, failure rate, desired runners, claims, and recent incidents. See CloudWatch.

Cost report and budget alarm#

A scheduled cost-report email summarizes spend on the deployed stack, and an AWS Budgets daily alarm fires when usage crosses a configured threshold. Configure them with AppBudgetDailyUsd and CostReportsEnabled. See Cost control.

SNS alerts and Slack#

Flex provisions an SNS topic and an email subscription for the out-of-band situations that can’t be surfaced on the run: truly unschedulable jobs and license issues. An optional Slack webhook receives the same notifications. Configured with the EmailAddress stack parameter (required at install) and AlertTopicSlackWebhookUrl (optional). See Flex alerts.

Housekeeping watchdogs#

Background reconcilers detect idle runners (10 minute timeout), hard-limit timeouts (12 hours), retry spot interruptions, and clean up stale resources. Mostly invisible until something goes wrong. See Housekeeping.

OTEL vs CloudWatch#

When you have a choice, prefer OTLP:

• OTLP ingestion is much cheaper than CloudWatch per byte, especially for metrics with attributes.
• OTLP backends (Grafana Cloud, SigNoz, Datadog, Honeycomb, …) give you richer correlation between server metrics, runner host metrics, and per-step spans than CloudWatch alone.
• CloudWatch is still the right answer when you don’t have an OTLP backend, when you want the AWS-native dashboard with no extra wiring, or when you specifically want the runs-on/action@v2 post-step ASCII charts.

These are not mutually exclusive — extras=otel and runs-on/action@v2 can run on the same job.

Flex vs Fleet#

Fleet shares the per-job surfaces bundled with the runner agent and CLI. It now also has unified control-plane OTLP and a CloudWatch dashboard. Budgets and email cost reports remain Flex stack features; Fleet uses its Terraform and AWS cost-allocation controls instead.