OpenTelemetry

This page covers RunsOn metrics end to end: the per-job runner metrics you get in the GitHub Actions UI, and the authoritative OpenTelemetry reference for Flex and Fleet — what each product emits, when, and where the automatic behavior stops.

At a glance#

RunsOn surfaces per-job resource metrics (CPU, memory, disk, network, I/O) through four paths, from zero-config inline charts to full OTLP export and a replayable capture. Server-side control-plane signals go out over OTLP only.

These are not mutually exclusive — a job can render inline charts and export to OTLP at the same time.

Per-job runner metrics#

Use these to rightsize runners and spot job-level constraints without leaving the GitHub Actions UI.

Built-in inline charts#

This is the default path on current RunsOn runners:

1. The runner collects local host metrics during the job.
2. RunsOn writes those samples into metrics.jsonl on the runner.
3. The Complete runner step renders ASCII charts from that file.
4. The same step uploads metrics.jsonl into your RunsOn S3 bucket for later inspection.

This flow is separate from remote OTLP export — you get the Complete runner charts and the uploaded metrics.jsonl file without enabling extras=otel.

Runner metadata#

Each job log separates static runner details from the identity of the assigned job:

Expand Set up job for the EC2 instance, RunsOn installation, and provisioning timings.

Expand Set up runner, then Job identity, for the organization, repository, workflow, and job ID.

RunsOn writes the job identity when the job starts. This keeps the values accurate when Fleet pre-provisions capacity before GitHub assigns a job.

Control-plane behavior#

When an OTLP endpoint is configured, Flex and Fleet export OTLP logs, metrics, and traces from their control planes. Flex uses service.name=runs-on-flex; Fleet uses service.name=runs-on-fleet. Both emit the unified RunsOn server-metrics catalog below, structured operator logs, and Go runtime metrics.

Current v3 installs do not expose a Prometheus /metrics endpoint. Use the product’s OTLP endpoint setting for server-side metrics: OtelExporterEndpoint for Flex CloudFormation, or otel_exporter_endpoint for Flex and Fleet Terraform.

Upgrading from a release before v3.2? The v3.2 upgrade guide maps the previous service name, metric names, and log events to the current ones.

Control-plane resource attributes#

Control-plane logs, metrics, and traces carry the server resource attributes: service.name, service.version, deployment.environment (the stack environment, which is also the environment RunsOn uses to filter jobs), service.namespace (the stack name), cloud.region, and service.instance.id.

To add attributes or replace RunsOn’s values, set otel_resource_attributes on the Flex or Fleet Terraform module. It takes the standard OTEL_RESOURCE_ATTRIBUTES format of comma-separated key=value pairs:

otel_resource_attributes = "deployment.environment=ci,team=platform"

• Your values take precedence over RunsOn’s, except service.name, which stays runs-on-flex or runs-on-fleet so that shared dashboards keep matching.
• A malformed pair is skipped and reported in the control-plane logs. It does not stop the stack.
• These attributes apply to control-plane signals only, not to runner host metrics.
• Flex CloudFormation has no equivalent parameter.

Metric temporality#

otel_exporter_temporality in Terraform, or OtelExporterTemporality in CloudFormation, sets how counters and histograms are exported. It applies to the control plane and to runners with OTEL enabled.

• cumulative (default): each export carries the running total. Use it for Prometheus-compatible backends.
• delta: each export carries the change since the previous export. Datadog requires it, and SigNoz works best with it. From v3.5.0, up-down counters, such as the Go runtime’s goroutine count and memory in use, stay cumulative so that they keep reporting a level.

SigNoz computes a cumulative counter’s increase from the difference between consecutive samples, so it never counts the first sample of a new series. RunsOn job counters are split by repository, workflow, conclusion, instance type, lifecycle, and pool, and each new control-plane task or version starts new series. With cumulative export, SigNoz counter panels therefore undercount. See SigNoz.

Runner-side behavior#

Runner-side OTEL behavior is available on Flex starting with RunsOn v2.12.0. Fleet uses the same runner exporter starting with RunsOn v3.2.0.

When the stack has an OTLP endpoint configured and runner OTEL is enabled, the runner exports its host metrics, with service.name=runs-on-agent, (CPU load and utilization, memory, disk I/O, network I/O) to your observability backend. It also exports bootstrap logs when otel_logs_enabled is true and agent and job spans when otel_traces_enabled is true.

The collector also exposes an OTLP/HTTP receiver on 127.0.0.1:4318 for programs running inside the workflow. From v3.5.0, every job gets OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, and TRACEPARENT, so OTLP clients send to it and join the job’s trace without configuration, and every runner keeps a local capture of the agent’s spans, workflow traces and logs, host metrics, and the bootstrap log, with or without extras=otel. If another process already listens on 127.0.0.1:4318 when the job starts, the job gets no endpoint or protocol default, and RunsOn adds a warning to the job that names the port. On Windows, the warning is only in the agent log. See Local OpenTelemetry collector for otel-cli and CargoWall examples.

runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel

runners = {
  linux-small = {
    # ...
    extras = ["otel"]
  }
}

• The collector always writes local metrics.jsonl for the built-in inline charts, independent of OTLP — enabling one does not require the other.
• Remote OTLP export only happens when the runner has OTEL enabled and the stack OTLP endpoint is configured. The local capture does not depend on either.
• RunsOn exports the same core runner host-metric set on Linux and Windows, with platform-specific device and mount filtering.
• Fleet workflows cannot add extras per job. Configure extras = ["otel"] on the Fleet runner definition instead.

Datadog resource attributes as tags#

RunsOn adds runner identity and context as OTLP resource attributes, including service.instance.id, stack_name, region, org, instance_type, and repo_full_name. When exporting directly to Datadog’s OTLP endpoint, ask Datadog to promote those resource attributes into metric tags:

otel_exporter_temporality = "delta"
otel_exporter_headers = "dd-api-key=<dd-key>,dd-otel-metric-config=${jsonencode({ resource_attributes_as_tags = true })}"

OtelExporterTemporality: delta
OtelExporterHeaders: dd-api-key=<dd-key>,dd-otel-metric-config={"resource_attributes_as_tags":true}

Without resource_attributes_as_tags, Datadog can receive the metrics while hiding useful runner context such as the EC2 instance id from metric tag filters.

Logs#

When the stack OTLP endpoint is configured, RunsOn exports control-plane logs from Flex or Fleet. Terraform enables log export by default through otel_logs_enabled; logs also remain available in CloudWatch through the usual AWS log groups.

RunsOn forwards the runner bootstrap log file (output.log) through the local collector when runner OTEL is enabled.

This is the RunsOn bootstrap and agent log path. It is not the full GitHub workflow job log stream.

If the EC2 instance log group is configured, the same bootstrap log file also remains available through the instance log group path in CloudWatch.

Traces#

RunsOn emits control-plane traces plus agent and instance lifecycle traces. Terraform enables control-plane trace export by default through otel_traces_enabled.

Agent and instance lifecycle spans go to the stack’s backend when the runner has OTEL enabled. From v3.5.0, they are also in every runner’s local capture, which roc otel replay sends to any backend.

Flex also emits per-step job spans automatically after a job completes. Those spans are emitted from the Flex control plane, and you don’t need to set extras=otel on jobs to get them.

Cost estimates and structured events#

runs_on_job_estimated_cost_usd is an operational estimate, not an AWS bill. It includes EC2 compute plus live root and sticky EBS usage. Retained sticky snapshots are exported separately as runs_on_sticky_snapshot_estimated_cost_usd_per_hour; that value is an hourly upper bound based on full volume size, while AWS bills only changed EBS blocks.

An instance RunsOn launched for a job but whose runner never ran it, for example a runner started just before GitHub held the job for a deployment approval, counts for at most 25 minutes. That is how long such an instance can live: its runner exits after 10 idle minutes, and RunsOn terminates any instance without a started job after 25.

If pricing or durable launch inputs are incomplete, RunsOn omits a cost estimate rather than recording zero. Use runs_on_job_cost_estimate_coverage and runs_on_pool_cost_estimate_coverage before treating cost charts as representative.

For a ready-made Flex and Fleet view, import the SigNoz operator dashboard. It uses job_launched, job_summary, operator_snapshot, and spot_interruption alongside the OTLP metrics.

Reference: metric inventory#

To tell the internal and overall queue durations apart, and to decide when to raise AppSize, see Read the queue-time metrics.

RunsOn server metrics#

Runner host metrics exported to OTLP by default#

Notes:

• Remote OTLP export requires extras=otel, a configured OTLP endpoint, and JobEnabled=true.
• Applies to Linux and Windows.
• On Linux, disk I/O covers detected whole devices, including additional instance-store disks; filesystem utilization covers the root, sticky-disk, runner tmpfs, and instance-store mount points, plus /var/lib/docker and /var/lib/containerd when a job mounts its own volume there.
• On Windows, disk I/O covers fixed drive-letter volumes and filesystem utilization covers the root volume. Directory-mounted sticky disks are not exposed by the Windows scrapers.
• Network metrics are limited to the detected primary network interface.

Go runtime appendix#

Reference: attribute inventory#

RunsOn server metric attributes#

RunsOn server resource attributes#

Runner metric attributes exported to OTLP by default#

Runner resource attributes#

Go runtime metric attributes#