self-host →

OpenTelemetry

Per-job runner metrics in the GitHub Actions UI, plus the exact OTLP signal behavior RunsOn exports server- and runner-side.

This page covers RunsOn metrics end to end: the per-job runner metrics you get in the GitHub Actions UI, and the authoritative OpenTelemetry reference for Flex and Fleet — what each product emits, when, and where the automatic behavior stops.

At a glance#

RunsOn surfaces per-job resource metrics (CPU, memory, disk, network, I/O) through three paths, from zero-config inline charts to full OTLP export. Server-side control-plane signals go out over OTLP only.

PathWhere it appearsSetupBest for
Built-in inline chartsComplete runner step + metrics.jsonl in S3None — on by defaultDefault rightsizing and single-job troubleshooting
runs-on/action@v2 CloudWatchPost Run runs-on/action@v2 stepAdd the action with metrics:AWS-native per-job metrics — see CloudWatch
Flex runner OTLP exportYour OTLP backendextras=otel + Flex OTLP endpointLong-term runner-host observability

These are not mutually exclusive — a job can render inline charts and export to OTLP at the same time.

Per-job runner metrics#

Use these to rightsize runners and spot job-level constraints without leaving the GitHub Actions UI.

Built-in inline charts#

This is the default path on current RunsOn runners:

  1. The runner collects local host metrics during the job.
  2. RunsOn writes those samples into metrics.jsonl on the runner.
  3. The Complete runner step renders ASCII charts from that file.
  4. The same step uploads metrics.jsonl into your RunsOn S3 bucket for later inspection.

This flow is separate from remote OTLP export — you get the Complete runner charts and the uploaded metrics.jsonl file without enabling extras=otel.

Example excerpt from a real Complete runner step

Runner metadata#

For each runner you also get detailed metadata about the EC2 instance, RunsOn installation, and runner timings — expand the “Set up job” section in the GitHub Actions UI:

Runner metadata right from the GitHub Actions UI
Runner metadata right from the GitHub Actions UI

Control-plane behavior#

When an OTLP endpoint is configured, Flex and Fleet export OTLP logs, metrics, and traces from their control planes. Flex uses service.name=runs-on-flex; Fleet uses service.name=runs-on-fleet. Both emit the unified RunsOn server-metrics catalog below, structured operator logs, and Go runtime metrics.

Current v3 installs do not expose a Prometheus /metrics endpoint. Use the product’s OTLP endpoint setting for server-side metrics: OtelExporterEndpoint for Flex CloudFormation, or otel_exporter_endpoint for Flex and Fleet Terraform.

Runner-side behavior#

Runner-side OTEL behavior is available on Flex starting with RunsOn v2.12.0.

When the stack has an OTLP endpoint configured and a job opts in with extras=otel, the runner exports its host metrics (CPU load and utilization, memory, disk I/O, network I/O) and Go-runtime metrics to your observability backend, alongside the per-step job spans. Add extras=otel to a job label, or to the runner spec used by a pool:

runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel
  • The collector always writes local metrics.jsonl for the built-in inline charts, independent of OTLP — enabling one does not require the other.
  • Remote OTLP export only happens when the runner has OTEL enabled and the stack OTLP endpoint is configured.
  • RunsOn exports the same core runner host-metric set on Linux and Windows, with platform-specific device and mount filtering.
  • Fleet does not expose extras=otel. Its OTLP configuration observes fleetd and Fleet job lifecycle signals rather than individual runner-host metrics.

Datadog resource attributes as tags#

RunsOn adds runner identity and context as OTLP resource attributes, including service.instance.id, stack_name, region, org, instance_type, and repo_full_name. When exporting directly to Datadog’s OTLP endpoint, ask Datadog to promote those resource attributes into metric tags:

Terraform
otel_exporter_temporality = "delta"
otel_exporter_headers = "dd-api-key=<dd-key>,dd-otel-metric-config=${jsonencode({ resource_attributes_as_tags = true })}"
CloudFormation parameter
OtelExporterTemporality: delta
OtelExporterHeaders: dd-api-key=<dd-key>,dd-otel-metric-config={"resource_attributes_as_tags":true}

Without resource_attributes_as_tags, Datadog can receive the metrics while hiding useful runner context such as the EC2 instance id from metric tag filters.

Logs#

When the stack OTLP endpoint is configured, RunsOn exports control-plane logs from Flex or Fleet. Terraform enables log export by default through otel_logs_enabled; logs also remain available in CloudWatch through the usual AWS log groups.

RunsOn forwards the runner bootstrap log file (output.log) through the local collector when runner OTEL is enabled.

This is the RunsOn bootstrap and agent log path. It is not the full GitHub workflow job log stream.

If the EC2 instance log group is configured, the same bootstrap log file also remains available through the instance log group path in CloudWatch.

Traces#

RunsOn emits control-plane traces plus agent and instance lifecycle traces. Terraform enables control-plane trace export by default through otel_traces_enabled.

Flex also emits per-step job spans automatically after a job completes. Those spans are emitted from the Flex control plane, and you don’t need to set extras=otel on jobs to get them.

Automatically emitted per-step job spans

v3.2 operator telemetry#

v3.2 gives Flex and Fleet one operator metric and structured-event contract. Configure the control-plane exporter on either Terraform module with the same inputs:

otel_exporter_endpoint = "https://your-otlp-endpoint"
otel_exporter_headers = "authorization=<token>"
otel_exporter_temporality = "cumulative"
otel_logs_enabled = true
otel_traces_enabled = true

Use the authentication header format required by your backend, and keep it in your normal secret-management path. For a Flex CloudFormation stack, use OtelExporterEndpoint, OtelExporterHeaders, and OtelExporterTemporality instead.

Update existing dashboards and alerts#

Update queries that use the previous Flex-only contract:

Previous queryv3.2 query or behavior
service.name = runs-on-serverFlex is runs-on-flex; Fleet is runs-on-fleet. Include both values in shared dashboards.
runs_on_pool_instances_totalruns_on_pool_instances
runs_on_provisioning_registration_retries_totalruns_on_provisioning_retries_total, filtered by provisioning_stage
runs_on_launch_failures_totalruns_on_provisioning_failures_total, filtered by provisioning_stage, error_code, and instance_lifecycle
runs_on_jobs_queue_fetched or runs_on_scaling_boost_activeRemoved. Use the current queue, runner, and provisioning metrics below.
metric_type=snapshotmetric_type=operator_snapshot
metric_type=job_eventRemoved. Use job_launched and job_summary.
CloudWatch RunsOn/minutes metricsRemoved. Use runs_on_jobs_total over OTLP, or count job_launched and job_summary structured logs.

The first Fleet maintenance pass after rollout can replay one in-flight lifecycle event from an older claim. Structured events have deterministic event_id values for deduplication; OTLP job counters can show a one-time bump.

Cost estimates and structured events#

runs_on_job_estimated_cost_usd is an operational estimate, not an AWS bill. It includes EC2 compute plus live root and sticky EBS usage. Retained sticky snapshots are exported separately as runs_on_sticky_snapshot_estimated_cost_usd_per_hour; that value is an hourly upper bound based on full volume size, while AWS bills only changed EBS blocks.

If pricing or durable launch inputs are incomplete, RunsOn omits a cost estimate rather than recording zero. Use runs_on_job_cost_estimate_coverage and runs_on_pool_cost_estimate_coverage before treating cost charts as representative.

For a ready-made Flex and Fleet view, import the SigNoz operator dashboard. It uses job_launched, job_summary, operator_snapshot, and spot_interruption alongside the OTLP metrics.

Reference: metric inventory#

RunsOn server metrics#

MetricKindUnitDescriptionAttributesTransport
runs_on_jobs_totalcounter{job}Total number of jobs by status.conclusion
instance_family
instance_lifecycle
instance_type
interrupted
org
pool_name
pool_type
product
repo
repo_full_name
status
workflow_name
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_runner_instancesobservable_gauge{instance}Current active runner instances by control plane, lifecycle, and state.fleet_name
instance_lifecycle
pool_name
product
state
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_rate_limiter_waits_totalcounter{wait}Rate-limiter acquisitions that blocked waiting for capacity or failed; zero-wait fast-path acquisitions are not counted.limiter
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_rate_limiter_wait_duration_secondshistogramsTime spent waiting for rate-limiter capacity when acquisition blocked or failed.limiter
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_internal_queue_duration_secondshistogramsTime from the recorded queue reference to runner scheduling. For capacity-backed Fleet claims, fleet_demand_observed is the claim-materialization time, not GitHub’s exact job queue timestamp.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
queue_time_source
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_overall_queue_duration_secondshistogramsTime from the recorded queue reference to job start. queue_time_source distinguishes exact GitHub timestamps from Fleet claim-materialization fallback.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
queue_time_source
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_duration_secondshistogramsTime from job started to completed.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_end_to_end_duration_secondshistogramsTime from the recorded queue reference to job completion. queue_time_source distinguishes exact GitHub timestamps from Fleet claim-materialization fallback.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
queue_time_source
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_launch_attempts_totalcounter{attempt}Total EC2 launch attempts attributable to workflow jobs.conclusion
instance_family
instance_lifecycle
instance_type
interrupted
org
pool_name
pool_type
product
repo
repo_full_name
status
workflow_name
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_estimated_cost_usdhistogram{USD}Estimated EC2 and live EBS cost attributable to a completed job.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_estimated_ec2_cost_usdhistogram{USD}Estimated EC2 compute cost attributable to a completed job.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_estimated_ebs_cost_usdhistogram{USD}Estimated root and sticky EBS volume cost attributable to a completed job.conclusion
instance_family
instance_lifecycle
interrupted
org
pool_name
product
repo
status
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_sticky_snapshot_estimated_cost_usd_per_hourobservable_gauge{USD}/hEstimated retained sticky snapshot storage cost per hour. Upper bound: sized from the full volume size, while EBS bills only changed blocks.org
repo
workflow_path
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_job_cost_estimate_coverageobservable_gauge1Fraction of completed jobs with a complete EC2 and EBS estimate since process start.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_pool_estimated_cost_usd_per_hourobservable_gauge{USD}/hEstimated current EC2 and root EBS hourly burn rate for pool-owned capacity. State groups are mutually exclusive; completed-job and retained-snapshot costs are separate.component
fleet_name
pool_name
state
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_pool_cost_estimate_coverageobservable_gauge1Fraction of current pool-owned resources with complete pricing inputs for each state and component.component
fleet_name
pool_name
state
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_pool_instancesobservable_gauge{instance}Current number of pool instances by state.installation_id
org
pool_name
state
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_github_operation_duration_secondshistogramsGitHub API operation duration per attempt; server-error retries record as separate samples.github_app_label
github_operation
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_rate_limiter_tokensobservable_gauge{token}Available tokens in a rate limiter.limiterExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_rate_limiter_burstobservable_gauge{token}Burst capacity of a rate limiter.limiterExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_aws_operation_duration_secondshistogramsAWS API operation duration.aws_operation
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_spot_circuit_breaker_activeobservable_gauge1Whether the spot circuit breaker is active; 1 means active and 0 means inactive.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_github_operations_totalcounter{operation}Total number of GitHub API call attempts by operation name; server-error retries count separately.github_app_label
github_operation
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_aws_operations_totalcounter{operation}Total number of AWS API operations by operation name.aws_operation
outcome
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_registration_jit_duration_secondshistogramsTime spent generating a GitHub JIT runner configuration.github_app_labelExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_reconciler_backlogobservable_gauge{job}Current number of jobs waiting to be reconciled.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_jobs_awaiting_launchobservable_gauge{job}Current jobs awaiting an EC2 launch by bounded scheduling reason.reasonExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_provisioning_backlogobservable_gauge{job}Current number of provisioning jobs waiting in the launch or registration queue.provisioning_stageExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_provisioning_inflightobservable_gauge{job}Current number of provisioning jobs actively being processed in the launch or registration stage.provisioning_stageExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_provisioning_stage_duration_secondshistogramsWall-clock time spent in a provisioning stage worker pass.provisioning_stageExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_provisioning_retries_totalcounter{retry}Total number of retryable provisioning attempts by stage.provisioning_stageExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_provisioning_failures_totalcounter{failure}Total number of provisioning failures by stage and bounded error code.error_code
instance_lifecycle
provisioning_stage
Exported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_spot_interruptions_totalcounter{interruption}Total number of EC2 spot interruption notices observed for owned runners.instance_familyExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_fleet_desired_runnersobservable_gauge{runner}Current desired runner count calculated from GitHub assigned Fleet demand.fleet_nameExported through OTLP metrics when the stack OTLP endpoint is configured.
runs_on_fleet_claimsobservable_gauge{claim}Current Fleet claim counts by bounded lifecycle state.fleet_name
state
Exported through OTLP metrics when the stack OTLP endpoint is configured.

Runner host metrics exported to OTLP by default#

Notes:

  • Remote OTLP export requires extras=otel, a configured OTLP endpoint, and JobEnabled=true.
  • Applies to Linux and Windows.
  • On Linux, disk I/O covers detected whole devices, including additional instance-store disks; filesystem utilization covers the root, sticky-disk, and runner tmpfs mount points.
  • On Windows, disk I/O covers fixed drive-letter volumes and filesystem utilization covers the root volume. Directory-mounted sticky disks are not exposed by the Windows scrapers.
  • Network metrics are limited to the detected primary network interface.
MetricKindUnitDescriptionAttributes
system.cpu.load_average.15mgauge{thread}Average CPU Load over 15 minutes.-
system.cpu.load_average.1mgauge{thread}Average CPU Load over 1 minute.-
system.cpu.load_average.5mgauge{thread}Average CPU Load over 5 minutes.-
system.cpu.utilizationgauge1Difference in system.cpu.time since the last measurement per logical CPU, divided by the elapsed time (value in interval [0,1]).cpu
state
system.disk.iosumByDisk bytes transferred.device
direction
system.disk.operationssum{operations}Disk operations count.device
direction
system.filesystem.utilizationgauge1Fraction of filesystem bytes used.device
mode
mountpoint
type
system.memory.utilizationgauge1Percentage of memory bytes in use.state
system.network.iosumByThe number of bytes transmitted and received.device
direction

Go runtime appendix#

MetricKindUnitDescriptionAttributesTransport
go.config.gogcobservable_up_down_counter%Heap size target percentage configured by the user, otherwise 100.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.goroutine.countobservable_up_down_counter{goroutine}Count of live goroutines.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.memory.allocatedobservable_counterByMemory allocated to the heap by the application.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.memory.allocationsobservable_counter{allocation}Count of allocations to the heap by the application.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.memory.gc.goalobservable_up_down_counterByHeap size target for the end of the GC cycle.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.memory.limitobservable_up_down_counterByGo runtime memory limit configured by the user, if a limit exists.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.memory.usedobservable_up_down_counterByMemory used by the Go runtime.go.memory.typeExported through OTLP metrics when the stack OTLP endpoint is configured.
go.processor.limitobservable_up_down_counter{thread}The number of OS threads that can execute user-level Go code simultaneously.-Exported through OTLP metrics when the stack OTLP endpoint is configured.
go.schedule.durationhistogramsThe time goroutines have spent in the scheduler in a runnable state before actually running.-Exported through OTLP metrics when the stack OTLP endpoint is configured.

Reference: attribute inventory#

RunsOn server metric attributes#

ContextAttributeActionDescription
aws_operationAWS API operation name.
componentEstimated cost component: ec2 or ebs.
conclusionGitHub job conclusion for completed jobs; empty for other states.
error_codeBounded launch failure code.
fleet_nameFleet name when applicable.
github_app_labelGitHub app label.
github_operationGitHub API operation name.
installation_idGitHub App installation identifier, or 0 for enterprise PAT targets.
instance_familyEC2 instance family derived from the resolved runner instance type.
instance_lifecycleRunner pricing lifecycle such as spot or on-demand.
instance_typeEC2 instance type when a job has been assigned to a runner.
interruptedWhether the job was interrupted before completion.
limiterRate limiter identifier.
orgGitHub organization name.
outcomeWait outcome: success or error.
pool_namePool name when the job is served from a pool-backed runner.
pool_typePool standby type when applicable.
productRunsOn control plane: flex or fleet.
provisioning_stageProvisioning stage, either launch or registration.
queue_time_sourceQueue reference provenance: github, fleet_demand_observed, or unknown.
reasonBounded reason the job is awaiting a launch.
repoGitHub repository name without its owner.
repo_full_nameRepository full name in owner/repo form.
stateBounded runner lifecycle state.
statusJob lifecycle status: queued, launched, in_progress, or completed.
workflow_nameGitHub workflow display name.
workflow_pathCanonical GitHub workflow file path.

RunsOn server resource attributes#

ContextAttributeActionDescription
cloud.regionAWS region when configured.
deployment.environmentStack environment name when configured.
service.instance.idServer host identifier when the hostname can be resolved.
service.nameruns-on-flex for Flex and runs-on-fleet for Fleet.
service.namespaceStack name when configured.
service.versionRunsOn app version when configured.

Runner metric attributes exported to OTLP by default#

ContextAttributeActionDescription
cpucpuLogical CPU number starting at 0.
cpustateBreakdown of CPU usage by type.
diskdeviceName of the disk.
diskdirectionDirection of flow of bytes/operations (read or write).
filesystemdeviceIdentifier of the filesystem.
filesystemmodeMountpoint mode such “ro”, “rw”, etc.
filesystemmountpointMountpoint path.
filesystemtypeFilesystem type, such as, “ext4”, “tmpfs”, etc.
memorystateBreakdown of memory usage by type.
networkdeviceName of the network interface.
networkdirectionDirection of flow of bytes/operations (receive or transmit).

Runner resource attributes#

ContextAttributeActionDescription
service.nameinsertInserted as runs-on-agent when the incoming telemetry does not already define a service name.
service.namespaceinsertInserted as runs-on for runner-generated telemetry when the incoming telemetry does not already define a service namespace.
service.instance.idinsertEC2 instance identifier.
deployment.environmentupsertConfigured RunsOn environment name.
deployment.environment.nameupsertConfigured RunsOn environment name.
stack_nameupsertRunsOn stack name.
regionupsertAWS region.
orgupsertGitHub organization name.
instance_typeupsertEC2 instance type.
instance_lifecycleupsertEC2 pricing lifecycle such as spot or on-demand.
availability_zoneupsertEC2 availability zone.
ami_idupsertAMI identifier used by the runner instance.
pool_nameupsertPool name when the runner comes from a pool.
pool_typeupsertPool standby type when the runner comes from a pool.
repo_full_nameupsertRepository full name in owner/repo form.
workflow_pathupsertGitHub workflow file path.
job_nameupsertGitHub job name.

Go runtime metric attributes#

ContextAttributeActionDescription
go-runtimego.memory.typeType of Go runtime memory, currently stack or other.