quickstart →

Local OpenTelemetry collector

Send workflow traces, logs, and metrics to the local OTLP endpoint on RunsOn runners. RunsOn captures them and forwards them to your backend.

Every RunsOn runner starts a local OpenTelemetry Collector. Workflow tools send OTLP/HTTP data to 127.0.0.1:4318; the collector adds runner and job context, keeps a copy in the job’s capture, and forwards it through the OTLP exporter configured on the RunsOn stack.

Every job already has the environment an OTLP client needs:

VariableValue
OTEL_EXPORTER_OTLP_ENDPOINThttp://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOLhttp/protobuf
TRACEPARENTthe job’s trace, under agent.job.runtime

A workflow configures nothing, and never sees the backend URL or ingestion header. The endpoint and protocol are defaults: the workflow’s, job’s, or step’s own env overrides them. Steps in a job container: don’t get them, because the collector listens on the host’s loopback interface.

If another process already listens on 127.0.0.1:4318 when the job starts, the job gets no endpoint or protocol default, and RunsOn adds a warning to the job that names the port. On Windows, the warning is only in the agent log.

Availability#

The local collector runs on Linux and Windows runners, on Flex and Fleet. From RunsOn v3.5.0, it accepts traces, logs, and metrics on every runner and keeps them in a local capture, and every job gets the variables above. In v3.2.1 to v3.4.0, the receiver also listens on every runner, but it keeps no capture, and workflows set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL themselves.

Three things control remote delivery:

  1. Configure an OTLP endpoint and, when required, an ingestion header on the RunsOn stack.
  2. Enable otel on the runner.
  3. Enable the pipeline for the signal you send.
Signal sent by the workflowRequired runner configuration
MetricsOTLP endpoint + otel runner extra
LogsOTLP endpoint + otel runner extra + remote logs enabled
TracesOTLP endpoint + otel runner extra + remote traces enabled

Without otel, the receiver still accepts data but sends nothing to your OTLP backend: workflow metrics go to the runner’s local metrics.jsonl, and workflow logs go to the EC2 instance log group in CloudWatch when the stack has one.

Terraform enables remote logs and traces by default through otel_logs_enabled and otel_traces_enabled. For all stack settings and the signals that RunsOn itself emits, see the OpenTelemetry reference.

Flex

Configure OtelExporterEndpoint, OtelExporterHeaders, and OtelExporterTemporality on a CloudFormation stack. Terraform uses the equivalent otel_exporter_* inputs. Then opt the job in with extras=otel:

.github/workflows/ci.yml
jobs:
instrumented:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel

You can also put otel in a Flex pool runner’s extras.

Fleet

Configure the OTLP exporter and add otel to the Terraform-owned runner:

Fleet Terraform
otel_exporter_endpoint = "https://your-otlp-endpoint"
otel_exporter_headers = "authorization=<ingestion-token>"
otel_logs_enabled = true
otel_traces_enabled = true
runners = {
linux-small = {
# ...
extras = ["otel"]
}
}

Fleet workflows select that runner normally. They cannot add extras per job.

On Fleet, the collector restarts once shortly after the job starts, to add the repository, workflow, and job attributes. Data sent in the first few seconds can miss those attributes. The RunsOn agent itself listens on 127.0.0.1:4318, so the port stays open during the restart, and each request waits up to 15 seconds for the new collector. Clients don’t need to retry. To get the attributes on a single otel-cli span, emit it after your first setup steps.

Connect a workflow tool#

OTLP/HTTP clients that read the standard OTEL_EXPORTER_OTLP_* variables, such as the OpenTelemetry SDKs and otel-cli, find the loopback receiver on their own. On RunsOn versions before v3.5.0, set them on the job:

.github/workflows/ci.yml
env:
OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf

The receiver supports OTLP over HTTP/protobuf. It does not listen for OTLP/gRPC, and it is not exposed outside the runner. It listens on the host’s loopback interface only, so a job container: or a Docker container on the default bridge network cannot reach it; use host networking for tools that run in a container.

RunsOn preserves an incoming service.name and inserts service.instance.id when it is missing. It also adds resource attributes such as the stack, AWS Region, organization, repository, workflow path, job name, instance type, and instance lifecycle. Those fields let you separate workflow telemetry from host and control-plane telemetry in the same backend.

Trace a command with otel-cli#

otel-cli ↗ can wrap a shell command in a span. It reads the job’s endpoint and TRACEPARENT, so the span joins the job’s trace with no configuration:

.github/workflows/otel-cli.yml
name: Trace a workflow command
on:
workflow_dispatch:
permissions:
contents: read
jobs:
trace:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel
env:
OTEL_CLI_FAIL: "true"
steps:
- name: Install otel-cli
run: |
mkdir -p "${RUNNER_TEMP}/bin"
curl -fsSL https://github.com/equinix-labs/otel-cli/releases/download/v0.4.5/otel-cli_0.4.5_linux_amd64.tar.gz \
| tar -xz -C "${RUNNER_TEMP}/bin" otel-cli
echo "${RUNNER_TEMP}/bin" >> "${GITHUB_PATH}"
- name: Run an instrumented command
run: |
otel-cli exec \
--service ci-workflow \
--name "test suite" \
--attrs "github.run.id=${GITHUB_RUN_ID}" \
-- bash -c 'echo "running instrumented work"'

On arm64 runners, download the linux_arm64 archive instead.

otel-cli exec sends its span before it exits, so the next step starts after the collector accepted it. OTEL_CLI_FAIL=true makes a span the collector refuses fail the step instead of passing silently. Drop extras=otel to keep the span in the job’s capture only.

You don’t need a final wait step. When the job ends, the collector sends anything still in its batch before the instance shuts down, on Flex and on Fleet, including when the Fleet control plane terminates the instance. A span accepted in the last second of a job still reaches the backend.

Export CargoWall network verdicts#

CargoWall ↗ intercepts outbound connections with eBPF. Its GitHub Action can emit each allow or deny verdict as an OTLP log record. Point it at the local collector:

.github/workflows/network-policy.yml
name: Enforce network policy
on:
workflow_dispatch:
permissions:
actions: read
contents: read
jobs:
network-policy:
runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel
steps:
- name: Enable CargoWall
uses: code-cargo/cargowall-action@7943209964cc5472fee8aa1e02cde8a2f0dd0414 # v1.3.7
env:
OTEL_EXPORTER_OTLP_COMPRESSION: gzip
OTEL_SERVICE_NAME: cargowall
OTEL_RESOURCE_ATTRIBUTES: github.run.id=${{ github.run_id }}
with:
mode: enforce
offline: "true"
fail-on-unsupported: "true"
allowed-hosts: |
example.com:443
ingest.example.com:443
- name: Make an allowed call
run: curl --fail https://example.com
- name: Verify an undeclared host is blocked
run: |
if curl --fail https://example.org; then
echo "CargoWall unexpectedly allowed example.org" >&2
exit 1
fi
echo "CargoWall intercepted example.org"
- name: Give CargoWall time to send its last batch
if: always()
run: sleep 20

CargoWall batches records in its own process before sending them to the runner collector, and it keeps running after your last step. The final wait gives it time to hand its last batch to the collector. The collector’s own batch needs no extra wait: RunsOn flushes it when the runner shuts down.

Replay a job’s telemetry#

From RunsOn v3.5.0, every runner keeps a capture of what its collector receives: the RunsOn agent’s spans, your spans and logs, host metrics, and the bootstrap log, as OTLP JSON lines. It doesn’t need a backend or the otel extra. When the instance shuts down, after the collector has written its last batch, the agent uploads the capture to the stack’s S3 cache bucket, under telemetry/<role-id>:<instance-id>/, where <role-id> is the unique ID of the runner instance role. Captured logs and spans can hold a repository’s private data, so a runner can write only its own prefix and no runner can read it. The capture expires with the bucket’s cache retention.

roc otel replay sends a job’s capture, unchanged, to any OTLP/HTTP endpoint, such as a local SigNoz or Jaeger, or your production backend:

roc otel replay https://github.com/my-org/my-repo/actions/runs/123/job/456 \
--endpoint http://localhost:4318

Spans keep their original timestamps and trace IDs, so set the backend’s time range to the job’s. The capture holds the runner side of the trace. On Flex, the control-plane spans (scheduling, instance launch, GitHub steps) go to the stack’s backend only. Each signal keeps up to about 100 MB per job. Fleet terminates the instance when the job completes, so its upload is best-effort.

Troubleshooting#

Check the receiver before debugging the backend:

curl --silent --output /dev/null --write-out '%{http_code}\n' \
--request POST \
--header 'Content-Type: application/x-protobuf' \
--data-binary '' \
http://127.0.0.1:4318/v1/traces

An HTTP 200 confirms that the local trace receiver accepted the request. It does not prove remote delivery. If data is still missing:

  • Replay the job’s capture into a local backend with roc otel replay. If the data is there, the problem is remote delivery.
  • Confirm that the runner has otel enabled and the stack has an OTLP endpoint.
  • Confirm that remote logs or traces are enabled for the signal you send.
  • Allow the remote OTLP host through any workflow-level firewall.
  • If data from the last seconds of a job is missing, look for a background process that batches telemetry on its own. It needs time to hand its last batch to the collector before the job ends.
  • Query your backend by service.name and the runner’s service.instance.id.