Local OpenTelemetry collector
Send workflow traces, logs, and metrics to the local OTLP endpoint on RunsOn runners. RunsOn captures them and forwards them to your backend.
Every RunsOn runner starts a local OpenTelemetry Collector. Workflow tools send
OTLP/HTTP data to 127.0.0.1:4318; the collector adds runner and job context,
keeps a copy in the job’s capture, and forwards it through the OTLP exporter
configured on the RunsOn stack.
Every job already has the environment an OTLP client needs:
| Variable | Value |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | http://127.0.0.1:4318 |
OTEL_EXPORTER_OTLP_PROTOCOL | http/protobuf |
TRACEPARENT | the job’s trace, under agent.job.runtime |
A workflow configures nothing, and never sees the backend URL or ingestion
header. The endpoint and protocol are defaults: the workflow’s, job’s, or
step’s own env overrides them. Steps in a job container: don’t get them,
because the collector listens on the host’s loopback interface.
If another process already listens on 127.0.0.1:4318 when the job starts,
the job gets no endpoint or protocol default, and RunsOn adds a warning to the
job that names the port. On Windows, the warning is only in the agent log.
Availability#
The local collector runs on Linux and Windows runners, on Flex and Fleet. From
RunsOn v3.5.0, it accepts traces, logs, and metrics on every runner and keeps
them in a local capture, and every job gets the
variables above. In v3.2.1 to v3.4.0, the receiver also listens on every
runner, but it keeps no capture, and workflows set
OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL themselves.
Three things control remote delivery:
- Configure an OTLP endpoint and, when required, an ingestion header on the RunsOn stack.
- Enable
otelon the runner. - Enable the pipeline for the signal you send.
| Signal sent by the workflow | Required runner configuration |
|---|---|
| Metrics | OTLP endpoint + otel runner extra |
| Logs | OTLP endpoint + otel runner extra + remote logs enabled |
| Traces | OTLP endpoint + otel runner extra + remote traces enabled |
Without otel, the receiver still accepts data but sends nothing to your OTLP
backend: workflow metrics go to the runner’s local metrics.jsonl, and workflow
logs go to the EC2 instance log group in CloudWatch when the stack has one.
Terraform enables remote logs and traces by default through
otel_logs_enabled and otel_traces_enabled. For all stack settings and the
signals that RunsOn itself emits, see the OpenTelemetry
reference.
Flex
Configure OtelExporterEndpoint, OtelExporterHeaders, and
OtelExporterTemporality on a CloudFormation stack. Terraform uses the
equivalent otel_exporter_* inputs. Then opt the job in with extras=otel:
jobs: instrumented: runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otelYou can also put otel in a Flex pool runner’s extras.
Fleet
Configure the OTLP exporter and add otel to the Terraform-owned runner:
otel_exporter_endpoint = "https://your-otlp-endpoint"otel_exporter_headers = "authorization=<ingestion-token>"otel_logs_enabled = trueotel_traces_enabled = true
runners = { linux-small = { # ... extras = ["otel"] }}Fleet workflows select that runner normally. They cannot add extras per job.
On Fleet, the collector restarts once shortly after the job starts, to add the
repository, workflow, and job attributes. Data sent in the first few seconds
can miss those attributes. The RunsOn agent itself listens on 127.0.0.1:4318,
so the port stays open during the restart, and each request waits up to 15
seconds for the new collector. Clients don’t need to retry. To get the
attributes on a single otel-cli span, emit it after your first setup steps.
Connect a workflow tool#
OTLP/HTTP clients that read the standard OTEL_EXPORTER_OTLP_* variables, such
as the OpenTelemetry SDKs and otel-cli, find the loopback receiver
on their own. On RunsOn versions before v3.5.0, set them on the job:
env: OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318 OTEL_EXPORTER_OTLP_PROTOCOL: http/protobufThe receiver supports OTLP over HTTP/protobuf. It does not listen for OTLP/gRPC,
and it is not exposed outside the runner. It listens on the host’s loopback
interface only, so a job container: or a Docker container on the default
bridge network cannot reach it; use host networking for tools that run in a
container.
RunsOn preserves an incoming service.name and inserts
service.instance.id when it is missing. It also adds resource attributes such
as the stack, AWS Region, organization, repository, workflow path, job name,
instance type, and instance lifecycle. Those fields let you separate workflow
telemetry from host and control-plane telemetry in the same backend.
Trace a command with otel-cli#
otel-cli ↗ can wrap a shell command
in a span. It reads the job’s endpoint and TRACEPARENT, so the span joins the
job’s trace with no configuration:
name: Trace a workflow command
on: workflow_dispatch:
permissions: contents: read
jobs: trace: runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel env: OTEL_CLI_FAIL: "true"
steps: - name: Install otel-cli run: | mkdir -p "${RUNNER_TEMP}/bin" curl -fsSL https://github.com/equinix-labs/otel-cli/releases/download/v0.4.5/otel-cli_0.4.5_linux_amd64.tar.gz \ | tar -xz -C "${RUNNER_TEMP}/bin" otel-cli echo "${RUNNER_TEMP}/bin" >> "${GITHUB_PATH}"
- name: Run an instrumented command run: | otel-cli exec \ --service ci-workflow \ --name "test suite" \ --attrs "github.run.id=${GITHUB_RUN_ID}" \ -- bash -c 'echo "running instrumented work"'On arm64 runners, download the linux_arm64 archive instead.
otel-cli exec sends its span before it exits, so the next step starts after
the collector accepted it. OTEL_CLI_FAIL=true makes a span the collector
refuses fail the step instead of passing silently. Drop extras=otel to keep
the span in the job’s capture only.
You don’t need a final wait step. When the job ends, the collector sends anything still in its batch before the instance shuts down, on Flex and on Fleet, including when the Fleet control plane terminates the instance. A span accepted in the last second of a job still reaches the backend.
Export CargoWall network verdicts#
CargoWall ↗ intercepts outbound connections with eBPF. Its GitHub Action can emit each allow or deny verdict as an OTLP log record. Point it at the local collector:
name: Enforce network policy
on: workflow_dispatch:
permissions: actions: read contents: read
jobs: network-policy: runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel steps: - name: Enable CargoWall uses: code-cargo/cargowall-action@7943209964cc5472fee8aa1e02cde8a2f0dd0414 # v1.3.7 env: OTEL_EXPORTER_OTLP_COMPRESSION: gzip OTEL_SERVICE_NAME: cargowall OTEL_RESOURCE_ATTRIBUTES: github.run.id=${{ github.run_id }} with: mode: enforce offline: "true" fail-on-unsupported: "true" allowed-hosts: | example.com:443 ingest.example.com:443
- name: Make an allowed call run: curl --fail https://example.com
- name: Verify an undeclared host is blocked run: | if curl --fail https://example.org; then echo "CargoWall unexpectedly allowed example.org" >&2 exit 1 fi echo "CargoWall intercepted example.org"
- name: Give CargoWall time to send its last batch if: always() run: sleep 20CargoWall batches records in its own process before sending them to the runner collector, and it keeps running after your last step. The final wait gives it time to hand its last batch to the collector. The collector’s own batch needs no extra wait: RunsOn flushes it when the runner shuts down.
Replay a job’s telemetry#
From RunsOn v3.5.0, every runner keeps a capture of what its collector receives:
the RunsOn agent’s spans, your spans and logs, host metrics, and the bootstrap
log, as OTLP JSON lines. It doesn’t need a backend or the otel extra. When the
instance shuts down, after the collector has written its last batch, the agent
uploads the capture to the stack’s S3 cache bucket, under
telemetry/<role-id>:<instance-id>/, where <role-id> is the unique ID of the
runner instance role. Captured logs and spans can hold a repository’s private
data, so a runner can write only its own prefix and no runner can read it. The
capture expires with the bucket’s cache retention.
roc otel replay
sends a job’s capture, unchanged, to any OTLP/HTTP endpoint, such as a local
SigNoz or Jaeger, or your production backend:
roc otel replay https://github.com/my-org/my-repo/actions/runs/123/job/456 \ --endpoint http://localhost:4318Spans keep their original timestamps and trace IDs, so set the backend’s time range to the job’s. The capture holds the runner side of the trace. On Flex, the control-plane spans (scheduling, instance launch, GitHub steps) go to the stack’s backend only. Each signal keeps up to about 100 MB per job. Fleet terminates the instance when the job completes, so its upload is best-effort.
Troubleshooting#
Check the receiver before debugging the backend:
curl --silent --output /dev/null --write-out '%{http_code}\n' \ --request POST \ --header 'Content-Type: application/x-protobuf' \ --data-binary '' \ http://127.0.0.1:4318/v1/tracesAn HTTP 200 confirms that the local trace receiver accepted the request. It
does not prove remote delivery. If data is still missing:
- Replay the job’s capture into a local backend with
roc otel replay. If the data is there, the problem is remote delivery. - Confirm that the runner has
otelenabled and the stack has an OTLP endpoint. - Confirm that remote logs or traces are enabled for the signal you send.
- Allow the remote OTLP host through any workflow-level firewall.
- If data from the last seconds of a job is missing, look for a background process that batches telemetry on its own. It needs time to hand its last batch to the collector before the job ends.
- Query your backend by
service.nameand the runner’sservice.instance.id.