Local OpenTelemetry collector

Every RunsOn runner starts a local OpenTelemetry Collector. Workflow tools can send OTLP/HTTP data to 127.0.0.1:4318; the collector batches it, adds runner and job context, and forwards it through the OTLP exporter configured on the RunsOn stack.

Your workflow only talks to a loopback address. It does not need the backend URL or ingestion header.

Availability#

The local collector runs on Linux and Windows runners. The workflow OTLP receiver is available in RunsOn v3.2.1 and later for both Flex and Fleet.

Three things control remote delivery:

1. Configure an OTLP endpoint and, when required, an ingestion header on the RunsOn stack.
2. Enable otel on the runner.
3. Enable the pipeline for the signal you send.

Terraform enables remote logs and traces by default through otel_logs_enabled and otel_traces_enabled. For all stack settings and the signals that RunsOn itself emits, see the OpenTelemetry reference.

Flex#

Configure OtelExporterEndpoint, OtelExporterHeaders, and OtelExporterTemporality on a CloudFormation stack. Terraform uses the equivalent otel_exporter_* inputs. Then opt the job in with extras=otel:

jobs:
  instrumented:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel

You can also put otel in a Flex pool runner’s extras.

Fleet#

Configure the OTLP exporter and add otel to the Terraform-owned runner:

otel_exporter_endpoint = "https://your-otlp-endpoint"
otel_exporter_headers  = "authorization=<ingestion-token>"
otel_logs_enabled      = true
otel_traces_enabled    = true

runners = {
  linux-small = {
    # ...
    extras = ["otel"]
  }
}

Fleet workflows select that runner normally. They cannot add extras per job.

Connect a workflow tool#

Point OTLP/HTTP clients at the loopback receiver:

env:
  OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318
  OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf

The receiver supports OTLP over HTTP/protobuf. It does not listen for OTLP/gRPC, and it is not exposed outside the runner.

RunsOn preserves an incoming service.name and inserts service.instance.id when it is missing. It also adds resource attributes such as the stack, AWS Region, organization, repository, workflow path, job name, instance type, and instance lifecycle. Those fields let you separate workflow telemetry from host and control-plane telemetry in the same backend.

Trace a command with otel-cli#

otel-cli ↗ can wrap a shell command in a span. This workflow sends the span to the local collector without exposing the remote backend or its credential:

name: Trace a workflow command

on:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  trace:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel
    env:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318
      OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
      OTEL_EXPORTER_OTLP_BLOCKING: "true"
      OTEL_CLI_FAIL: "true"

    steps:
      - name: Install otel-cli
        env:
          GOBIN: ${{ runner.temp }}/bin
        run: |
          mkdir -p "${GOBIN}"
          go install github.com/equinix-labs/otel-cli@v0.4.5
          echo "${GOBIN}" >> "${GITHUB_PATH}"

      - name: Run an instrumented command
        run: |
          otel-cli exec \
            --service ci-workflow \
            --name "test suite" \
            --attrs "github.run.id=${GITHUB_RUN_ID}" \
            -- bash -c 'echo "running instrumented work"'

      - name: Allow the collector batch to flush
        if: always()
        run: sleep 15

OTEL_EXPORTER_OTLP_BLOCKING=true makes otel-cli wait until the local collector accepts the span. The collector then handles the remote connection, authentication, resource enrichment, and batching.

Export CargoWall network verdicts#

CargoWall ↗ intercepts outbound connections with eBPF. Its GitHub Action can emit each allow or deny verdict as an OTLP log record. Point it at the local collector:

name: Enforce network policy

on:
  workflow_dispatch:

permissions:
  actions: read
  contents: read

jobs:
  network-policy:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=otel
    steps:
      - name: Enable CargoWall
        uses: code-cargo/cargowall-action@7943209964cc5472fee8aa1e02cde8a2f0dd0414 # v1.3.7
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318
          OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
          OTEL_EXPORTER_OTLP_COMPRESSION: gzip
          OTEL_SERVICE_NAME: cargowall
          OTEL_RESOURCE_ATTRIBUTES: github.run.id=${{ github.run_id }}
        with:
          mode: enforce
          offline: "true"
          fail-on-unsupported: "true"
          allowed-hosts: |
            example.com:443
            ingest.example.com:443

      - name: Make an allowed call
        run: curl --fail https://example.com

      - name: Verify an undeclared host is blocked
        run: |
          if curl --fail https://example.org; then
            echo "CargoWall unexpectedly allowed example.org" >&2
            exit 1
          fi
          echo "CargoWall intercepted example.org"

      - name: Allow both OTEL batches to flush
        if: always()
        run: sleep 20

CargoWall batches records before sending them to the runner collector, and the collector has its own batch. The final wait covers both intervals on an ephemeral runner.

Troubleshooting#

Check the receiver before debugging the backend:

curl --silent --output /dev/null --write-out '%{http_code}\n' \
  --request POST \
  --header 'Content-Type: application/x-protobuf' \
  --data-binary '' \
  http://127.0.0.1:4318/v1/traces

An HTTP 200 confirms that the local trace receiver accepted the request. It does not prove remote delivery. If data is still missing:

• Confirm that the runner has otel enabled and the stack has an OTLP endpoint.
• Confirm that remote logs or traces are enabled for the signal you send.
• Allow the remote OTLP host through any workflow-level firewall.
• Allow enough time for the collector’s 10-second batch to flush.
• Query your backend by service.name and the runner’s service.instance.id.