A Bazel remote cache in the runner agent: how we built and measured it

RunsOn v3.6.0 adds a Bazel remote cache that needs one action input and no service to run. This post is about how it went from an idea to a measured feature: the design, the benchmark, and the two things the numbers made us build that were not in the first version.

The idea#

RunsOn already ships Magic Cache: the agent on each runner serves GitHub’s cache API locally and stores entries in an S3 bucket in your own AWS account. Bazel users asked for a remote cache.

Bazel’s HTTP cache protocol is tiny: GET and PUT on /ac/<sha256> for action results, and on /cas/<sha256> for content-addressed outputs. So the question was whether the same agent could serve it, with no new service and no infrastructure change.

The design#

• Same local server as Magic Cache, turned on by remote_cache: bazel in runs-on/action. The action writes the cache address into /etc/bazel.bazelrc, so the build needs no new flag.
• Scoped by the job’s token, without Bazel sending it. Bazel’s HTTP cache client sends no credentials, and a credential helper over plain HTTP makes Bazel warn on every command. So the action registers the job’s GitHub token with the agent, which answers with a cache address derived from it. Nothing that lacks the token can compute that address.
• GitHub’s cache scoping, reused. Bazel trusts action results without verifying them. So a job writes them only to its own branch or pull request, and reads them from the default branch first, then from its own. A pull request can use main’s results and can never change them.
• Outputs shared across the repository. They are content-addressed and verified by digest on both ends, so any branch can reuse any output.
• Uploads streamed straight to S3, hashed as they pass. An upload that does not match its digest is aborted before S3 keeps it.
• Instrumented from day one. The agent counts hits per scope, misses, uploads and latencies. Every later decision came from those numbers.

A benchmark that shows something#

The first candidate builds finished in a few minutes, too short to show what caching changes. We settled on openai/codex ↗: a 161-crate Rust workspace that builds with both Bazel and Cargo, runs about 11,000 actions and produces 89 GB of outputs.

Every variant ran on the same commit and the same runner type (32 vCPU, local NVMe), through these scenarios:

1. Cold: an empty cache.
2. Warm: the same commit again.
3. Leaf change: a change in a crate nothing depends on.
4. Core change: a change in a crate 105 others depend on.
5. Fan-out: four branches building at once.
6. Shards: four concurrent jobs on one branch, each running a quarter of the tests, followed by a full build of the same commit.

We compared three setups: Bazel with a --disk_cache saved as one archive through actions/cache (what most Bazel users on GitHub Actions do today), Cargo with rust-cache, and Bazel with the RunsOn remote cache.

The results#

Median job time over three rounds:

• Warm builds take about a minute. With --jobs=200 they take 54 seconds: a fully cached build is bound by how many cache lookups run at once.
• Branches read main’s results instead of restoring a 16.7 GB archive, and never upload one.
• Concurrent shards keep all their results. With an archive, the last shard to save wins: the follow-up build re-ran 253 actions, against 20 with the remote cache.
• Cold builds are faster too, because there is no 82-second archive upload at the end.

What about plain Cargo?#

Since the workspace builds with Cargo too, we measured that as well. sccache ↗ cuts about 80 seconds from the build step when little has changed. But cargo test spent about 500 seconds running tests in every job, cached or not: it runs test binaries one after the other and caches no results. That is where Bazel’s cached test results make the difference. Our Rust guide has the details.

What the numbers made us change#

Lookups on branches#

On a branch, every lookup first checked the branch’s own scope, which is almost empty, before finding the result in main. That was about 11,450 S3 reads per build that found nothing, and 7 to 8 seconds on a leaf-change build.

Lookups now go to main first, and to the branch’s own scope only when main has no answer. An action result is stored under a digest of the action’s command and inputs, so main’s result and the branch’s stand for the same action, and main is the scope a pull request cannot write. The agent keeps no state for it. An action that only the branch has built costs a second read, and a branch build has few of those.

We first tried two ways to keep reading the branch first without the wait: a listing of the branch’s own scope, refreshed every minute, and reading both scopes in parallel. Both worked. Reading main first was as fast, with nothing to keep up to date and a third fewer reads than the parallel version.

The parallel and main-first runs alternated on a second day, because the first build of a morning tends to be slower: that day’s first run, a main-first one, took 72 seconds. Builds that download every output end with those downloads, 80 GB in all, which took 91 to 124 seconds depending on the host, whatever the lookup order.

Compression#

Storage was the one place the remote cache lost to the archive: 89 GB per cold build, against 16.7 GB compressed. Bazel’s HTTP protocol has no compression, so the agent does it where Bazel cannot see it. It hashes what Bazel sends, compresses it with zstd on the way to S3, and decompresses on the way back.

It costs no measurable build time, even while 32 cores compile, and builds that download a lot got a few seconds faster.

What we measured and did not build#

• Skipping uploads of outputs that already exist. Within one job there is about one per build. Skipping the ones other jobs stored would let an output expire before the result that refers to it.
• Parallel part uploads for large outputs. They nearly doubled per-file upload speed and changed no job time: the end of a build is bound by the network card, not by any one upload.
• A local cache layer in front of S3. On ephemeral runners it starts empty on every job, and a higher --jobs already recovers what it could save.

Where it ended up#

One action input, and no change to the build:

runs-on: runs-on=${{ github.run_id }}/runner=16cpu-linux-x64/extras=s3-cache
steps:
  - uses: runs-on/action@v2
    with:
      remote_cache: bazel
  - uses: actions/checkout@v7
  - run: bazel test //...

• Warm builds in about a minute, instead of four with a cache archive.
• Branch and pull request builds that reuse main’s work and cannot write to it.
• Concurrent jobs that share results instead of overwriting each other.
• Storage on par with a compressed archive, shared across branches, in your own bucket.

The documentation has the setup for Flex and Fleet, the recommended flags and the limitations.