When an S3 cache is not enough: PostHog builds with sticky disks

The previous experiment ended with a deliberately boring recommendation: use the S3 cache for the small Rust job. It was as fast as a persistent disk and simpler to operate.

PostHog is where that recommendation breaks.

First, the cold build that populated each cache:

We then made one controlled change to frontend/src/index.tsx and rebuilt from those warm caches:

Green percentages show the cost reduction from the GitHub baseline in the same table. RunsOn totals use the reported EC2 instance lifetime. Sticky totals include the prorated volume cost; S3 storage and request usage remains separate because one cache object can serve many jobs.

The sticky job is 1.84 times faster than the S3 job and 3.17 times faster than GitHub. More importantly, the logs show why: the S3-backed BuildKit job spent 268.3 seconds preparing and writing its cache export.

That export took longer than the entire sticky build job.

What we tested#

We used runs-on-demo/benchmark-posthog ↗, which pins PostHog as a submodule and runs its production Docker build. Every lane had four vCPUs and BuildKit max-parallelism=1:

• GitHub: ubuntu-24.04, observed with four vCPUs, using the GitHub Actions BuildKit cache;
• RunsOn S3: m8a.xlarge, 4 vCPUs and 16 GiB RAM, with the s3-cache extra;
• RunsOn sticky: the same m8a.xlarge, with BuildKit’s native state on a 120 GB gp3 sticky disk at 300 MB/s.

The live AWS non-spot price for m8a.xlarge in us-east-1 is $0.24344/hour. The lowest spot price observed on July 13, 2026 was $0.0709/hour. The sticky volume costs $16.60/month: $9.60 for 120 GB of gp3 storage plus $7.00 for the 175 MB/s provisioned above gp3’s included 125 MB/s.

The change was small on purpose#

A no-change build mostly measures cache lookup. A clean build mostly measures the machine. Neither represents the edit-build-push loop we wanted to study.

Before each measured partial rebuild, the workflow appended one comment to PostHog’s frontend entry point:

- name: Apply controlled frontend change
  run: |
    printf '\n// RunsOn cache benchmark: controlled frontend change.\n' \
      >> upstream/frontend/src/index.tsx

That invalidates the frontend copy and build path while leaving dependency and unrelated image layers reusable. It is a real partial rebuild, not a cache-hit demo that does no work.

The remote cache spent 4m28s exporting#

The RunsOn S3 lane still uses BuildKit’s familiar GitHub Actions cache interface:

- uses: docker/build-push-action@v4
  with:
    context: ./upstream
    cache-from: type=gha,scope=posthog-buildkit
    cache-to: type=gha,scope=posthog-buildkit,mode=max

With the RunsOn s3-cache extra enabled, those cache API objects are served from S3 in the runner’s AWS environment. That removes the public cache service as a bottleneck, but BuildKit must still turn its local graph into a remote-cache representation.

The decisive lines from the successful job are:

#83 preparing build cache for export
#83 preparing build cache for export 252.4s done
#83 DONE 268.3s

The complete docker/build-push-action step took 7m35s. Cache preparation and writing consumed 4m28s of it. The expensive part was not S3 object latency. It was serializing, compressing, and exporting a large BuildKit cache at the end of an otherwise incremental build.

Remote caches have to pay that bill because the next runner starts empty. A persistent BuildKit volume does not.

Sticky disk keeps BuildKit native#

The sticky lane asks runs-on/action for the BuildKit cache and passes the emitted builder to setup-buildx-action:

- id: runs-on
  uses: runs-on/action@ea79945d2c9578cfa268f7ec4c62221020805b42
  with:
    sticky_cache: buildkit

- uses: docker/setup-buildx-action@v4
  with:
    name: ${{ steps.runs-on.outputs.buildkit-builder }}
    cleanup: false
    buildkitd-config-inline: |
      ${{ steps.runs-on.outputs.buildkit-inline-config }}

- uses: docker/build-push-action@v4
  with:
    builder: ${{ steps.runs-on.outputs.buildkit-builder }}
    context: ./upstream

This distinction matters. Creating a second BuildKit container would put the build back on ephemeral storage and make the sticky disk benchmark meaningless. The action-provided builder points BuildKit at the restored volume, so snapshots, metadata, and content stay in BuildKit’s own on-disk format.

At job completion, RunsOn snapshots the EBS volume. The next ephemeral runner restores it. The VM remains disposable; only the named cache survives.

Cold, warm, and partial rebuilds#

The 300 MB/s series produced these results.

Cold build#

No-change warm build#

Controlled frontend edit#

GitHub costs assume the current $0.012/minute 4-core Linux larger runner ↗ on a private repository. GitHub rounds partial job minutes up, so the 30m01s cold job bills 31 minutes and the 1m18s warm job bills two. The public benchmark repository itself was not charged.

RunsOn compute uses the longer instance lifetime reported by the runner rather than the shorter GitHub job duration. The non-spot column uses $0.24344/hour. The spot estimate applies the lowest observed July 13 rate of $0.0709/hour to the same lifetime; actual spot prices and placement vary by Availability Zone and time. These figures exclude the RunsOn subscription and shared control-plane costs. S3 storage and request charges are not included. Sticky volume cost is shown separately and remains the same in the spot and non-spot totals.

The no-change warm result is worth noticing. S3 and sticky are tied at 46 and 45 seconds. Just like the Rust experiment, a remote cache is sufficient when little or no new cache state must be exported.

The frontend edit changes the economics. Sticky saves 3m42s and costs about 39% less than S3 compute alone. Against a private GitHub 4-core runner, it saves 9m35s and about 88% of the measured variable cost.

Persistent does not mean unlimited#

BuildKit state grows, so we also tested cleanup instead of pretending the disk has infinite capacity.

A 40 GB disk eventually filled and produced no space left on device. On the next job, RunsOn detected less than 5% free space, removed the BuildKit root, and completed a clean rebuild. That proves the emergency reset works, but 40 GB is too small for this graph.

A fresh 60 GB disk with BuildKit GC configured completed four consecutive frontend mutations without errors:

GC restored space in v3 and v4, but v4 became much slower after useful layers were evicted. Sixty gigabytes is enough for correctness, not for consistently fast PostHog rebuilds. The production-minded choice from this experiment is the 120 GB disk at 300 MB/s: enough headroom, without paying for the 750 MB/s configuration that showed no repeatable speed advantage.

Use the smallest cache architecture that fits the state#

The two experiments lead to one rule:

Use S3 for cacheable directories. Use a sticky disk when exporting the cache costs as much as the build.

For spotify-player, S3 and sticky finish within three seconds of each other, so S3 is the obvious choice. For PostHog, BuildKit spends 268 seconds preparing and writing the remote cache. Keeping its native state on a sticky volume cuts the partial rebuild nearly in half.

A sticky disk is not a long-lived runner. Every job still gets a clean EC2 instance. It is a narrow persistence primitive for the part of CI that actually benefits from being local.