Disks? Where we're going, we don't need disks

In our disk benchmark, a RunsOn runner looks broken. On 4 vCPU, its EBS gp3 volume answers 1,828 small random reads per second. GitHub’s own runner answers 6,035. Blacksmith’s answers 84,602.

If a CI job used its disk the way that test does, a build on RunsOn would take 46 times longer than on Blacksmith. It does not. The Rust build takes 1m55s on RunsOn and 1m59s on Blacksmith. The TypeScript build takes 59s and 54s.

The Docker build is the only one where the disk shows: 9m05s against 5m19s. That is 1.7 times, not 46.

So I looked at what a build asks of a disk, and then I stopped using one:

runs-on: runs-on=${{ github.run_id }}/family=r8a.xlarge/image=ubuntu26-full-x64/extras=tmpfs

With that label the Docker build takes 5m02s and costs $0.0135, 13 times less than on GitHub’s own runner. That is the lowest cost of any 4 vCPU x64 runner we measured. Namespace, the next fastest, is 5 seconds behind.

A build does not read like a benchmark#

The test behind those first numbers reads 4 KB blocks at random places, one at a time. EBS is network storage, so each read is a round trip, and the volume answers 1,828 of them per second.

A build does not work like that. It goes through the Linux page cache. Writes pile up in memory and leave in large contiguous requests. A file the job has just written is read back from memory.

RunsOn prints disk charts at the end of every job log, so I did not have to guess. These are the last three runs of each build on m8a.xlarge, on its default volume:

The average request is 12 to 36 times the size of the benchmark’s. That is why a 46 times gap on small random reads turns into nothing on the Rust build.

The Rust build peaks at 519 operations per second on a volume that allows 3,000. No disk can make that build faster.

The Docker build is the one that hits the ceiling. It stops at about 3,250 operations per second in all three runs. On an m8a.2xlarge it stops at about 3,200 on the default volume, and peaks between 10,800 and 12,200 on a volume provisioned for 16,000.

So the default volume is fast enough for most CI jobs, and it is a cap for the ones that write a lot at once.

What the label does#

extras=tmpfs mounts a tmpfs at /mnt/tmpfs and moves three things onto it:

• /home/runner, so your checkout and everything the job writes next to it
• /tmp
• Docker’s data

The workflow does not change. The paths are the same, they just live in memory. A RunsOn runner is thrown away after the job, so there is nothing on that disk I wanted to keep anyway.

It works on Linux runners, with Flex and Fleet.

The disk benchmark from the top of this post has a Storage pill. Add tmpfs to it and the same chart looks like this:

1,109,976 reads per second, 13 times Blacksmith. It is RAM, so of course it is. And that score says as little about a build as the 1,828 did, so here are the builds.

Same machine, with and without#

The fair test first: one instance type, m8a.2xlarge (8 vCPU, 32 GB), once on its gp3 volume and once with extras=tmpfs. Same CPU, same price per minute. The three builds are the ones from our runner benchmarks.

The Rust build did not move. It was never waiting for the disk.

The TypeScript build is where I noticed it first. bun install went from 24 seconds to 8. The typecheck took 19 seconds both times.

The Docker build lost 1m45s, and a third of its cost.

Then the faster volume. I gave the same m8a.2xlarge a gp3 with 1,000 MiB/s and 16,000 IOPS. Its small random read score did not change: 1,775 per second against 1,679. The Docker build still got 21% faster, at 4m27s, which says again how little that score tells you about a build.

It cost $0.0291. That is faster than the default volume, slower than tmpfs, and the most expensive of the three, because AWS bills the provisioned throughput for every minute the runner exists.

At 4 vCPU you need more memory#

PostHog’s build writes about 29 GB into Docker’s data. An m8a.xlarge has 16 GB, so at 4 vCPU I moved to r8a.xlarge. It has the same EPYC cores and 32 GB.

The instance costs more: $0.105 an hour at spot against $0.069. The build is 45% shorter, so each build costs less.

Here is the PostHog image build on every provider’s 4 vCPU x64 runner:

I count Namespace’s 5 seconds as a tie on time. The cost is far apart. This build costs 3.4 times as much on Blacksmith, 4.8 times on Namespace and 13 times on GitHub.

It holds at 8 vCPU. The m8a.2xlarge with tmpfs builds the image in 3m54s for $0.0155. Namespace’s 8 vCPU / 16 GB runner takes 3m57s for $0.0522, Blacksmith’s 3m58s for $0.0704.

It holds on Graviton too. r9g.xlarge with tmpfs takes 5m27s for $0.0124, against 9m10s and $0.0194 for m9g.xlarge on its volume. That one has three runs so far.

RunsOn costs are what each job billed, as measured by the RunsOn control plane: EC2 at spot plus the EBS volume. At on-demand prices the r8a.xlarge build costs $0.032, which is still under everyone except Ubicloud. The other providers are job time multiplied by their list rate. The RunsOn license is a fixed monthly cost and is not in these figures; this post works out where it breaks even.

The one thing EC2 was missing#

A specialized CI runner like Blacksmith’s or Namespace’s always comes with a fast disk. That was the one thing an EC2 instance did not have against them. tmpfs closes it, for the jobs that need it.

It works because memory is the cheap part of EC2. Going from m8a.xlarge to r8a.xlarge adds 16 GB of RAM for 3.6 cents an hour at spot. With its volume, r8a.xlarge costs $0.0020 a minute at spot and $0.0056 on-demand.

The specialized providers mostly do not sell memory that way. The 32 GB runners in our catalog all come with 8 vCPU, at $0.016 a minute for Blacksmith and WarpBuild, $0.022 for GitHub and $0.024 for Namespace. Ubicloud is the closest at $0.005.

Where it does nothing#

Do not put a Rust build on an r8a because of this post. It takes 1m59s there and 1m55s on m8a.xlarge, for $0.0063 instead of $0.0042.

tmpfs pays when a job writes more than the volume lets through: Docker builds, package installs, cache restores (a 4 GB entry comes back in 7 seconds instead of 13) and tests that run a database. On the same m8a.2xlarge, pgbench ran more than 8 times as many read/write transactions per second.

It is called YOLO mode for a reason#

In RunsOn this feature is called YOLO mode. The RAM is the disk, and when it is full the job dies. You have to size the runner properly, once.

I did not, the first time. My first two runs of the Docker build on r8a.xlarge ran out of memory. The tmpfs was capped at 100% of RAM and a 32 GB runner had no swap, which leaves no room for a build that writes 29 GB.

Most of what a Docker build writes is cold. So from the next RunsOn release, tmpfs runners get compressed swap in memory (zram) at any size, and their tmpfs can grow to 150% of RAM. The same build has passed four times out of four since.

That release also puts /var/lib/containerd on the tmpfs. That is where Docker 29, on the Ubuntu 26.04 images, keeps its layers.

16 GB is still not enough for this build. With tmpfs on, the Complete runner step shows a Tmpfs Memory Utilization (%) chart, so you can see how close a job gets.

What I would do#

Open the disk charts at the end of one of your job logs. If the job sits near 3,000 operations per second or 400 MiB/s, add extras=tmpfs on the instance you already use.

If it does not fit in memory, do not go up a size. Change the family to r8a or r9g: twice the memory with the same cores, for a few cents an hour.

If the job already uses most of the runner’s memory itself, keep the disk and ask for a faster one with the volume label. I tested volume=80gb:gp3:1000mbs:16000iops, and the Docker build got 21% faster on an m8a.2xlarge.

That was more than the build needed. It peaked at about 12,200 operations per second and 540 MiB/s. AWS bills what you provision, not what you use, so read the peaks from your own charts and ask for a little more than that. Even so, the build cost more than on the default volume, so try tmpfs first.

If the charts stay far under those limits, leave the job on its volume.