Fastest GitHub Actions runners: disk I/O.
fio inside the job workspace of 94 runners from 9 providers, 2 to 12 vCPU. These are the disk's limits. A build goes through the Linux page cache and often never reaches them.
How this shows up in real builds EC2 storage: NVMe, gp3 and tmpfs on the same instances
Latency per provider
Each provider's best disk, RunsOn once per storage (the Storage pill). QD1 is one 4k read at a time: what a database or a cold read pays. Most build steps batch their I/O through the page cache. tmpfs is RAM, not a disk: add it to see the gap on the same scale.
Fastest x64 QD1 read: Namespace (106,191), ahead of Blacksmith (73,779).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 synced writes: Namespace (30,836), ahead of Blacksmith (17,925).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 2 vCPU QD1 read: Namespace (104,675), ahead of Blacksmith (73,779).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 2 vCPU synced writes: Namespace (30,063), ahead of Blacksmith (17,755).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 4 vCPU QD1 read: Namespace (105,744), ahead of Blacksmith (71,728).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 4 vCPU synced writes: Namespace (30,836), ahead of Blacksmith (17,925).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 8 vCPU QD1 read: Namespace (106,191), ahead of Blacksmith (70,703).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 8 vCPU synced writes: Namespace (29,050), ahead of Blacksmith (17,670).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 QD1 read: Blacksmith (32,740), ahead of Warpbuild (30,249).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 synced writes: RunsOn m9gd.2xlarge · NVMe (17,389), ahead of Blacksmith (9,537).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 2 vCPU QD1 read: Blacksmith (29,968), ahead of Avrea (20,905).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 2 vCPU synced writes: Blacksmith (8,115), ahead of RunsOn m9gd.large · NVMe (7,267).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 4 vCPU QD1 read: Blacksmith (32,740), ahead of Avrea (19,951).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 4 vCPU synced writes: RunsOn m9gd.xlarge · NVMe (14,860), ahead of Blacksmith (9,366).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 8 vCPU QD1 read: Blacksmith (29,312), ahead of Avrea (20,743).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 8 vCPU synced writes: RunsOn m9gd.2xlarge · NVMe (17,389), ahead of Blacksmith (9,537).volatile and nobarrier mounts make each flush cheaper by making it less durable.
What the numbers say
Extremes on x64, and RAM against disk.
- On x64, one small read at a time (4k, QD1) comes back 106,191 times a second on Namespace 8x16 and 1,471 on RunsOn m8a.2xlarge · EBS gp3, 72× apart; p99 latency is 10 µs against 963 µs.
- Synced 4k writes on x64 run from 486 per second on RunsOn m8i.2xlarge · EBS gp3 (workspace mounted nobarrier) to 30,836 on Namespace 4x8.
- r8a.large with tmpfs ($0.0009/min at spot) answers 1,106,090 QD1 reads and 428,067 synced writes per second: 90× and 33× the best local NVMe at 2 vCPU (i7i.large, $0.0010) and 611× and 791× default EBS gp3 (c8a.large, $0.0008); its workspace is 14 GiB, from RAM. (EC2 storage suite) What that does to real builds.
More findings23: EBS, arm64, NVMe vs EBS, provisioned gp3, tmpfs, storage classes
- Every EBS-backed workspace (3 AWS CodeBuild x64 runners, 25 RunsOn configurations and 3 Warpbuild configurations) answers between 1,471 and 1,837 QD1 reads and 486 to 571 synced writes per second, whoever runs it and whatever throughput it is provisioned for: each request is a round trip to network storage.
- On arm64, one small read at a time (4k, QD1) comes back 32,740 times a second on Blacksmith 4 vCPU and 1,702 on RunsOn m9g.large · EBS gp3, 19× apart; p99 latency is 37 µs against 905 µs.
- Synced 4k writes on arm64 run from 537 per second on RunsOn c8g.large · EBS gp3 (workspace mounted nobarrier) to 17,389 on RunsOn m9gd.2xlarge · local NVMe.
- Same Xeon 6975P-C CPU, different disk: m8id.2xlarge on local NVMe answers 11,025 QD1 reads and 7,516 synced writes per second, m8i.2xlarge on EBS gp3 1,565 and 486.
- Same Xeon 6975P-C CPU, different disk: m8id.large on local NVMe answers 11,102 QD1 reads and 5,555 synced writes per second, m8i-flex.large on EBS gp3 1,771 and 530.
- Same Xeon 6975P-C CPU, different disk: m8id.xlarge on local NVMe answers 10,866 QD1 reads and 7,903 synced writes per second, m8i-flex.xlarge on EBS gp3 1,753 and 525.
- Same Neoverse-V3 CPU, different disk: m9gd.2xlarge on local NVMe answers 13,454 QD1 reads and 17,389 synced writes per second, m9g.2xlarge on EBS gp3 1,768 and 543.
- Same Neoverse-V3 CPU, different disk: m9gd.xlarge on local NVMe answers 13,375 QD1 reads and 14,860 synced writes per second, m9g.xlarge on EBS gp3 1,837 and 543.
- Same Neoverse-V3 CPU, different disk: m9gd.large on local NVMe answers 13,517 QD1 reads and 7,267 synced writes per second, m9g.large on EBS gp3 1,702 and 542.
- Provisioning gp3 at 1000 MiB/s on m8a.2xlarge takes sequential reads from 401 MiB/s to 1,001 MiB/s (2.5×); QD1 reads go (1,587 → 1,776 IOPS).
- Provisioning gp3 at 1000 MiB/s on m8azn.xlarge takes sequential reads from 401 MiB/s to 1,001 MiB/s (2.5×); QD1 reads go (1,583 → 1,772 IOPS).
- tmpfs on m8a.2xlarge is not a disk at all: 1,136,061 QD1 reads and 429,749 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on m9g.2xlarge is not a disk at all: 653,110 QD1 reads and 300,477 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on m9g.xlarge is not a disk at all: 667,201 QD1 reads and 303,829 synced writes per second, from RAM. The workspace is 14 GiB, sized by memory.
- tmpfs on r8a.2xlarge is not a disk at all: 1,153,982 QD1 reads and 430,608 synced writes per second, from RAM. The workspace is 59 GiB, sized by memory.
- tmpfs on r8a.xlarge is not a disk at all: 1,102,306 QD1 reads and 433,841 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on r9g.large is not a disk at all: 652,948 QD1 reads and 297,441 synced writes per second, from RAM. The workspace is 14 GiB, sized by memory.
- tmpfs on r9g.xlarge is not a disk at all: 643,948 QD1 reads and 301,871 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs: 644k–1.2M QD1 reads and 297k–434k synced writes per second across 8 runners (8 RunsOn configurations).
- VM disk: 4.8k–106k QD1 reads and 749–31k synced writes per second across 44 runners from 7 providers (6 Avrea runners, 6 Blacksmith runners, 3 boat x64 runners, 3 GitHub runners, 12 Namespace runners, 9 Ubicloud runners and 5 Warpbuild runners).
- local NVMe: 11k–14k QD1 reads and 5.6k–17k synced writes per second across 9 runners (9 RunsOn configurations).
- network disk: 6.9k–7.3k QD1 reads and 3.5k–4.2k synced writes per second across 2 runners (2 GitHub runners).
- EBS: 1.5k–1.8k QD1 reads and 486–571 synced writes per second across 31 runners from 3 providers (3 AWS CodeBuild x64 runners, 25 RunsOn configurations and 3 Warpbuild configurations).
Reading these numbers
fio measures what a disk can do. Whether a job gets there depends on the job.
What each column says about a job
- QD1 reads and synced writes: databases, SQLite-backed tests and anything that flushes each write. Every flush waits for the storage, and a network disk pays a round trip each time.
-
Random writes at depth and sequential throughput:
docker buildlayers, package installs and cache archives. They batch their writes and run into the disk's ceiling, not its latency. - None of them: compile jobs. A compiler waits for the CPU, and it reads its files back from the page cache.
Every runner
Each provider's best disk first; its other machines sit behind +N more. Bars are log scale.
| One request at a time | 4k, 32 in flight | Sequential, 1M | |||||
|---|---|---|---|---|---|---|---|
| Runner | QD1 read ↑ | p99 ↓ | Sync write ↑ | Read ↑ | Write ↑ | Read ↑ | Write ↑ |
| x64 | |||||||
| Namespace 8x16 | 10 µs | 931,530 | 546,288 | 4.5 GiB/s | 2.9 GiB/s | ||
| 8x32 | 10 µs | 1,095,705 | 569,755 | 5.0 GiB/s | 3.4 GiB/s | ||
| 4x16 | 11 µs | 1,202,685 | 564,373 | 5.0 GiB/s | 3.4 GiB/s | ||
| 4x8 | 10 µs | 1,209,187 | 607,540 | 5.0 GiB/s | 3.4 GiB/s | ||
| 2x8 | 10 µs | 820,729 | 574,093 | 4.5 GiB/s | 3.0 GiB/s | ||
| Blacksmith | 17 µs | 703,229 | 166,801 | 5.5 GiB/s | 2.0 GiB/s | ||
| 8 vCPU | 17 µs | 511,769 | 231,934 | 6.4 GiB/s | 1.9 GiB/s | ||
| 4 vCPU | 16 µs | 728,035 | 218,915 | 7.0 GiB/s | 1.7 GiB/s | ||
| Avrea | 14 µs | 564,218 | 245,269 | 16.0 GiB/s | 10.6 GiB/s | ||
| 8 vCPU | 18 µs | 343,658 | 200,865 | 16.8 GiB/s | 11.2 GiB/s | ||
| 4 vCPU | 21 µs | 453,992 | 236,446 | 15.9 GiB/s | 11.0 GiB/s | ||
| Warpbuild | 37 µs | 199,592 | 90,570 | 4.2 GiB/s | 2.7 GiB/s | ||
| 8 vCPU | 27 µs | 189,921 | 75,793 | 4.4 GiB/s | 3.1 GiB/s | ||
| 4 vCPU | 31 µs | 198,918 | 75,037 | 4.2 GiB/s | 2.9 GiB/s | ||
| RunsOn i7i.large | 289 µs | 75,023 | 41,231 | 503 MiB/s | 395 MiB/s | ||
| m8azn.3xlarge | 811 µs | 2,997 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| c8a.2xlarge | 897 µs | 2,995 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| r8a.2xlarge | <1 µs | 4,310,049 | 3,831,509 | 18.4 GiB/s | 10.0 GiB/s | ||
| m8a.2xlarge | <1 µs | 4,238,812 | 3,808,170 | 18.6 GiB/s | 9.9 GiB/s | ||
| m8a.2xlarge | 795 µs | 15,995 | 15,992 | 1,001 MiB/s | 1,001 MiB/s | ||
| m8a.2xlarge | 963 µs | 2,995 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| m8azn.xlarge | 918 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8azn.xlarge | 799 µs | 15,994 | 15,991 | 1,001 MiB/s | 1,001 MiB/s | ||
| m8i.2xlarge | 995 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8id.2xlarge | 239 µs | 134,258 | 67,076 | 905 MiB/s | 432 MiB/s | ||
| m8i-flex.2xlarge | 954 µs | 2,993 | 2,995 | 401 MiB/s | 401 MiB/s | ||
| m8a.xlarge | 807 µs | 2,994 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| c8a.xlarge | 897 µs | 2,994 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| i7i.2xlarge | 324 µs | 300,467 | 165,110 | 2.0 GiB/s | 1.5 GiB/s | ||
| r8a.xlarge | <1 µs | 4,201,526 | 3,800,170 | 17.8 GiB/s | 9.9 GiB/s | ||
| m8azn.large | 754 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8id.xlarge | 249 µs | 67,090 | 33,526 | 451 MiB/s | 216 MiB/s | ||
| m8i.xlarge | 954 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8i-flex.xlarge | 918 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| i7i.xlarge | 281 µs | 150,053 | 82,487 | 1,010 MiB/s | 788 MiB/s | ||
| m8a.large | 872 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r8a.large | 860 µs | 2,995 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r8a.large | <1 µs | 2,147,299 | 1,930,574 | 18.0 GiB/s | 10.0 GiB/s | ||
| c8a.large | 754 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8id.large | 167 µs | 33,541 | 16,753 | 226 MiB/s | 108 MiB/s | ||
| m8i.large | 987 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8i-flex.large | 881 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| t8i.medium | 827 µs | 3,001 | 2,998 | 401 MiB/s | 401 MiB/s | ||
| boat | 218 µs | 460,252 | 120,989 | 10.2 GiB/s | 832 MiB/s | ||
| 8 vCPU | 545 µs | 309,424 | 85,110 | 8.1 GiB/s | 1,020 MiB/s | ||
| 4 vCPU | 297 µs | 385,820 | 154,383 | 7.8 GiB/s | 1.3 GiB/s | ||
| Ubicloud premium | 515 µs | 289,856 | 19,284 | 2.9 GiB/s | 993 MiB/s | ||
| standard 8 vCPU | 587 µs | 258,356 | 20,072 | 2.6 GiB/s | 989 MiB/s | ||
| standard 4 vCPU | 713 µs | 176,445 | 24,130 | 2.4 GiB/s | 620 MiB/s | ||
| premium 4 vCPU | 860 µs | 224,242 | 23,696 | 2.3 GiB/s | 902 MiB/s | ||
| standard 2 vCPU | 763 µs | 128,623 | 23,218 | 2.4 GiB/s | 710 MiB/s | ||
| premium 2 vCPU | 1,733 µs | 116,749 | 23,514 | 2.5 GiB/s | 832 MiB/s | ||
| GitHub | 231 µs | 38,976 | 18,540 | 784 MiB/s | 540 MiB/s | ||
| 4-core | 241 µs | 19,575 | 11,216 | 394 MiB/s | 244 MiB/s | ||
| 2-core | 262 µs | 9,377 | 8,367 | 199 MiB/s | 166 MiB/s | ||
| AWS CodeBuild medium | 786 µs | 2,992 | 2,974 | 241 MiB/s | 251 MiB/s | ||
| large | 741 µs | 2,989 | 2,977 | 243 MiB/s | 251 MiB/s | ||
| small | 803 µs | 2,988 | 2,972 | 251 MiB/s | 251 MiB/s | ||
| arm64 | |||||||
| Blacksmith | 37 µs | 253,175 | 35,622 | 1.7 GiB/s | 829 MiB/s | ||
| 8 vCPU | 36 µs | 251,923 | 32,466 | 1.7 GiB/s | 837 MiB/s | ||
| 2 vCPU | 40 µs | 250,107 | 32,701 | 1.7 GiB/s | 832 MiB/s | ||
| Warpbuild xfast (Apple M4 Pro) | 33 µs | 383,964 | 5,773 | 11.0 GiB/s | 11.6 GiB/s | ||
| xfast (Apple M4 Pro) 6 vCPU | 48 µs | 362,075 | 5,561 | 12.2 GiB/s | 10.8 GiB/s | ||
| 8 vCPU | 823 µs | 4,995 | 4,993 | 401 MiB/s | 401 MiB/s | ||
| 4 vCPU | 856 µs | 4,194 | 4,193 | 302 MiB/s | 302 MiB/s | ||
| 2 vCPU | 819 µs | 4,194 | 4,194 | 302 MiB/s | 301 MiB/s | ||
| Avrea | 60 µs | 152,164 | 17,974 | 31.1 GiB/s | 14.7 GiB/s | ||
| 8 vCPU | 56 µs | 143,512 | 14,241 | 40.1 GiB/s | 16.0 GiB/s | ||
| 4 vCPU | 59 µs | 147,857 | 16,821 | 36.8 GiB/s | 15.6 GiB/s | ||
| Namespace Apple silicon 2x8 | 78 µs | 153,165 | 15,192 | 31.2 GiB/s | 11.8 GiB/s | ||
| Apple silicon 4x16 | 97 µs | 145,293 | 23,426 | 32.6 GiB/s | 12.7 GiB/s | ||
| 8x32 | 90 µs | 183,882 | 50,801 | 919 MiB/s | 772 MiB/s | ||
| 8x16 | 94 µs | 183,210 | 39,558 | 923 MiB/s | 695 MiB/s | ||
| 4x8 | 92 µs | 142,916 | 37,163 | 930 MiB/s | 650 MiB/s | ||
| 4x16 | 94 µs | 147,080 | 34,602 | 932 MiB/s | 680 MiB/s | ||
| 2x8 | 103 µs | 75,288 | 25,671 | 924 MiB/s | 663 MiB/s | ||
| RunsOn m9gd.large | 81 µs | 43,609 | 21,784 | 293 MiB/s | 141 MiB/s | ||
| m9gd.2xlarge | 83 µs | 174,573 | 87,209 | 1.1 GiB/s | 561 MiB/s | ||
| m9g.2xlarge | <1 µs | 2,615,598 | 2,487,852 | 13.0 GiB/s | 9.1 GiB/s | ||
| m9g.2xlarge | 901 µs | 2,995 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| c8g.2xlarge | 856 µs | 2,994 | 2,993 | 401 MiB/s | 401 MiB/s | ||
| m9gd.xlarge | 111 µs | 87,221 | 43,591 | 585 MiB/s | 281 MiB/s | ||
| m9g.xlarge | 868 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r9g.xlarge | <1 µs | 2,605,268 | 2,485,222 | 14.0 GiB/s | 8.9 GiB/s | ||
| m9g.xlarge | <1 µs | 2,686,147 | 2,522,480 | 14.7 GiB/s | 9.3 GiB/s | ||
| c8g.xlarge | 860 µs | 2,992 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m9g.large | 905 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| r9g.large | <1 µs | 1,277,889 | 1,234,447 | 13.8 GiB/s | 9.2 GiB/s | ||
| c8g.large | 836 µs | 2,993 | 2,993 | 401 MiB/s | 401 MiB/s | ||
| Ubicloud standard | 389 µs | 234,460 | 77,711 | 755 MiB/s | 509 MiB/s | ||
| standard 4 vCPU | 420 µs | 147,368 | 34,056 | 738 MiB/s | 379 MiB/s | ||
| standard 2 vCPU | 510 µs | 98,852 | 28,872 | 761 MiB/s | 414 MiB/s | ||
| GitHub | 215 µs | 9,376 | 6,264 | 199 MiB/s | 164 MiB/s | ||
| 4-core | 225 µs | 19,575 | 12,113 | 394 MiB/s | 325 MiB/s | ||
- IOPS are 4k blocks, medians of finished jobs
- bold best disk measured on the arch (tmpfs is RAM, left out)
- , : cheaper, less durable flushes
Pick the disk per job on RunsOn
-
The instance type sets the storage under the workspace: EBS gp3 by default, provisioned gp3 for bandwidth, local NVMe
on the
dandifamilies, or tmpfs when the job fits in RAM. Pick it with thefamilylabel; each one, measured: EC2 storage benchmark. - What tmpfs does to real builds, what it costs and where it does nothing: Disks? Where we're going, we don't need disks.
How it's measured
fio runs inside the job, in GITHUB_WORKSPACE: the disk a build writes to, on the filesystem the provider
mounted there. Every runner is compared, whatever its shape.
fio settingsblock sizes, queue depths, mounts
- QD1 random read: 4k blocks, one request in flight; IOPS and p99 latency.
- Sync write: 4k writes, each followed by
fdatasync. - Random read and write: 4k blocks, 4 jobs with 32 requests in flight each.
- Sequential read and write: 1M blocks, 32 requests in flight.
- The workspace filesystem, its size and its mount options come from the job's own mount table.
- Earlier versions of this page used a different harness and other fio settings on 2 vCPU runners; those numbers are gone rather than mixed in.