Fastest GitHub Actions runners: disk I/O.
fio inside the job workspace of 94 runners from 9 providers, 2 to 12 vCPU. These are the disk's limits. A build goes through the Linux page cache and often never reaches them.
How this shows up in real builds
Latency per provider
Each provider's best disk, RunsOn once per storage (the Storage pill). QD1 is one 4k read at a time: what a database or a cold read pays. Most build steps batch their I/O through the page cache. tmpfs is RAM, not a disk: add it to see the gap on the same scale.
Fastest x64 QD1 read: Namespace (106,191), ahead of Blacksmith (73,779).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 synced writes: Namespace (30,836), ahead of Blacksmith (17,925).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 2 vCPU QD1 read: Namespace (104,675), ahead of Blacksmith (73,779).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 2 vCPU synced writes: Namespace (30,063), ahead of Blacksmith (17,755).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 4 vCPU QD1 read: Namespace (105,744), ahead of Blacksmith (71,728).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 4 vCPU synced writes: Namespace (30,836), ahead of Blacksmith (17,925).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest x64 8 vCPU QD1 read: Namespace (106,191), ahead of Blacksmith (70,703).tmpfs for scale: 1,153,982 on RunsOn r8a.2xlarge · tmpfs.
Fastest x64 8 vCPU synced writes: Namespace (29,050), ahead of Blacksmith (17,670).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 QD1 read: Blacksmith (32,740), ahead of Warpbuild (30,249).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 synced writes: RunsOn m9gd.2xlarge · NVMe (17,389), ahead of Blacksmith (9,537).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 2 vCPU QD1 read: Blacksmith (29,968), ahead of Avrea (20,905).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 2 vCPU synced writes: Blacksmith (8,115), ahead of RunsOn m9gd.large · NVMe (7,267).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 4 vCPU QD1 read: Blacksmith (32,740), ahead of Avrea (19,951).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 4 vCPU synced writes: RunsOn m9gd.xlarge · NVMe (14,860), ahead of Blacksmith (9,366).volatile and nobarrier mounts make each flush cheaper by making it less durable.
Fastest arm64 8 vCPU QD1 read: Blacksmith (29,312), ahead of Avrea (20,743).tmpfs for scale: 667,201 on RunsOn m9g.xlarge · tmpfs.
Fastest arm64 8 vCPU synced writes: RunsOn m9gd.2xlarge · NVMe (17,389), ahead of Blacksmith (9,537).volatile and nobarrier mounts make each flush cheaper by making it less durable.
What the numbers say
Extremes on x64, and RAM against disk.
- On x64, one small read at a time (4k, QD1) comes back 106,191 times a second on Namespace 8x16 and 1,471 on RunsOn m8a.2xlarge · EBS gp3, 72× apart; p99 latency is 10 µs against 963 µs.
- Synced 4k writes on x64 run from 486 per second on RunsOn m8i.2xlarge · EBS gp3 (workspace mounted nobarrier) to 30,836 on Namespace 4x8.
- r8a.large with tmpfs ($0.0009/min at spot) answers 1,106,090 QD1 reads and 428,067 synced writes per second: 90× and 33× the best local NVMe at 2 vCPU (i7i.large, $0.0010) and 611× and 791× default EBS gp3 (c8a.large, $0.0009); its workspace is 14 GiB, from RAM. (EC2 storage suite) What that does to real builds.
More findings23: EBS, arm64, NVMe vs EBS, provisioned gp3, tmpfs, storage classes
- Every EBS-backed workspace (3 AWS CodeBuild x64 runners, 25 RunsOn configurations and 3 Warpbuild configurations) answers between 1,471 and 1,837 QD1 reads and 486 to 571 synced writes per second, whoever runs it and whatever throughput it is provisioned for: each request is a round trip to network storage.
- On arm64, one small read at a time (4k, QD1) comes back 32,740 times a second on Blacksmith 4 vCPU and 1,702 on RunsOn m9g.large · EBS gp3, 19× apart; p99 latency is 37 µs against 905 µs.
- Synced 4k writes on arm64 run from 537 per second on RunsOn c8g.large · EBS gp3 (workspace mounted nobarrier) to 17,389 on RunsOn m9gd.2xlarge · local NVMe.
- Same Xeon 6975P-C CPU, different disk: m8id.2xlarge on local NVMe answers 11,025 QD1 reads and 7,516 synced writes per second, m8i.2xlarge on EBS gp3 1,565 and 486.
- Same Xeon 6975P-C CPU, different disk: m8id.large on local NVMe answers 11,102 QD1 reads and 5,555 synced writes per second, m8i-flex.large on EBS gp3 1,771 and 530.
- Same Xeon 6975P-C CPU, different disk: m8id.xlarge on local NVMe answers 10,866 QD1 reads and 7,903 synced writes per second, m8i-flex.xlarge on EBS gp3 1,753 and 525.
- Same Neoverse-V3 CPU, different disk: m9gd.2xlarge on local NVMe answers 13,454 QD1 reads and 17,389 synced writes per second, m9g.2xlarge on EBS gp3 1,768 and 543.
- Same Neoverse-V3 CPU, different disk: m9gd.xlarge on local NVMe answers 13,375 QD1 reads and 14,860 synced writes per second, m9g.xlarge on EBS gp3 1,837 and 543.
- Same Neoverse-V3 CPU, different disk: m9gd.large on local NVMe answers 13,517 QD1 reads and 7,267 synced writes per second, m9g.large on EBS gp3 1,702 and 542.
- Provisioning gp3 at 1000 MiB/s on m8a.2xlarge takes sequential reads from 401 MiB/s to 1,001 MiB/s (2.5×); QD1 reads go (1,587 → 1,776 IOPS).
- Provisioning gp3 at 1000 MiB/s on m8azn.xlarge takes sequential reads from 401 MiB/s to 1,001 MiB/s (2.5×); QD1 reads go (1,583 → 1,772 IOPS).
- tmpfs on m8a.2xlarge is not a disk at all: 1,136,061 QD1 reads and 429,749 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on m9g.2xlarge is not a disk at all: 653,110 QD1 reads and 300,477 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on m9g.xlarge is not a disk at all: 667,201 QD1 reads and 303,829 synced writes per second, from RAM. The workspace is 14 GiB, sized by memory.
- tmpfs on r8a.2xlarge is not a disk at all: 1,153,982 QD1 reads and 430,608 synced writes per second, from RAM. The workspace is 59 GiB, sized by memory.
- tmpfs on r8a.xlarge is not a disk at all: 1,102,306 QD1 reads and 433,841 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs on r9g.large is not a disk at all: 652,948 QD1 reads and 297,441 synced writes per second, from RAM. The workspace is 14 GiB, sized by memory.
- tmpfs on r9g.xlarge is not a disk at all: 643,948 QD1 reads and 301,871 synced writes per second, from RAM. The workspace is 29 GiB, sized by memory.
- tmpfs: 644k–1.2M QD1 reads and 297k–434k synced writes per second across 8 runners (8 RunsOn configurations).
- VM disk: 4.8k–106k QD1 reads and 749–31k synced writes per second across 44 runners from 7 providers (6 Avrea runners, 6 Blacksmith runners, 3 boat x64 runners, 3 GitHub runners, 12 Namespace runners, 9 Ubicloud runners and 5 Warpbuild runners).
- local NVMe: 11k–14k QD1 reads and 5.6k–17k synced writes per second across 9 runners (9 RunsOn configurations).
- network disk: 6.9k–7.3k QD1 reads and 3.5k–4.2k synced writes per second across 2 runners (2 GitHub runners).
- EBS: 1.5k–1.8k QD1 reads and 486–571 synced writes per second across 31 runners from 3 providers (3 AWS CodeBuild x64 runners, 25 RunsOn configurations and 3 Warpbuild configurations).
Reading these numbers
fio measures what a disk can do. Whether a job gets there depends on the job.
What each column says about a job
- QD1 reads and synced writes: databases, SQLite-backed tests and anything that flushes each write. Every flush waits for the storage, and a network disk pays a round trip each time.
-
Random writes at depth and sequential throughput:
docker buildlayers, package installs and cache archives. They batch their writes and run into the disk's ceiling, not its latency. - None of them: compile jobs. A compiler waits for the CPU, and it reads its files back from the page cache.
Every runner
Each provider's best disk first; its other machines sit behind +N more. Bars are log scale.
| One request at a time | 4k, 32 in flight | Sequential, 1M | |||||
|---|---|---|---|---|---|---|---|
| Runner | QD1 read ↑ | p99 ↓ | Sync write ↑ | Read ↑ | Write ↑ | Read ↑ | Write ↑ |
| x64 | |||||||
| Namespace 8x16 | 10 µs | 931,530 | 546,288 | 4.5 GiB/s | 2.9 GiB/s | ||
| 8x32 | 10 µs | 1,095,705 | 569,755 | 5.0 GiB/s | 3.4 GiB/s | ||
| 4x16 | 11 µs | 1,202,685 | 564,373 | 5.0 GiB/s | 3.4 GiB/s | ||
| 4x8 | 10 µs | 1,209,187 | 607,540 | 5.0 GiB/s | 3.4 GiB/s | ||
| 2x8 | 10 µs | 820,729 | 574,093 | 4.5 GiB/s | 3.0 GiB/s | ||
| Blacksmith | 17 µs | 703,229 | 166,801 | 5.5 GiB/s | 2.0 GiB/s | ||
| 8 vCPU | 17 µs | 511,769 | 231,934 | 6.4 GiB/s | 1.9 GiB/s | ||
| 4 vCPU | 16 µs | 728,035 | 218,915 | 7.0 GiB/s | 1.7 GiB/s | ||
| Avrea | 14 µs | 564,218 | 245,269 | 16.0 GiB/s | 10.6 GiB/s | ||
| 8 vCPU | 18 µs | 343,658 | 200,865 | 16.8 GiB/s | 11.2 GiB/s | ||
| 4 vCPU | 21 µs | 453,992 | 236,446 | 15.9 GiB/s | 11.0 GiB/s | ||
| Warpbuild | 37 µs | 199,592 | 90,570 | 4.2 GiB/s | 2.7 GiB/s | ||
| 8 vCPU | 27 µs | 189,921 | 75,793 | 4.4 GiB/s | 3.1 GiB/s | ||
| 4 vCPU | 31 µs | 198,918 | 75,037 | 4.2 GiB/s | 2.9 GiB/s | ||
| RunsOn i7i.large | 289 µs | 75,023 | 41,231 | 503 MiB/s | 395 MiB/s | ||
| m8azn.3xlarge | 811 µs | 2,997 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| c8a.2xlarge | 897 µs | 2,995 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| m8a.2xlarge | <1 µs | 4,238,812 | 3,808,170 | 18.6 GiB/s | 9.9 GiB/s | ||
| m8a.2xlarge | 795 µs | 15,995 | 15,992 | 1,001 MiB/s | 1,001 MiB/s | ||
| r8a.2xlarge | <1 µs | 4,310,049 | 3,831,509 | 18.4 GiB/s | 10.0 GiB/s | ||
| m8a.2xlarge | 963 µs | 2,995 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| m8azn.xlarge | 918 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8azn.xlarge | 799 µs | 15,994 | 15,991 | 1,001 MiB/s | 1,001 MiB/s | ||
| m8i.2xlarge | 995 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8id.2xlarge | 239 µs | 134,258 | 67,076 | 905 MiB/s | 432 MiB/s | ||
| m8i-flex.2xlarge | 954 µs | 2,993 | 2,995 | 401 MiB/s | 401 MiB/s | ||
| m8a.xlarge | 807 µs | 2,994 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| c8a.xlarge | 897 µs | 2,994 | 2,991 | 401 MiB/s | 401 MiB/s | ||
| i7i.2xlarge | 324 µs | 300,467 | 165,110 | 2.0 GiB/s | 1.5 GiB/s | ||
| r8a.xlarge | <1 µs | 4,201,526 | 3,800,170 | 17.8 GiB/s | 9.9 GiB/s | ||
| m8azn.large | 754 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8id.xlarge | 249 µs | 67,090 | 33,526 | 451 MiB/s | 216 MiB/s | ||
| m8i-flex.xlarge | 918 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8i.xlarge | 954 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| i7i.xlarge | 281 µs | 150,053 | 82,487 | 1,010 MiB/s | 788 MiB/s | ||
| m8a.large | 872 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r8a.large | 860 µs | 2,995 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r8a.large | <1 µs | 2,147,299 | 1,930,574 | 18.0 GiB/s | 10.0 GiB/s | ||
| c8a.large | 754 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m8id.large | 167 µs | 33,541 | 16,753 | 226 MiB/s | 108 MiB/s | ||
| m8i.large | 987 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m8i-flex.large | 881 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| t8i.medium | 827 µs | 3,001 | 2,998 | 401 MiB/s | 401 MiB/s | ||
| boat | 218 µs | 460,252 | 120,989 | 10.2 GiB/s | 832 MiB/s | ||
| 8 vCPU | 545 µs | 309,424 | 85,110 | 8.1 GiB/s | 1,020 MiB/s | ||
| 4 vCPU | 297 µs | 385,820 | 154,383 | 7.8 GiB/s | 1.3 GiB/s | ||
| Ubicloud premium | 515 µs | 289,856 | 19,284 | 2.9 GiB/s | 993 MiB/s | ||
| standard 8 vCPU | 587 µs | 258,356 | 20,072 | 2.6 GiB/s | 989 MiB/s | ||
| standard 4 vCPU | 713 µs | 176,445 | 24,130 | 2.4 GiB/s | 620 MiB/s | ||
| premium 4 vCPU | 860 µs | 224,242 | 23,696 | 2.3 GiB/s | 902 MiB/s | ||
| standard 2 vCPU | 763 µs | 128,623 | 23,218 | 2.4 GiB/s | 710 MiB/s | ||
| premium 2 vCPU | 1,733 µs | 116,749 | 23,514 | 2.5 GiB/s | 832 MiB/s | ||
| GitHub | 231 µs | 38,976 | 18,540 | 784 MiB/s | 540 MiB/s | ||
| 4-core | 241 µs | 19,575 | 11,216 | 394 MiB/s | 244 MiB/s | ||
| 2-core | 262 µs | 9,377 | 8,367 | 199 MiB/s | 166 MiB/s | ||
| AWS CodeBuild medium | 786 µs | 2,992 | 2,974 | 241 MiB/s | 251 MiB/s | ||
| large | 741 µs | 2,989 | 2,977 | 243 MiB/s | 251 MiB/s | ||
| small | 803 µs | 2,988 | 2,972 | 251 MiB/s | 251 MiB/s | ||
| arm64 | |||||||
| Blacksmith | 37 µs | 253,175 | 35,622 | 1.7 GiB/s | 829 MiB/s | ||
| 8 vCPU | 36 µs | 251,923 | 32,466 | 1.7 GiB/s | 837 MiB/s | ||
| 2 vCPU | 40 µs | 250,107 | 32,701 | 1.7 GiB/s | 832 MiB/s | ||
| Warpbuild xfast (Apple M4 Pro) | 33 µs | 383,964 | 5,773 | 11.0 GiB/s | 11.6 GiB/s | ||
| xfast (Apple M4 Pro) 6 vCPU | 48 µs | 362,075 | 5,561 | 12.2 GiB/s | 10.8 GiB/s | ||
| 8 vCPU | 823 µs | 4,995 | 4,993 | 401 MiB/s | 401 MiB/s | ||
| 4 vCPU | 856 µs | 4,194 | 4,193 | 302 MiB/s | 302 MiB/s | ||
| 2 vCPU | 819 µs | 4,194 | 4,194 | 302 MiB/s | 301 MiB/s | ||
| Avrea | 60 µs | 152,164 | 17,974 | 31.1 GiB/s | 14.7 GiB/s | ||
| 8 vCPU | 56 µs | 143,512 | 14,241 | 40.1 GiB/s | 16.0 GiB/s | ||
| 4 vCPU | 59 µs | 147,857 | 16,821 | 36.8 GiB/s | 15.6 GiB/s | ||
| Namespace Apple silicon 2x8 | 78 µs | 153,165 | 15,192 | 31.2 GiB/s | 11.8 GiB/s | ||
| Apple silicon 4x16 | 97 µs | 145,293 | 23,426 | 32.6 GiB/s | 12.7 GiB/s | ||
| 8x32 | 90 µs | 183,882 | 50,801 | 919 MiB/s | 772 MiB/s | ||
| 8x16 | 94 µs | 183,210 | 39,558 | 923 MiB/s | 695 MiB/s | ||
| 4x8 | 92 µs | 142,916 | 37,163 | 930 MiB/s | 650 MiB/s | ||
| 4x16 | 94 µs | 147,080 | 34,602 | 932 MiB/s | 680 MiB/s | ||
| 2x8 | 103 µs | 75,288 | 25,671 | 924 MiB/s | 663 MiB/s | ||
| RunsOn m9gd.large | 81 µs | 43,609 | 21,784 | 293 MiB/s | 141 MiB/s | ||
| m9gd.2xlarge | 83 µs | 174,573 | 87,209 | 1.1 GiB/s | 561 MiB/s | ||
| m9g.2xlarge | 901 µs | 2,995 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| m9g.2xlarge | <1 µs | 2,615,598 | 2,487,852 | 13.0 GiB/s | 9.1 GiB/s | ||
| c8g.2xlarge | 856 µs | 2,994 | 2,993 | 401 MiB/s | 401 MiB/s | ||
| m9gd.xlarge | 111 µs | 87,221 | 43,591 | 585 MiB/s | 281 MiB/s | ||
| m9g.xlarge | 868 µs | 2,994 | 2,992 | 401 MiB/s | 401 MiB/s | ||
| r9g.xlarge | <1 µs | 2,605,268 | 2,485,222 | 14.0 GiB/s | 8.9 GiB/s | ||
| m9g.xlarge | <1 µs | 2,686,147 | 2,522,480 | 14.7 GiB/s | 9.3 GiB/s | ||
| c8g.xlarge | 860 µs | 2,992 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| m9g.large | 905 µs | 2,993 | 2,994 | 401 MiB/s | 401 MiB/s | ||
| r9g.large | <1 µs | 1,277,889 | 1,234,447 | 13.8 GiB/s | 9.2 GiB/s | ||
| c8g.large | 836 µs | 2,993 | 2,993 | 401 MiB/s | 401 MiB/s | ||
| Ubicloud standard | 389 µs | 234,460 | 77,711 | 755 MiB/s | 509 MiB/s | ||
| standard 4 vCPU | 420 µs | 147,368 | 34,056 | 738 MiB/s | 379 MiB/s | ||
| standard 2 vCPU | 510 µs | 98,852 | 28,872 | 761 MiB/s | 414 MiB/s | ||
| GitHub | 215 µs | 9,376 | 6,264 | 199 MiB/s | 164 MiB/s | ||
| 4-core | 225 µs | 19,575 | 12,113 | 394 MiB/s | 325 MiB/s | ||
- IOPS are 4k blocks, medians of finished jobs
- bold best disk measured on the arch (tmpfs is RAM, left out)
- , : cheaper, less durable flushes
Pick the disk per job on RunsOn
-
The instance type sets the storage under the workspace: EBS gp3 by default, provisioned gp3 for bandwidth, local NVMe
on the
dandifamilies, or tmpfs when the job fits in RAM. Pick it with thefamilylabel; each one, measured: EC2 storage benchmark. - What tmpfs does to real builds, what it costs and where it does nothing: Disks? Where we're going, we don't need disks.
How it's measured
fio runs inside the job, in GITHUB_WORKSPACE: the disk a build writes to, on the filesystem the provider
mounted there. Every runner is compared, whatever its shape.
fio settingsblock sizes, queue depths, mounts
- QD1 random read: 4k blocks, one request in flight; IOPS and p99 latency.
- Sync write: 4k writes, each followed by
fdatasync. - Random read and write: 4k blocks, 4 jobs with 32 requests in flight each.
- Sequential read and write: 1M blocks, 32 requests in flight.
- The workspace filesystem, its size and its mount options come from the job's own mount table.
- Earlier versions of this page used a different harness and other fio settings on 2 vCPU runners; those numbers are gone rather than mixed in.