GitHub Actions Providers Benchmark
This page offers a detailed comparison of CPU speeds and queuing times for various providers such as RunsOn, GitHub, Warpbuild, Buildjet, Blacksmith, and Ubicloud. The aim is to assist in identifying the best GitHub Action provider for your projects, based on your specific needs.
Key metrics such as the processor model, single-thread CPU speed, queue time, pricing, and the underlying infra provider are compared. The CPU single-threaded rating is a crucial metric as it is the most significant factor (unless your job is massively parallel) in accelerating any of your workflows.
It’s important to note that x64 and arm64 CPU speed numbers are not directly comparable. By exploring the tables below, you can gain a better understanding of the performance of various GitHub Action runner providers and make an informed decision for your projects.
Note: for RunsOn, you can view the on-demand, spot pricing, and spot interruption percentages, on the AWS EC2 Instances Benchmark page.
Results
x64 runners
| Provider | Processor | Samples |
CPU speed
(avg | p95) |
Queue time (s)
(avg | p95) | Infra | Pricing |
|---|---|---|---|---|---|---|
Namespace
| AMD EPYC (x86_64) | 13 | 3852 | 4027 | 21 | 23 | Hetzner Online | 2x cheaper |
Blacksmith
| AMD EPYC (x86_64) | 39 | 3404 | 3652 | 19 | 36 | Hetzner Online | 2x cheaper |
RunsOn
| Intel Xeon Platinum 8488C (x86_64) | 39 | 2834 | 2984 | 32 | 49 | Amazon.com | 10x cheaper |
RunsOn
| Intel Xeon Gold 6455B (x86_64) | 38 | 3152 | 3158 | 31 | 43 | Amazon.com | 10x cheaper |
RunsOn
| AMD EPYC 9R14 (x86_64) | 39 | 2871 | 2904 | 31 | 40 | Amazon.com | 10x cheaper |
Buildjet
| AMD Ryzen 9 5950X 16-Core Processor (x86_64) | 55 | 2802 | 2995 | 30 | 34 | Hetzner Online | 2x cheaper |
Buildjet
| AMD Ryzen 9 7950X3D 16-Core Processor (x86_64) | 37 | 3865 | 4151 | 28 | 33 | Hetzner Online GmbH | 2x cheaper |
Warpbuild
| AMD EPYC 9B14 (x86_64) | 39 | 2698 | 2704 | 18 | 53 | Google Cloud | 2x cheaper |
Warpbuild
| AMD EPYC 7B13 (x86_64) | 53 | 2375 | 2385 | 20 | 47 | Google Cloud | 2x cheaper |
GitHub
| AMD EPYC 7763 64-Core Processor (x86_64) | 78 | 2350 | 2404 | 11 | 14 | Microsoft Azure | - |
Ubicloud
| AMD EPYC 7502P 32-Core Processor (x86_64) | 62 | 2006 | 2030 | 45 | 64 | Hetzner Online | 10x cheaper |
Ubicloud
| AMD EPYC 9454P 48-Core Processor (x86_64) | 28 | 2606 | 2972 | 40 | 60 | Hetzner Online | 10x cheaper |
arm64 runners
| Provider | Processor | Samples |
CPU speed
(avg | p95) |
Queue time (s)
(avg | p95) | Infra | Pricing |
|---|---|---|---|---|---|---|
RunsOn
| (aarch64) | 31 | 1521 | 1552 | 31 | 42 | Amazon.com | 10x cheaper |
Buildjet
| Neoverse-N1 (aarch64) | 32 | 1320 | 1325 | 35 | 42 | Hetzner Online | 2x cheaper |
Ubicloud
| Neoverse-N1 (aarch64) | 31 | 1318 | 1322 | 38 | 80 | Hetzner Online | 10x cheaper |
Namespace
| Neoverse-N1 (aarch64) | 10 | 1317 | 1320 | 24 | 26 | Hetzner Online | 2x cheaper |
Warpbuild
| Neoverse-N1 (aarch64) | 29 | 1312 | 1316 | 515 | 92 | Google Cloud | 2x cheaper |
Analysis
The best provider for Linux workflows will heavily depend on your primary requirement:
-
Fastest machines: Buildjet and Blacksmith (x64 only) are good choices. Be aware that they are hosted at Hetzner, with very variable performance for the network speeds. Buildjet also doesn’t guarantee you to end up on one of their fastest runners.
-
Cheapest pricing: RunsOn or Ubicloud are the best options. Be aware that Ubicloud has low CPU speeds and high queue times, which means they won’t necessarily end up being cheaper than competitors.
-
Low queue times: GitHub, Warpbuild, Blacksmith are good options, although Warpbuild has issues with ARM64 queuing times recently.
-
All rounder: RunsOn is the best option, due to top-of-the-line CPU speeds (especially ARM64), cheapest pricing, stable queue times across x64 and arm64, and the ability to deploy on-premise.
Note: this benchmark is published by RunsOn, but I try my best to provide a fair comparison with the details available. Note that Depot.dev rejected my account creation, due to RunsOn being a competitor, and as such I cannot publish accurate benchmarks for them.
When to use GitHub Action official runners
The adage that says that GitHub Action is where older Azure hardware gets to die seems to be true, with the default Ubuntu runners having abysmal single-speed CPU performance. They are not without their use though, because they boot fast, and are ideal for short-lived jobs or those that do not require fast CPUs.
As a user of RunsOn, we always advise to keep using official GitHub runners for short-lived jobs, unless you have so many that the bill starts to be too high. This is because RunsOn runners currently take around 30s to start processing workflows, vs 5-20s for GitHub-hosted Ubuntu runners.
We strongly discourage the use of the larger GitHub runners though, because they can take up to a few minutes to boot, are not very fast, and cost $$$.
About those benchmarks
Benchmarks are performed using the Passmark benchmarking tool ↗, using the CPU Single Threaded metric.
The table displays the last 30 days of data, before the last updated date.
Last updated: Mon Apr 29 2024.