Skip to content

Benchmarks - GHA Providers

This page offers a detailed comparison of CPU speeds and queuing times for various providers such as RunsOn, GitHub, Warpbuild, Buildjet, and Ubicloud. The aim is to assist in identifying the best GitHub Action provider for your projects, based on your specific needs.

Key metrics such as the processor model, single-thread CPU speed, queue time, and the underlying infra provider are compared. The CPU single-threaded rating is a crucial metric as it is the most significant factor (unless your job is massively parallel) in accelerating any of your workflows.

It’s important to note that x64 and arm64 CPU speed numbers are not directly comparable. By exploring the tables below, you can gain a better understanding of the performance of various GitHub Action runner providers and make an informed decision for your projects.

Note: for RunsOn, you can view the on-demand, spot pricing, and spot interruption percentages, on the Benchmarks - AWS EC2 Instances page.

Results

x64 runners

Provider Processor Samples CPU speed
(avg | p95)
Queue time (s)
(avg | p95)
Infra
RunsOn
  • runs-on,family=m7i-flex
Intel Xeon Platinum 8488C (x86_64) 20 2821 | 2926 34 | 39 Amazon.com
RunsOn
  • runs-on,family=c7a
AMD EPYC 9R14 (x86_64) 20 2843 | 2903 33 | 40 Amazon.com
RunsOn
  • runs-on,family=r7iz
Intel Xeon Gold 6455B (x86_64) 19 3152 | 3157 33 | 44 Amazon.com
Buildjet
  • buildjet-2vcpu-ubuntu-2204
  • buildjet-16vcpu-ubuntu-2204
AMD Ryzen 9 5950X 16-Core Processor (x86_64) 32 2764 | 2977 30 | 34 Hetzner Online
Buildjet
  • buildjet-16vcpu-ubuntu-2204
  • buildjet-2vcpu-ubuntu-2204
AMD Ryzen 9 7950X3D 16-Core Processor (x86_64) 21 3850 | 4046 29 | 33 Hetzner Online
Warpbuild
  • warp-ubuntu-latest-x64-16x
AMD EPYC 9B14 (x86_64) 20 2700 | 2702 18 | 44 Google Cloud
Warpbuild
  • warp-ubuntu-latest-x64-2x
AMD EPYC 7B13 (x86_64) 33 2375 | 2385 22 | 50 Google Cloud
GitHub
  • github-16cpu
  • ubuntu-latest
AMD EPYC 7763 64-Core Processor (x86_64) 40 2351 | 2403 12 | 13 Microsoft Azure
Ubicloud
  • ubicloud-standard-16
  • ubicloud-standard-2
AMD EPYC 7502P 32-Core Processor (x86_64) 38 2003 | 2030 45 | 65 Hetzner Online
Ubicloud
  • ubicloud-standard-2
  • ubicloud-standard-16
AMD EPYC 9454P 48-Core Processor (x86_64) 15 2576 | 2982 45 | 61 Hetzner Online

arm64 runners

Provider Processor Samples CPU speed
(avg | p95)
Queue time (s)
(avg | p95)
Infra
RunsOn
  • runs-on,family=m7g
(aarch64) 15 1492 | 1552 33 | 45 Amazon.com
Buildjet
  • buildjet-2vcpu-ubuntu-2204-arm
Neoverse-N1 (aarch64) 15 1318 | 1324 35 | 44 Hetzner Online
Ubicloud
  • ubicloud-standard-2-arm
Neoverse-N1 (aarch64) 15 1317 | 1322 36 | 93 Hetzner Online
Warpbuild
  • warp-ubuntu-latest-arm64-2x
Neoverse-N1 (aarch64) 13 1310 | 1314 13 | 15 Google Cloud

Analysis

RunsOn is consistently the fastest (and cheapest) GitHub Action runner provider, with the lowest variance of the bunch across CPU types. For x64, queuing times are higher than the standard 2cpu official GitHub runners since they are launched on-demand instead of from a warm pool. Larger GitHub runners have very high boot time variance though.

Buildjet does have some faster CPUs in their inventory, but at this time the assignment is very random. Irrespective of whether you ask for 2cpu or larger runners.

Warpbuild uses older generation (m6i?) AWS instances. They have consistent and fast boot times for x64 and ARM64.

Ubicloud has very variable boot times, and slow instances.

When to use GitHub Action official runners

The adage that says that GitHub Action is where older Azure hardware gets to die seems to be true, with the default Ubuntu runners having abysmal single-speed CPU performance. They are not without their use though, because they boot fast, and are ideal for short-lived jobs or those that do not require fast CPUs.

As a user of RunsOn, we always advise to keep using official GitHub runners for short-lived jobs, unless you have so many that the bill starts to be too high. This is because RunsOn runners will always take between 30s (arm64 with full image), and 30-50s (x64 with full image) to start processing workflows, vs 5-20s for classic Ubuntu runners.

We strongly discourage the use of the larger GitHub runners though, because they can take up to a few minutes to boot, are not very fast, and cost $$$.

About those benchmarks

Benchmarks are performed using the Passmark benchmarking tool, using the CPU Single Threaded metric.

The table displays the last 15 days of data, before the last updated date.

Last updated: Sun Apr 14 2024.