How Coveo halved its PR check times with RunsOn

TL;DR: In 2026, we replaced our in-house GitHub Actions runner orchestrator with RunsOn. PR checks now finish twice as fast (3.3 to 1.5 minutes). Using the Warm pools feature, runners can even start in about 6 seconds. We no longer have an orchestrator to maintain, and changing runners for hundreds of repos was easier than anticipated.

About Coveo#

Coveo builds an AI-powered search and generative experience platform for enterprises. We are serving millions of people and billions of interactions across every digital experience.

Our engineering organization is globally distributed. While most of the workforce is in the Eastern Time Zone, we have teams in the Western and Central European time zones as well as all over the world.

Our Dev Tooling team maintains an extensible CI/CD platform for the company, including the self-hosted GitHub Actions runners behind more than 100,000 workflow runs a month.

Before RunsOn#

After migrating our CI from Jenkins to GitHub Actions, we quickly identified the need for self-hosted runners. With few options available on the market at the time, we decided to build our own orchestrator. We evolved that solution over time, supporting many configurations, custom AMIs, large disks, Windows. Runners could run on either EC2 or EKS in multiple AWS accounts.

After years of hardening, it was stable, but it still cost engineering time and never reached a 100% job-serving rate.

• Slow start: a typical job used to wait about 1 minute for a Linux runner (up to 2 minutes on EC2) and about 6 minutes for a Windows one.
• Complexity: EKS was fast, but did not support running Docker containers. The burden was on the end-user to understand those nuances.
• Jobs that never ran: throttling, runners “stolen” by another job, jobs queued forever, leaked instances. At one point, we built a self-healing component, but it ended up over-spawning instances during a GitHub webhook outage.
• Hard to debug: when a runner failed to start or was killed mid-run, finding out why took our time and blocked the developer.
• No caching: efficient Docker-layer caching was lost in the move from Jenkins to GitHub Actions.
• Standing still: every feature request or fix competed with the rest of our roadmap.

The compute cost of runners was not significant. The issue was owning and evolving a critical system that wasn’t our product.

Requirements#

The replacement had to beat our in-house orchestrator on speed and reliability without giving us more infrastructure to run.

• Environments
  • AWS: run in our accounts.
  • Regions: run in either North America or Europe.
  • Windows: still needed for several years (legacy products).
  • Custom runners: high CPU and memory, large disks, GPU, etc.
• Performance
  • Fast starts: a runner in under 1 minute for everyone.
  • Reliability: runners always start; anomalies are detected and self-heal quickly.
• Usability
  • End-users just set the runs-on attribute in their workflows.
  • Operators can change defaults (e.g. machine size) without a PR in every repo.
• Maintainability
  • Low maintenance: easy to deploy, upgrade and extend.
  • Troubleshooting: a clear trace of each runner request’s lifecycle.
  • Cost: total cost (orchestrator, license, compute) in line with what we were already spending.
• Security
  • Ephemeral runners: one job per machine, for isolation.
  • StepSecurity: integrated in every runner.

Why RunsOn#

In 2025, we compared four alternatives to our in-house orchestrator and chose RunsOn: it met every requirement, ran in our AWS accounts, and handed orchestrator maintenance to a vendor for a reasonable license fee. We made the decision in July 2025.

Option	Why not / why
GitHub-hosted runners	Fastest and zero maintenance, but based on our estimates at the time, equivalent machines would have cost several times our EC2 compute.
Keep our in-house orchestrator	Would need several more weeks of work to reach a 100% serving rate. Start times were still slower than we’d like. It would keep evolving only on our time.
actions-runner-controller (ARC)	Fast and resilient in a 2024 proof of concept, but meant running a Kubernetes cluster, predefined runner sets instead of per-job labels, and weak Windows support. Required a lot of configuration and fine-tuning.
terraform-aws-github-runner	Evaluated in an earlier review; at the time, it did not fit our maintenance and reliability requirements.
RunsOn	Chosen. Linux runners in about 17 s in testing; per-job sizing through runs-on labels; S3/EFS caches; spot instances; environments and regions; logs in the job and in CloudWatch; StepSecurity integration; deployed as a CloudFormation stack in our AWS accounts.

Risks accepted: RunsOn was young (first commit December 2023) and backed by a very small company. We addressed vendor-continuity risk through contractual protections and technical due diligence.

The migration#

Setup was quick: RunsOn ships as a CloudFormation stack, which we wrapped in our existing Terraform tooling. Upgrading is a stack update. RunsOn has since released a Terraform module, which we adopted.

We rolled RunsOn out in waves: our team’s repositories first, then the Platform Engineering unit, then the rest of R&D. For most teams, migrating meant reviewing and merging one pull request.

How we rolled it out

• Automated PRs: a migration script opened PRs across repositories in our GitHub organization. We handled edge cases using targeted manual reviews.
• No more custom images: We ditched our manually maintained images in favor of the RunsOn-provided ones. Being based on the GitHub-provided images meant we maintained compatibility with GitHub-hosted runners.
• Security tooling: StepSecurity runs on every runner, made easy by its native integration in RunsOn’s images.
• Warm pools: pre-started instances scheduled around peak demand for the most-used runner types, including Windows.

The migration work was smooth, and the technology was flexible enough to easily debug and tackle more complex use cases.

Results#

PR checks now finish twice as fast: a typical one went from 3.3 to 1.5 minutes, so developers get feedback about 2 minutes sooner on every push. 11 of the 13 busy repositories we measured got faster, and one high-volume repository went from 8.7 to 1.8 minutes, 4.8x faster. Across our busiest repositories, that adds up to about 500 fewer hours of PR checks every month.

runs-on: Coveo · 2026

PR checks, twice as fast

In-house orchestrator RunsOn

• Typical PR check 2.2× faster In-house orchestrator: 3.3 min RunsOn: 1.5 min
• High-volume repository 4.8× faster In-house orchestrator: 8.7 min RunsOn: 1.8 min

runs-on: Coveo · 2026

Busy repositories that got faster

11 of 13 busy repositories measured

runs-on: Coveo · 2026

Typical wait for a Linux runner

• In-house orchestrator 55 s
• RunsOn, cold start ~20 s
• RunsOn, warm instance ~6 s

RunsOn figures exclude our initialization scripts, which add about 10 seconds to every job.

Warm pools are what make RunsOn feel instant: when a job lands on a warm instance, RunsOn hands it a runner in about 6 seconds.

And our team no longer spends time maintaining an orchestrator.

Improved CI speed with virtually zero friction for dev teams. A clear slam dunk! Well done!

Working with RunsOn#

RunsOn behaves like an extension of our team: questions get answered quickly, often the same day, and our requests regularly ship in the next releases.

Some of our requests that RunsOn shipped:

• StepSecurity integration: added at our request, before we were a customer.
• Organization-wide defaults: a stack-level _extends so every repo inherits our shared runner config without boilerplate.
• Cache isolation: magic cache scoped like GitHub’s own cache (per repo and branch).
• Instance selection: an ordered list of preferred instance types, so we can favor newer, more cost-effective generations.
• Images: StepSecurity variants of new images, such as Ubuntu 26, shortly after release.

Direct vendor collaboration helps us diagnose edge cases and provide precise reports.