apt package caching

Since v3.6.0, the apt-cache extra stores the Ubuntu and Debian packages your jobs download in your stack’s S3 cache bucket. The first job that installs a package fetches it from the archive; every later job on any runner of the stack gets it from S3. Only the packages your workflows use are stored: there is no mirror to set up or keep in sync.

Nothing changes in your workflows: apt-get keeps using the archive URLs it already has, and verifies every package as usual.

Availability#

It caches archive.ubuntu.com, security.ubuntu.com, ports.ubuntu.com, deb.debian.org and security.debian.org, the hosts the RunsOn images use. A custom image whose sources point at other mirrors, such as the EC2 regional ones (us-east-1.ec2.archive.ubuntu.com), gets no caching. It does not need the s3-cache extra, and it works the same on stacks with cache isolation.

Windows runners, and Linux images without apt such as Amazon Linux, ignore the extra without a warning, so it can be one of a stack’s mandatory extras.

Enable it#

runs-on:
  - runs-on=${{ github.run_id }}
  - runner=4cpu-linux-x64
  - extras=s3-cache+apt-cache

runners = {
  linux-build = {
    cpu    = 4
    family = ["c8i"]
    extras = ["s3-cache", "apt-cache"]
  }
}

What to expect#

On a c7a.large runner with the ubuntu24-full-x64 image, after another job installed the same packages:

The gain is largest for installs of many small packages, where each download waits on a round trip to the archive. apt-get update takes about as long either way: decompressing the package indexes, not downloading them, is most of its time. Unpacking and configuring the packages is unchanged, and for large installs it takes much longer than the download.

On a cache miss, the download takes about as long as without the extra, up to a second longer for large installs, and stores what it fetched for the next jobs.

Private subnets and NAT gateways#

When runners use private networking, every download from the archive goes through the NAT gateway, which charges for each GB it processes. Cache traffic to S3 does not, as long as the VPC has an S3 gateway endpoint. The Flex CloudFormation stack creates one; with Terraform, which runs in your VPC, add one if it has none, or S3 traffic goes through the NAT gateway too.

With the cache on, a package or an index crosses the NAT gateway once per stack, however many jobs fetch it. Only the small signed InRelease files cross it on every apt-get update: about 0.6 MB, against about 33 MB of indexes for Ubuntu 24.04.

When the archive is down#

If the archive fails, or does not answer within 8 seconds, the runner serves the InRelease files it stored last, and apt keeps installing the indexes and packages the cache holds. Packages no job of the stack fetched before still need the archive.

A stored InRelease is only used if it was signed in the last 7 days for the suites that receive updates, such as noble-updates and noble-security, or in the last 180 days for Debian’s base suites, such as trixie, which change at each point release. Ubuntu release suites, such as noble, never change once released.

How it works#

At boot, before preinstall, the runner points apt at a proxy in the RunsOn agent, for the archive hosts above only:

• It adds a proxy entry for each host in /etc/apt/apt.conf.d/99runs-on-apt-cache.
• It switches those hosts’ sources from https:// to http://, so that the proxy sees apt’s requests. The agent itself fetches from the archive over HTTPS. A runner booted without the extra gets the image’s original sources back.

The proxy stores the package files (pool/) and the indexes apt fetches by hash (by-hash/). InRelease, the signed file that lists every index’s hash, comes from the archive; the runner also stores it, for when the archive is down.

Each file is stored with the SHA-256 apt will check as its S3 checksum: the one in an index’s by-hash path, or the one apt’s package lists give a package. S3 refuses to store content that does not match the checksum it is uploaded with, and the runner serves a stored file only when its checksum is the one apt expects.

What is cached#

• Packages and indexes from the archive hosts above, in preinstall and prerun scripts and in job steps.
• Everything else goes to its source unchanged: other repositories (PPAs, Docker, Microsoft, and so on), and sources you add with https:// URLs for the archive hosts.
• Jobs in a container: and Docker builds run their own apt, with their own configuration, and go straight to the archive.

Security#

• The cache cannot change what apt installs. apt checks the signature of InRelease, and checks every index and package against the hashes it lists. A corrupted or tampered entry makes the download fail instead of installing different content.
• Old package lists are refused. During an archive outage, job code could plant an older, genuinely signed InRelease to keep runners on old packages. The runner refuses one signed more than 7 days ago for the suites that receive updates, or 180 days ago for Debian’s base suites.
• Entries are shared within the stack. They are public packages, stored under cache/pkg/apt/v1/ in the cache bucket, which every runner of the stack can read and write with its instance role. Job code could plant a bad entry, but S3 will not store it under the checksum of the genuine file, so the runner never serves it: it downloads that file from the archive instead and replaces the entry.

Retention#

Entries expire with the cache bucket’s lifecycle rule (S3CacheExpirationInDays in CloudFormation, cache_expiration_days in Terraform), 10 days by default.

Troubleshooting#

• If the cache cannot start, the job continues with a warning annotation, and apt downloads go directly to the archive.
• The cache stays off when the image already sends apt through a proxy (an Acquire::http::Proxy or Acquire::https::Proxy setting, or a proxy variable in /etc/environment).
• Packages come from the archive, never from the cache, when apt keeps its package lists compressed in a format the runner does not read, such as lz4 with Acquire::GzipIndexes. The unverifiable counter counts them.
• curl -s http://127.0.0.1:6871/_runs-on/apt-cache/stats on the runner prints the cache’s counters: hits, misses, bytes served, entries stored, entries replaced because they did not match, and fallbacks, the InRelease files served while the archive was down.