Skip to content

aws

5 posts with the tag “aws”

πŸš€ v2.8.2 is out, with EFS, Ephemeral Registry support, and YOLO mode (tmpfs)!

Check out the new documentation pages for:

Now for the full release notes:

πŸ“ v2.8.2

Support for EFS, TMPFS, and ECR ephemeral registry for fast docker builds. Also some bug fixes.

What's changed

EFS

  • Embedded networking stack can now create an Elastic File System (EFS), and runners will auto-mount it at /mnt/efs if the extras label include efs. Useful to share artefacts across job runs, with classic filesystem primitives.
jobs:
  with-efs:
    runs-on: runs-on=${{ github.run_id }},runner=2cpu-linux-x64,extras=efs
    steps:
      - run: df -ah /mnt/efs
      # 127.0.0.1:/      8.0E   35G  8.0E   1% /mnt/efs
πŸ“ Example use case for maintaining mirrors For instance this can be used to maintain local mirrors of very large github repositories and avoid long checkout times for every job:
env:
  MIRRORS: "https://github.com/PostHog/posthog.git"
  # can be ${{ github.ref }} if same repo as the workflow
  REF: main

jobs:
  with-efs:
    runs-on: runs-on=${{ github.run_id }},runner=2cpu-linux-x64,extras=efs
    steps:
      - name: Setup / Refresh mirrors
        run: |
          for MIRROR in ${{ env.MIRRORS }}; do
            full_repo_name=$(echo $MIRROR | cut -d/ -f4-)
            MIRROR_DIR=/mnt/efs/mirrors/$full_repo_name
            mkdir -p "$(dirname $MIRROR_DIR)"
            test -d "${MIRROR_DIR}" || git clone --mirror ${MIRROR/https:\/\//https:\/\/x-access-token:${{ secrets.GITHUB_TOKEN }}@} "${MIRROR_DIR}"
            ( cd "$MIRROR_DIR" && \
              git remote set-url origin ${MIRROR/https:\/\//https:\/\/x-access-token:${{ secrets.GITHUB_TOKEN }}@} && \
              git fetch origin ${{ env.REF }} )
          done
      - name: Checkout from mirror
        run: |
          git clone file:///mnt/efs/mirrors/PostHog/posthog.git --branch ${{ env.REF }} --single-branch --depth 1 upstream

Ephemeral registry

  • Support for an Ephemeral ECR registry: can now automatically create an ECR repository that can act as an ephemeral registry for pulling/pushing images and cache layers from your runners. Especially useful with the type=registry buildkit cache instruction. If the extras label includes ecr-cache, the runners will automatically setup docker credentials for that registry at the start of the job.
jobs:
  ecr-cache:
    runs-on: runs-on=${{ github.run_id }},runner=2cpu-linux-x64,extras=ecr-cache
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v4
        env:
          TAG: ${{ env.RUNS_ON_ECR_CACHE }}:my-app-latest
        with:
          context: .
          push: true
          tags: ${{ env.TAG }}
          cache-from: type=registry,ref=${{ env.TAG }}
          cache-to: type=registry,ref=${{ env.TAG }} }},mode=max,compression=zstd,compression-level=22

Tmpfs

Support for setting up a tmpfs volume (size: 100% of available RAM, so only to be used on high-memory instances), and binding the /tmp, /home/runner, and /var/lib/docker folders on it. /tmp and /home/runner are mounted as overlays, preserving their existing content.

Can speed up some IO-intensive workflows. Note that if tmpfs is active, instances with ephemeral disks won't have those mounted since it would conflict with the tmpfs volume.

jobs:
  with-tmpfs:
    runs-on: runs-on=${{ github.run_id }},family=r7,ram=16,extras=tmpfs
    steps:
      - run: df -ah /mnt/tmpfs
      # tmpfs            16G  724K   16G   1% /mnt/tmpfs
      - run: df -ah /home/runner
      # overlay          16G  724K   16G   1% /home/runner
      - run: df -ah /tmp
      # overlay          16G  724K   16G   1% /tmp
      - run: df -ah /var/lib/docker
      # tmpfs            16G  724K   16G   1% /var/lib/docker

You can obviously combine options, i.e. extras=efs+tmpfs+ecr-cache+s3-cache is a valid label πŸ˜„

Instance-storage mounting changes

Until now, when an instance has locally attached NVMe SSDs available, they would be automatically formatted and mounted so that /var/lib/docker and /home/runner/_work directories would end up on the local disks. Since a lot of stuff (caches etc.) seem to end up within the /home/runner folder itself, the agent now uses the same strategy as for the new tmpfs mounts above (i.e. the whole /home/runner folder is mounted as an overlay on the local disk volume, as well as the /tmp folder. /var/lib/docker remains mounted as a normal filesystem on the local disk volume). Fixes #284.

Misc

  • Move all RunsOn-specific config files into /runs-on folder on Linux. More coherent with Windows (C:\runs-on), and avoids polluting /opt folder.
  • Fix app_version in logs (was previously empty string due to incorrect env variable being used in v2.8.1).
  • Fix "Require any Amazon EC2 launch template not to auto-assign public IP addresses to network interfaces" from AWS Control Tower. When the Private mode is set to only, no longer enable public ip auto-assignment in the launch templates. Thanks @temap!

Upgrade

Info card

v2.6.5 - Optimized GPU images, VpcEndpoint stack parameter, tags for custom runners

πŸ‘‹ v2.6.4 and v2.6.5 have been released in the last weeks, with the following changes.

Note: v2.6.6 β†— has been released to fix an issue with the VpcEndpoints stack parameter.

πŸ“ v2.6.5

Optimized GPU images, new VpcEndpoints stack parameter, ability to specify custom instance tags for custom runners.

Note: there appears to be some issues with the new VPC endpoints. I'm on it! If you need that feature, please hold on to your current version of RunsOn.

What's Changed

  • New GPU images ubuntu22-gpu-x64 and ubuntu24-gpu-x64: 1-1 compatibility with GitHub base images + NVidia GPU drivers, CUDA toolkit, and container toolkit.
  • Add new VpcEndpoints stack parameter (fixes #213), and reorganize template params. Note that the EC2 VPC endpoint was previously automatically created when Private mode was enabled. This is no longer the case, so make sure you select the VPC endpoints that you need when you update your CloudFormation stack.
  • Suspend versioning for cache bucket (fixes #191).
  • Allow to specify instance tags for runners (fixes #205). Tag keys can't start with runs-on- prefix, and key and values will be sanitized according to AWS rules.

Info card

πŸ“ v2.6.4

CLI 0.0.1 released, fix for Magic Cache, fleet objects deletion.

What's changed

  • CLI released: https://github.com/runs-on/cli. Allows to easily view logs (both server logs and cloud-init logs) for a workflow job by just pasting its GitHub URL or ID. Also allows easy connection to a runner through SSM.
  • Fix race-condition in Magic Cache (fixes #209).
  • Delete the fleet instead of just the instance (fixes #217).

Upgrade

Info card

How to verify that VPC traffic to S3 is going through your S3 gateway?

Gateway endpoints for Amazon S3 are a must-have whenever your EC2 instances send and receive traffic from S3, because they allow the traffic to stay within the AWS network, hence better security, bandwidth, throughput, and costs. They can easily be created, and added to your VPC route tables.

But how do you verify that traffic is indeed going through the S3 gateway, and not crossing the outer internet?

Using traceroute, you can probe the routes and see whether you are directly hitting the S3 servers (i.e. no intermediate gateway). In this example, the instance is running from a VPC located in us-east-1:

Terminal window
$ traceroute -n -T -p 443 s3.us-east-1.amazonaws.com
traceroute to s3.us-east-1.amazonaws.com (52.216.215.72), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 52.216.215.72 0.890 ms 0.916 ms 0.892 ms
Terminal window
$ traceroute -n -T -p 443 s3.amazonaws.com
traceroute to s3.amazonaws.com (52.217.139.232), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 52.217.139.232 0.268 ms 0.275 ms 0.252 ms

Both outputs produce the expected result, i.e. no intermediary gateway. This is what would happen if you were accessing a bucket located in the us-east-1 region.

Let’s see what happens if we try to access an S3 endpoint located in another zone:

Terminal window
$ traceroute -n -T -p 443 s3.eu-west-1.amazonaws.com
traceroute to s3.eu-west-1.amazonaws.com (52.218.25.211), 30 hops max, 60 byte packets
1 * * *
2 240.4.88.37 0.275 ms 240.0.52.64 0.265 ms 240.4.88.39 0.215 ms
3 240.4.88.49 0.205 ms 240.4.88.53 0.231 ms 240.4.88.51 0.206 ms
4 100.100.8.118 1.369 ms 100.100.6.96 0.648 ms 240.0.52.57 0.233 ms
5 240.0.228.5 0.326 ms * *
6 240.0.32.16 0.371 ms 240.0.48.30 0.362 ms *
7 * 240.0.228.31 0.251 ms *
8 * * *
9 * * 240.0.32.27 0.392 ms
10 * * *
11 * 242.0.154.49 1.321 ms *
12 * * 52.93.28.131 1.491 ms
13 * * 100.100.6.108 1.286 ms
14 100.92.212.7 67.909 ms 52.218.25.211 67.356 ms 67.929 ms

As you can see, the route is completely different, and as expected does not hit straight to the S3 endpoint.

TL;DR: make sure your route tables are correct, and only point to S3 buckets located in the same region.

GitHub Action runner images (AMI) for AWS EC2

As part of the RunsOn service, we automatically maintain and publish replicas of the official GitHub runner images β†— images as AWS-formatted images (AMIs) in this repository: https://github.com/runs-on/runner-images-for-aws β†—.

New images are automatically released every 2 weeks, and are slightly trimmed to remove outdated software, or (mostly useless) caches.

Supported images

  • ubuntu22-full-x64
  • ubuntu22-full-arm64
  • ubuntu24-full-x64
  • ubuntu24-full-arm64

Supported regions

  • North Virginia (us-east-1)
  • Ohio (us-east-2)
  • Oregon (us-west-2)
  • Ireland (eu-west-1)
  • London (eu-west-2)
  • Paris (eu-west-3)
  • Frankfurt (eu-central-1)
  • Mumbai (ap-south-1)
  • Tokyo (ap-northeast-1)
  • Singapore (ap-southeast-1)
  • Sydney (ap-southeast-2)

Find the AMI

For the x86_64 image, search for:

  • name: runs-on-v2.2-<IMAGE_ID>-*
  • owner: 135269210855

For instance, for the ubuntu22-full-x64 image, search for:

  • name: runs-on-v2.2-ubuntu22-full-x64-*
  • owner: 135269210855

Notes

  • SSH daemon is disabled by default, so be sure to enable it in a user-data script if needed.

You can find more details on https://github.com/runs-on/runner-images-for-aws β†—.

Automatically cleanup outdated AMIs in all AWS regions

Here is a script you can use to automatically cleanup AMIs older than 60 days (configurable), while keeping the 2 most recent AMIs in each region. This helps to remove outdated images, as well as reducing storage costs for your AMIs.

Particularly useful in the case of runs-on.com, where we regularly rebuild base images whenever GitHub releases a new version of the image runner.

The bin/cleanup script (simply adjust the filters as needed):

#!/bin/bash
# Deregisters old AMIs and deletes associated snapshots, in all regions
set -e
set -o pipefail
APPLICATION="RunsOn"
REGIONS="$(aws ec2 describe-regions --query "Regions[].RegionName" --output text)"
# Number of days to keep AMIs
DAYS_TO_KEEP=${DAYS_TO_KEEP:=60}
# Define the age threshold in seconds (60 days)
AGE_THRESHOLD=$((DAYS_TO_KEEP*24*3600))
# Get the current timestamp in seconds since epoch
CURRENT_TIMESTAMP=$(date +%s)
for region in ${REGIONS[@]}; do
echo "---- Region: ${region} ---"
# List all your AMIs and extract relevant information using the AWS CLI
image_count=$(aws ec2 describe-images --owner self --filters "Name=tag:application, Values=${APPLICATION}" --query 'length(Images)' --region "$region" --output text)
echo " Total AMIs in this region: ${image_count}"
if [ "$image_count" -lt 2 ]; then
echo " Less than 2 AMIs found, skipping"
continue
fi
aws ec2 describe-images --owner self --region "${region}" --filters "Name=tag:application, Values=${APPLICATION}" --query 'Images[*].[Name,ImageId,CreationDate]' --output text | \
while read -r name image_id creation_date; do
# Parse the creation date into seconds since epoch
image_timestamp=$(date -d "$creation_date" +%s)
# Calculate the age of the AMI in seconds
age=$((CURRENT_TIMESTAMP - image_timestamp))
# Check if the AMI is older than the threshold
if [ $age -gt $AGE_THRESHOLD ]; then
echo " ! Deregistering AMI: ${image_id} (${name}) created on $creation_date"
snapshot_id=$(aws ec2 describe-images --image-ids "$image_id" --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId" --region "${region}" --output text)
if [ "$DRY_RUN" = "true" ]; then
echo " DRY_RUN is set to true, skipping deregistering AMI ${image_id} and deleting snapshot ${snapshot_id}"
continue
fi
aws ec2 deregister-image --image-id "$image_id" --region "${region}"
echo " ! Deleting snapshot ${snapshot_id} for AMI ${image_id}"
aws ec2 delete-snapshot --snapshot-id "${snapshot_id}" --region "${region}"
fi
done
done

Example output:

---- Region: ap-southeast-2 ---
Total AMIs in this region: 0
---- Region: eu-central-1 ---
Total AMIs in this region: 5
Deregistering AMI: ami-0576e83d0a0f89fbe (runner-ubuntu2204-1699888130) created on 2023-11-13T16:53:48.000Z
Deleting snapshot snap-07db95a7f230d3f76 for AMI ami-0576e83d0a0f89fbe
Deregistering AMI: ami-004d4d18e6db2f812 (runner-ubuntu-22-1699873337) created on 2023-11-13T12:40:48.000Z
Deleting snapshot snap-0500b0e3fb95ab36a for AMI ami-004d4d18e6db2f812
Deregistering AMI: ami-0e6239eae649effcd (runner-ubuntu22-20231115.7-1700233930) created on 2023-11-17T17:01:58.000Z
Deleting snapshot snap-05e795e4c6fe9e66f for AMI ami-0e6239eae649effcd
Deregistering AMI: ami-0dd7f6b263a3ce28c (runner-ubuntu22-20231115-1700156105) created on 2023-11-16T19:24:38.000Z
Deleting snapshot snap-02c1aef800c429b76 for AMI ami-0dd7f6b263a3ce28c
---- Region: us-east-1 ---
Total AMIs in this region: 4
Deregistering AMI: ami-0b56f2d6af0d58ce0 (runner-ubuntu2204-1699888130) created on 2023-11-13T15:54:22.000Z
Deleting snapshot snap-0f2e8759bea8f3937 for AMI ami-0b56f2d6af0d58ce0
Deregistering AMI: ami-04266841492472a95 (runner-ubuntu22-20231115.7-1700233930) created on 2023-11-17T16:02:34.000Z
Deleting snapshot snap-0f0fcf9c6406c3ad9 for AMI ami-04266841492472a95
Deregistering AMI: ami-0738c7108915044fe (runner-ubuntu22-20231115-1700156105) created on 2023-11-16T18:21:40.000Z
Deleting snapshot snap-03f16588f59ed7cea for AMI ami-0738c7108915044fe
...

Example GitHub Action workflow file to schedule a cleanup every night:

name: Cleanup
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
on:
workflow_dispatch:
schedule:
- cron: '0 2 * * *'
jobs:
check:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- run: bin/cleanup