Sharing files across workflow jobs with EFS

Since v2.8.2, you can configure RunsOn to automatically create an Elastic File System (EFS), which can be used for storing artefacts across workflow jobs. Compared to the classic actions/cache behaviour, EFS doesn’t have any size limit, and skips the compression step, which makes it faster for storing large numbers of files.

Usage

Flex

Enable EFS from the job label and read or write shared files under /mnt/efs:

jobs:
  efs:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=efs
    steps:
      - run: echo "Hello, world!" > /mnt/efs/hello.txt
      - run: cat /mnt/efs/hello.txt
      - run: df -ah /mnt/efs

Fleet

EFS cache and shared mounts are not exposed by the Fleet module yet. Use Flex for workloads that need /mnt/efs; see Fleet current limitations.

Use cases

EFS is a good fit for e.g.:

1. Maintaining git mirrors of very large git repositories, so that your jobs don’t need to go to GitHub to fetch the code.
2. Storing frequently-accessed data files close to your runners, so that your jobs don’t need to download them from scratch.
3. Programs that need to manipulate caches but can’t use an object storage bucket. In that case, EFS is just seen as a normal filesystem by your runners.

Accessing EFS

Runners automatically get permission to access the EFS file system, using the EC2 instance profile assigned to the runners.

The RUNS_ON_EFS_ID environment variable is available in your steps, and contains the EFS file system ID (e.g. fs-12345678901234567).

If you include efs in the extras parameter of your workflow job, the EFS file system will be mounted in the /mnt/efs directory.

Note: EFS is disabled by default. To use it you must enable the EnableEfs CloudFormation parameter (or enable_efs in Terraform), which defaults to false. See EnableEfs. Enabling it provisions an encrypted EFS file system, a dedicated security group, and mount targets in the stack subnets.

Using EFS in your workflow

The EFS file system is mounted in the /mnt/efs directory. You can access it as you would any other filesystem.

jobs:
  efs:
    runs-on: runs-on=${{ github.run_id }}/runner=2cpu-linux-x64/extras=efs
    steps:
      - run: echo "Hello, world!" > /mnt/efs/hello.txt
      - run: cat /mnt/efs/hello.txt
      - run: df -ah /mnt/efs
      # 127.0.0.1:/      8.0E   35G  8.0E   1% /mnt/efs

Cleanup

There is currently no mechanism to automatically cleanup the EFS file system.

Considerations

Security-wise, the EFS file system is shared across all runners launched by the same RunsOn stack. So make sure you only store files that can be shared across all runners. You can create multiple RunsOn environments if you need better isolation.

Limitations

• EFS is only available on Linux runners.
• If you request extras=efs on a stack where EFS is not enabled, the mount is silently skipped (no error) and /mnt/efs will be absent. EFS is only mounted when the stack was deployed with the EFS feature enabled and the runner image ships amazon-efs-utils.