Job retries and housekeeping
Jobs processed with RunsOn are regularly inspected to ensure they were able to get a runner assigned, as well as detect idle runners and remove them.
RunsOn performs the following housekeeping tasks:
-
detect idle runners (those that didn’t start a job after 10 minutes) and remove them. This is automatically handled by the RunsOn agent running on the idle runner.
-
detect jobs that didn’t get a runner assigned due to AWS server errors (happens on high load). In this case, a new runner is launched by the RunsOn server.
-
detect failure modes where the agent somehow didn’t launch on the assigned runner instances. This is detected when an instance has no job-started tag and is older than 25 minutes (
instanceDanglingTimeout). In this case, since the root cause is unknown, the runner is terminated and no new runner is launched. This could happen when AWS has network issues, or some custom image is not properly configured. So we don’t want to re-spawn instances blindly in this case. -
detect Spot interruptions, so that the job is properly shut down and the GitHub UI is updated. RunsOn automatically starts up to two failed-job reruns after a Spot interruption unless you opt out. GitHub reruns every failed job in that workflow attempt, including failed jobs that ran on on-demand instances.
-
detect runner instances exceeding the maximum runtime (
RunnerMaxRuntime, default 720 minutes, minimum 30) and forcefully terminate them regardless of job state.