Five minutes in which my CI scheduled jobs onto dead machines

TL;DR

TL;DR: GitLab's docker-autoscaler learns which workers exist through the fleeting plugin, from the instance list of the Auto Scaling Group. When AWS takes back a spot instance, the group keeps listing it as InService for about five minutes, until its own health check notices. In that time the runner hands new jobs to a machine that no longer exists, and each one fails with instance no longer running. A shorter update_interval does not help; it just reads the same stale list more often. What helped was a small Lambda that takes the instance out of the group as soon as the interruption warning arrives. Since then no job has failed this way.

The setup

My GitLab runners use the docker-autoscaler executor. A small manager accepts the jobs and hands them to workers. The workers are spot instances in several Auto Scaling Groups, one per pool. In between sits the fleeting plugin for AWS: it scales the groups and regularly reads which instances are running in them. How often is set by update_interval.

Spot is cheap for CI, but in return AWS can take an instance back at any time. About two minutes before that, a notice arrives: the Spot Instance Interruption Warning. A job running at that moment is lost, and I had expected that. What I had not expected was that jobs which had not even started yet would fail afterwards too.

What happened

The manager's journal held errors like this one, often dozens in a row:

ERROR: Preparation failed: creating docker connection: creating docker tunnel:
preparing environment: getting instance connect info: … instance no longer running

GitLab reports this as runner_system_failure and restarts the job up to twice. But the retry comes only seconds later, so it often lands on the same dead instance. By the time the pipeline turned red, a job had three failed attempts behind it. Counted by job rather than by error line, three consecutive days looked like this:

I followed one of these interruptions minute by minute. Three workers got the notice at 12:59 and were terminated at 13:01. Only at 13:06 did the Auto Scaling Group take them off its list. To the plugin they were ordinary workers for seven minutes, and for five of those they were already dead.

Why the obvious fixes do not help

My first thought was update_interval. The default is one minute, but one of my pools had long been running at five seconds. That very pool still lost 196 jobs on the worst day. The plugin asks the group more often, but gets the same answer every time. As long as the group lists the instance as InService, it counts as a healthy worker.

Next, an update of the plugin seemed obvious. Version 1.3.0 clears out instances faster once they have disappeared from the group. But that only helps after the health check, not in the minutes before it. The plugin does have its own spot check: before every assignment it asks whether the instance is still running and whether an interruption notice is pending. gitlab-runner, however, only calls it when the feature flag FF_USE_FLEETING_ACQUIRE_HEARTBEATS is set, and that is off by default. I only found this in the source code while writing this post.

On-demand instances would have solved the problem, but they were out of the question because of the cost. So the gap is not in the runner. It lies between the notice from EC2 and the health check of the Auto Scaling Group.

The fix: take the instance out of the group at the notice

EventBridge delivers the Spot Instance Interruption Warning as an event, at no extra cost. A rule forwards it to a Lambda, which checks whether the instance belongs to one of my runner groups and is InService there, and then detaches it from the group:

row = client.describe_auto_scaling_instances(
    InstanceIds=[instance_id])["AutoScalingInstances"][0]
if row["AutoScalingGroupName"] in runner_groups \
        and row["LifecycleState"] == "InService":
    client.detach_instances(
        InstanceIds=[instance_id],
        AutoScalingGroupName=row["AutoScalingGroupName"],
        ShouldDecrementDesiredCapacity=False,
    )

Two details make the solution work. ShouldDecrementDesiredCapacity=False leaves the desired size of the group unchanged, so it starts a replacement at once rather than minutes later. And the check for InService leaves alone every instance the plugin is retiring itself, because those are already on Terminating.

If detaching fails, for instance because the group was faster, the Lambda sets the instance to Unhealthy instead. That takes it off the list as well. A second rule reacts when an instance changes to shutting-down or terminated, and so catches the cases in which no notice came.

The rules themselves are short:

{ "source": ["aws.ec2"],
  "detail-type": ["EC2 Spot Instance Interruption Warning"] }

{ "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": { "state": ["shutting-down", "terminated"] } }

The Lambda's role may call DescribeAutoScalingInstances on all resources, because that action cannot be scoped. DetachInstances and SetInstanceHealth it may call only on the runner groups. On top of that, every pool runs with update_interval = "5s", so the plugin stops seeing the detached instance a few seconds later.

Pitfalls during the switch

gitlab-runner did read the new update_interval when it reloaded config.toml, but it did not set up the autoscaler again. Only a restart of the service took effect. I sent SIGQUIT instead of a normal restart, so the runner finishes running jobs instead of aborting them. My systemd unit then waits two minutes before restarting, and for that long the CI took no new jobs.

The rule for the state change is noisy. It fires for every instance in the account, including every worker the plugin retires in the normal way. For me that means a few thousand invocations a day, almost all ending in skipped:Terminating. That is well within the free tier, but expect it in the log.

And an interruption does not only hit the job that is running. In my case it also took down a release job on a dead instance. The next step, which tells dependent projects about the new version, was skipped as a result. The pipeline was red, but nobody noticed that a release was only half done.

The result

In the first 17 hours after the switch, the Lambda took 267 workers out of their groups at the interruption warning. In the same period not a single job failed with instance no longer running. The day before, the same time window had 415.

The cost is negligible: AWS events on the default EventBridge bus are free, the Lambda's invocations stay within the free tier, and I keep the log for 14 days. Jobs that are running when AWS takes the instance back are still lost. What is saved is everything that would have been handed to the instance afterwards.

Recommendations

1. Count by job, not by error line. One job produces up to three attempts and each attempt several error lines. Only the number of jobs shows how much work is really lost.
2. Distrust automatic retries on spot. Retries help against independent failures. A stale instance list makes the failures dependent, and every attempt runs into the same gap.
3. Look at the source of the instance list. When a scheduler reads from an Auto Scaling Group, its health check is the slowest part of the chain. Faster polling changes nothing about that.
4. Use the notice AWS sends anyway. Two minutes of warning are enough to take an instance out of the group before it disappears.
5. Check follow-up jobs after system failures. Otherwise a skipped step after a failed release only stands out when someone misses the new version.

Conclusion

Spot interruptions are priced in for CI. What cost me was not the interruption itself but the five minutes after it, in which everything new went to a machine that no longer existed. The fix is about a hundred lines of Python and two rules. If your runners are on spot and you now and then see jobs that end in a system failure after a few seconds, this gap is worth a look.