One tenant stops the rollout for all

01 — Tenants roll out one after another

Argo CD keeps a Kubernetes cluster in the state a Git repository describes. Every tenant on the platform is an Application there, a unit Argo CD syncs on its own. A single ApplicationSet generates these Applications. It is a template that turns every tenant file in the Git repository into an Application.

So that a faulty change does not hit all tenants at once, the ApplicationSet rolls them out with the RollingSync strategy. That means step by step, in a fixed order.

strategy:
  type: RollingSync
  rollingSync:
    steps:
      - matchExpressions:
          - key: example.org/tenant
            operator: In
            values: [tenant-a]
      - matchExpressions:
          - key: example.org/tenant
            operator: In
            values: [tenant-b]

A step starts only once the previous one is healthy. That is the point: an error should stay with one tenant instead of spreading to all of them. What it costs only shows in operation. When a step stops, everything behind it stops with it.

02 — A setup hook fails

It started with a new version of a TYPO3 extension. It shipped a settings template, an ext_conf_template.txt. Before every rollout a setup hook runs: a Job that Argo CD starts before the actual sync. It sets up schema and extensions, and in doing so TYPO3 writes every template's defaults into settings.php.

In our pods, though, the code is a read-only image volume. That is a filesystem taken from a container image, and nobody can write to it.

In ConfigurationManager.php line 387:
  config/system/settings.php is not writable.

The hook is a Job annotated argocd.argoproj.io/hook: PreSync. Argo CD runs it before it changes anything about the tenant. If it fails, the sync aborts and the running pods stay as they are. That is exactly what happened: the tenant kept running its previous version, and its visitors noticed nothing.

So far the behaviour was right. A setup that fails should not roll out half of a new version.

03 — The tenant behind it waits in silence

The failing tenant was second in line. My own site comes further back, and its new version had long been in the Git repository too. Argo CD showed it as OutOfSync, meaning not in the state Git describes. It attempted no sync, though. It reported no error either.

A look at the Application made it more puzzling at first. Automated sync was switched off there.

syncPolicy:
  automated:
    enabled: false
    prune: true
    selfHeal: true

It looks as if someone had switched the automation off. In fact it is the normal state. With RollingSync, the ApplicationSet controller writes enabled: false into every generated Application and drives the syncs itself, step by step. The true in the template has no effect. Anyone who does not know this looks in the wrong place.

The actual cause sat two steps earlier. The second tenant's Job had reached its backoff limit, the number of attempts after which Kubernetes gives up. As long as that step did not turn healthy, nothing moved on.

04 — What the alert saw, and when

There is an alert for exactly this case: ArgoAppOutOfSync fires when an Application has been out of sync for more than an hour. It did fire, for the failing tenant and for my site, one hour after the jam began.

I had noticed earlier, and by chance. I was waiting for the new version and had a script check which version runs in each pod. Two tenants showed it, two did not. Only asking why led from the Application to the Job, and from there to the error message.

The hour of delay in the alert is deliberate. A normal rollout is in sync within minutes, and an alert that fires at every short gap is soon ignored. The price is an hour in which a fix sits in Git and arrives nowhere.

05 — How the jam cleared

Nothing in the cluster needed fixing; the extension did. Version 1.1.1 came without the settings template and reads its values straight from the configuration, with defaults in code. A test fails as soon as the file comes back.

From there, the chain that delivers every other change did the rest. The dependency bot raised the version in every website, each website built a new release, and the bot entered it in the tenant files. This time the second tenant's setup hook went through, its step turned healthy, and the steps behind it started one after another. I did not touch the cluster.

One question is still open. Two tenants got through the same hook with the same faulty version without it failing. Both Jobs mount /app read-only in exactly the same way. Why TYPO3 did not try to write there, I have not worked out.

06 — What I take from it

• A PreSync hook is the right place for setup. It kept half a version from going live. An init container in the app pod would have left the new pods failing in a loop.
• RollingSync couples the tenants. That is intended, because the first tenant serves as a pilot. It also means the slowest or most broken tenant sets the pace for everyone behind it.
• enabled: false is not a finding. Applications generated by an ApplicationSet with RollingSync always carry it. The question is which step is stuck.
• A read-only code tree finds assumptions. Software that wants to write into its own directory at deploy time fails there immediately. Uncomfortable, and good.

One point is not built yet. An alert that reports not just “out of sync” but the step the rollout is stuck on would have saved the hour of searching.

Conclusion

A staged rollout does what it should: it holds an error at one tenant. That it holds every tenant behind it as well is part of the contract, even if in operation it feels like a second error.

On that Friday no site went down. For three hours no change arrived either, and the only trail led through a setting that looked like a fault and was none. Anyone using RollingSync should know where to look when a tenant does not follow: at the step before it.