You Can't Blindly Bundle CI Jobs

TL;DR

TL;DR: Merging N jobs into one drops per-job allow_failure, per-job artifacts/JUnit, and red/green-per-check in the UI. Build the merged job additively and validate it differentially — run new-vs-old on a real repo and compare verdicts — before you swap anything fleet-wide.

What happened

The lint stage ran eight separate jobs on the same PHP image: composer normalize / validate / psr, php-lint, json, yaml, typoscript, commit-msg. Eight jobs means eight pulls of the same image and eight runner slots. On a spot pool that caps at sixteen concurrent, that is the bottleneck. Fold them into one job → one pull, one slot. Easy win, right?

Here's what "one job" quietly costs you, none of which shows up until it bites a specific repo:

• Per-job allow_failure. A fork in the fleet sets lint:composer:normalize: allow_failure: true because its upstream code can't meet the normalize gate. Bundled, that tolerated failure has nowhere to land — the whole job goes red.
• Per-job artifacts. The eslint job emits a JUnit report the merge-request UI renders inline. One combined job means one artifact contract.
• Per-check red/green. Eight jobs give you eight signals in the pipeline graph. One job gives you one — "lint failed," go read the log.
• Parallelism. Eight jobs run concurrently; one job runs the checks in sequence.

So the merged job needs to re-implement the semantics it's collapsing. Change-gating moves inside the script (each check runs only if files it cares about changed — fail-safe: if the diff base can't be resolved, run everything rather than skip and go falsely green). And allow_failure comes back as a per-check soft-checks input:

soft-checks — allow_failure, per check

run() { # $1=key $2=label $3=fn
  if ! ( set -eu; "$3" ); then
    case " $SOFT " in
      *" $1 "*) echo "⚠ $2 failed — soft, not failing the job" ;;
      *)        echo "✗ failed: $2"; FAILED=1 ;;
    esac
  fi
}

Then — and this is the actual point — don't trust it because the YAML lints and bash -n passes. Validate differentially: ship the new job additively (old eight untouched), add it to one real consumer alongside the originals, and compare verdicts check-by-check on the same commit.

That one exercise caught three things no amount of reading would have:

differential run — first consumer

lint:composer:normalize   failed   (allow_failure — tolerated)
lint:composer:validate    success
lint:composer:psr-verify  success
lint:php:all              failed   ← why?
  ── composer normalize   ⚠ soft   soft-checks works ✓
  ── composer validate    ok
  ── json lint            ✗       jsonlint binary missing → see post 01

• The allow_failure gap — the fork would have broken on a blind swap. (Fixed with soft-checks.)
• A hard dependency on a jsonlint binary that isn't everywhere. (Fixed with a fallback.)
• The register_argc_argv bug in that fallback — which only surfaced because a real repo ran it in the real image. (part 01.)

None of those are happy-path. A green pipeline on the new job would have "passed" while proving nothing about equivalence. The conclusion I keep re-learning: the merged abstraction stays an opt-in for repos that pass cleanly, and the fleet-wide swap becomes a planned migration — because every consumer that overrode a per-job setting has to be mapped, not assumed.

The lesson

A green pipeline proves the happy path, not that your new thing is equivalent to the old thing. Differential validation on a real consumer is the cheapest insurance you'll ever buy before a fleet-wide change.

Conclusion

Job consolidation saves runner slots, but only free on paper. Every per-job setting that disappears has to be deliberately rebuilt in the merged job — and the only reliable proof that it worked is a differential run against a real consumer, not a green YAML lint.