Grundschutz++ as code, part 4: Measurement as evidence

01 — What a check result is in OSCAL

OSCAL has a model of its own for checks: assessment results. It keeps apart what runs together in a log:

• Observation: what a check saw on a subject, a project for example, together with the time and the method.
• Finding: what follows for a requirement, satisfied or not satisfied, and which observations support it.
• Subject: what an observation refers to, as a resource of its own with a link.

In my results there is therefore no finding without an observation and no observation without a subject. That is what turns a statement into evidence.

02 — Checks that already run

I did not build new checks for this. What interested me was which of the existing ones evidence a Grundschutz++ requirement, over weeks rather than on a single day.

• Protected default branches evidence TEST.1.3 and TEST.2.1: changes reach the default branch only by the intended path, and its history cannot be rewritten.
• Protected release tags evidence TEST.4.1: only someone with the right role turns a commit into a release.
• Merge only with a green pipeline evidences TEST.3.1.6: the tests run on every change before it lands.
• Infrastructure drift evidences KONF.2.5: a scheduled tofu plan compares the account with the code.
• Backup freshness evidences NOT.4.4 and NOT.4.10: the copies in the second region are younger than a set limit.

Which check evidences which requirement is a judgement. So I put it next to the checks as data rather than into the code. Anyone who sees it differently changes one line and sees in the next run what follows from it.

03 — From check report to result

Every check writes a small run file: which requirements, which subjects, what it saw on each. results.mjs from grundschutz-worksheets turns it into a valid OSCAL document.

{
  "check": "protected-tags",
  "title": "Protected release tags",
  "method": "TEST",
  "start": "2026-10-01T03:00:00Z", "end": "2026-10-01T03:02:41Z",
  "catalog": { "path": "…/Grundschutz++-resolved_catalog.json", "commit": "367d775…" },
  "requirements": ["TEST.4.1"],
  "subjects": [
    { "title": "group/app", "href": "https://git.example.org/group/app",
      "satisfied": true, "detail": "v* protected, maintainers only" }
  ]
}

node results.mjs run.json > results.json
node validate.mjs results.json

The generator follows three rules:

• One outlier is enough: if a single subject is not in order, the requirement counts as not satisfied in this run, and the finding names exactly those observations.
• No subject, no result: a run that looked at nothing stops with an error. It cannot evidence anything.
• Only IDs from the catalogue: every requirement has to exist in the pinned state of the catalogue, otherwise generation stops.

04 — What was not checked has to be stated

The tricky part is what a check leaves out. Archived projects or repositories without CI have nothing to say on “merge only with a green pipeline”. Simply dropping them would make the result look better than it is.

The mapping therefore knows three verdicts: satisfied, exempt and not satisfied. Exempt projects do not appear as observations. The result's description names them, though, with count and reason. Everything else counts as not satisfied, including a verdict nobody expected.

That happened on the very first day. One check emitted a verdict of its own for archived projects that the mapping did not know, and the first run promptly reported 45 violations that were none. Since then a test checks on every change that every possible verdict is mapped. A gap like that now shows up in the build and not in the audit.

The same holds for sources outside GitLab. A source that is not set up yet is missing from the result, and the log says why. A source that is set up but cannot be read counts as not satisfied. A measurement that did not take place evidences nothing.

And on a small scale: if the API does not answer for a single project, that project gets the verdict “unreadable” and counts as not satisfied. The report on all other projects is unaffected. At first, one read timeout still threw away the evidence for every project.

05 — The date is part of the evidence

“The default branches are protected” describes today. An audit asks for more. It wants to know whether that held throughout the past months, and a single report has no answer to that. So the checks run every six hours, and every run produces a document of its own with a start and an end.

The IDs in it derive from check, start time and subject (UUID version 5). The same run yields the same document, byte for byte. Two runs can be laid side by side, and a change between them stands out at once. After the first seven days there were 34 to 38 results per check. The schedule alone yields 28; manual runs and runs after changes to the checks came on top.

06 — Signed and impossible to delete

Evidence the operator can change afterwards is not worth keeping. Every result therefore lands in two places:

• As an OCI artefact in the registry, signed with the same key as the images. Verifying it takes only the public key.
• As a locked copy in a bucket with Object Lock in COMPLIANCE mode. For 400 days nobody can delete or overwrite the copy, not even the account that owns the bucket.

oras pull registry.example.org/evidence/results:protected-tags-2026-10-01T0300Z
cosign verify --key cosign.pub registry.example.org/evidence/results@sha256:<digest>

Why both? The signature stops anyone from changing a result unnoticed. It can still be deleted. For images that does not matter, they can be rebuilt from the code. Yesterday's measurement cannot be repeated.

Three details that are easy to get wrong:

• 400 days, not 365: one audit year plus some slack in case the date moves. After that a lifecycle rule removes the copy.
• No key of your own for the bucket: with a customer-managed key, the locked copies could be made unreadable simply by deleting the key. The lock would then be a formality.
• Do the clean-up yourself: the registry's clean-up rule reaches back only 90 days. The job therefore deletes older tags itself once they are 400 days old.

07 — The deliberately broken control

A check that has only had green runs so far has not shown that it can recognise red at all. That takes a counter-proof, and I brought one about on purpose.

The basis is an empty test project, configured like all the others: only maintainers may create release tags. On 30 September the test went like this:

• 20:35 UTC: the run rates the test project “as required”. The result is signed and stored locked.
• 20:37: I remove the test project's tag protection, nothing else.
• 20:57: the next run reports exactly this project as “NOT as required – missing”, as an observation with a link to the project, and rates TEST.4.1 as not satisfied. This result, too, is signed and locked for 400 days.
• 20:59: I restore the protection. At 21:25 the run rates the project “as required” again.

What remains are three documents that can no longer be changed. That there was a gap between 20:37 and 20:59 is therefore settled, and the check saw it. That is what counts.

It did not go smoothly, by the way. On the first attempt that morning the check did spot the break, but a read timeout on another project threw away the whole report. That is where the “unreadable” rule from section 04 comes from.

08 — Three bugs that did not show up locally

All three sat in the steps that measure, sign and store. None of them occurred on my machine.

• cosign 3 and the transparency log: signing without a public transparency log takes more than --tlog-upload=false in cosign 3. Without the other flags cosign refuses the call. Image signing had this right long ago; the new step had copied only half the line.
• pipefail and an empty grep: the runner executes scripts with set -o pipefail. During clean-up, grep filters out the tags that carry a date. If there are none, for instance because only signature tags exist, the whole line fails and the job stops without a message.
• IPv6 only on the runner: the backup check waited an hour for the IPv4-only endpoints of the AWS SDK until the job was cancelled. Locally the same check finished in seven seconds. It now uses the dual-stack endpoints and short timeouts.

For an evidence job that means: it has to fail as loudly as a check that finds something. A silent abort looks like a quiet day on the dashboard.

09 — What a result does not say

• Not the whole requirement: the drift check evidences that the infrastructure matches its code. KONF.2.5 asks for more, a defined way of handling deviations, for example. Only that one part is measured.
• Not what runs in the cluster: whatever runs there without being in a repository stays invisible to these checks. That is the second level, and it gets a part of its own.
• No compliance: a result records what one run saw. Nothing more. Whether a system meets the requirements is a judgement, and in the end it is made by a person who also knows what no check measures.

Part 5 shows how measurements, component definitions and that judgement become a security plan.

10 — The series at a glance

1. Grundschutz++ against your own documentation: the method, the catalogue as a data model, the comparison with IaC and GitOps.
2. Applicability as data: the statement of applicability as an OSCAL profile.
3. Containers that say which requirements they meet: a signed component definition on every image the pipeline builds.
4. Measurement as evidence: checks write their results as OSCAL, dated, signed, impossible to delete. This post.
5. The ISMS as a build product: security plan, assessment plan, results and plan of action as an OSCAL chain, rebuilt with every measurement.
6. What no repository knows: the second level, the running system against its code.
7. One measurement, many standards: NIS2, DORA, C5 and SOC 2 as mappings onto Grundschutz++.
8. The chain to take away: general checks, the report over a period and a template that wires it all together.

Parts 1 to 6 describe what runs in my ISMS. Each appears only once its step runs in operation and has been measured. Parts 7 and 8 show tools I built after feedback at the InfoSec-Stammtisch and do not use myself. The code is in grundschutz-worksheets. A map of the whole series is OSCAL in practice: eight models, one chain.

Conclusion

The checks existed before. What is new is that their result stays: as a dated OSCAL document, signed, in a copy nobody can delete. The real work lay less in the format than in two decisions, namely what counts as satisfied and what was not checked. Both sit next to the code as data, and a test makes sure no gap goes unnoticed.

Continue with part 5: The ISMS as a build product.