Kai Ole Hartwig
11 min read

A Go fix is not a package fix

On October 27, 2026, my report across all container images showed 0 HIGH and 0 MEDIUM. One day later, after the first ten images had moved from Wolfi to openSUSE Tumbleweed, it showed 16 HIGH and 33 MEDIUM, every one of them in the moved images. The distribution was not worse. It simply follows a different rule for when a program gets rebuilt, and that rule was the one thing I had never checked before the switch.

TL;DR

TL;DR: A Go program has the standard library and its modules compiled in, so a fix only reaches it through a new build. openSUSE Factory rebuilds a package only when its own sources change. A new Go release alone does not trigger a rebuild there. Wolfi, on the other hand, rebuilds all Go packages on a Go release, and that is why my images were clean there. Since the move I build every Go, Rust and Python program in my images myself, with the current Go. That took k6 from 5 HIGH and 13 MEDIUM to zero. The price is 40 to 60 specs of my own, each rebuilt for two architectures on every Go release.

What the report showed

My first suspicion was the C libraries, but the jump did not come from there. bash, reuse and my own tool pinup scanned clean on Tumbleweed as well. The findings sat in two places, and both came down to the same question: when a package was last built.

Go programs from the distribution

k6 carried the standard library of Go 1.26.8 and golang.org/x/net 0.58. crane, govulncheck and helm looked the same. All four were openSUSE packages, built with the Go that was current when their last version came out. The fixes released for Go and x/net since then had never reached these binaries.

A newer Tumbleweed snapshot changed nothing. There simply was no fixed package to move to: the program had no new version, so there was no new build either.

The aarch64 port

A second, smaller effect came on top. On aarch64, Tumbleweed is a port. Its packages come from a separate tree under download.opensuse.org/ports/aarch64, and the project definition of openSUSE:Factory lists only x86_64 and i586 for the standard repository. So the port runs at its own pace. brotli 1.2.1 and bind-utils 9.20.30 were already available on x86_64 and not yet on aarch64. My clusters run on arm64, so this hit every image.

Two rules for the same fix

Go links the standard library and all modules into the binary. Even when a program links dynamically against libc through cgo, the Go part sits entirely inside the file. A fix in Go or in x/net can therefore reach the program in only one way: through a new build. Updating the Go toolchain on the system does nothing for binaries that are already built.

Wolfi draws the consequence. When a new Go release comes out, the distribution rebuilds every package built with Go. Chainguard describes one such case: after the Go update, a second pull request kicked off rebuilds of 192 Wolfi packages, and they were done within a few hours. That is why my images scanned clean there. I had taken this for a property of the images. In fact it was a service of the distribution, and it disappears completely when you switch.

openSUSE takes a different route. In the Open Build Service, each project defines what triggers a rebuild. The default is transitive: when a package changes, every dependent package gets rebuilt. openSUSE:Factory, which Tumbleweed is made from, is set to rebuild="local" instead, which means only packages whose own sources changed get built. A new Go package therefore does not automatically lead to a rebuild of k6. That only happens when someone touches the k6 spec, typically for the next upstream release.

For a distribution of this size that makes sense, because transitive rebuilds across thousands of packages cost an enormous amount of build time. For tools that consist mostly of Go modules, though, it means a fix can wait weeks for the program's next release.

None of the distribution comparisons I had read before the switch asks this question. They compare package count, image size and the advisory feed. Who rebuilds what, and when, is in none of those tables.

Finding outdated Go binaries in an image

Before you choose a path, take a look at the binaries themselves. Go writes build information into every binary, and you can read it without the source code:

 

go version -m /usr/bin/k6

 

The first line names the toolchain the binary was built with. Below it come the main module (mod), every dependency (dep) with its version, and the build settings (build). If the toolchain is two patch releases behind the current Go, you know without any scanner that this package is waiting for the program's next version.

Whether the old toolchain actually brings a vulnerability into the image is a question for govulncheck in binary mode:

 

govulncheck -mode=binary /usr/bin/k6

 

govulncheck uses the symbol table to look for the affected functions. That is more precise than a plain version comparison, but it has limits. Binary mode gives you no call stacks, and code that sits in the binary but is never called can still be reported. If the symbol table is missing, govulncheck reports every vulnerability in the modules the binary contains.

One point is easy to miss: many scanners judge files installed by a package manager only against the distribution's advisories. Trivy does this by default, and only --detection-priority comprehensive widens the check, at the price of more false positives. Whether you see an outdated toolchain at all therefore also depends on how your scanner is configured and how quickly the distribution maintains its own advisories.

My report comes from Trivy, run through GitLab's container scanning component. The component sets the detection priority to comprehensive, so that binaries installed by packages reach the language analyzer too. That was less obvious than it sounds. GitLab's analyzer builds Trivy's command line itself and reads the switch from CS_TRIVY_DETECTION_PRIORITY, not from Trivy's own TRIVY_DETECTION_PRIORITY. For a while my tag scans therefore ran in precise mode without anyone noticing, and reported images as clean without a single Go finding. It only came out when a scan of the same image with plain Trivy showed two findings.

For the report of October 28 this means: the findings in k6, crane, govulncheck and helm came from the binaries' module graph, matched against the Go advisories. In the default mode Trivy would have checked these files against the SUSE feed only. The findings in brotli and bind-utils, on the other hand, came from the openSUSE advisories, where the fix had been published and simply had not reached aarch64 yet.

The decision: build every application program myself

Going back to Wolfi was not an option; the reason for leaving is in part 1 of the "New foundation" series. So Tumbleweed stays the base. But every application program now comes from my own package repository, and openSUSE only supplies what it does well: the C libraries and the base system.

How a package of my own is built

Every Go, Rust and Python program in an image is a package of my own. The spec builds from the upstream tag, with the current Go rather than the one that was current when the program was released. Before the build I raise the affected modules and vendor them, so the build is reproducible and runs without network access:

 

go get golang.org/x/net@<fixed version>
go mod tidy
go mod vendor
GOTOOLCHAIN=local go build -mod=vendor -trimpath -o k6 .

 

GOTOOLCHAIN=local makes sure the installed toolchain really does the build, and that go does not quietly download another one because the project's go.mod asks for a specific version. For Rust I vendor the crates, for Python the dependencies.

This is what it looks like in the k6 spec, shortened. The vendored modules come in as a source of their own, and the build uses the Go from my own package:

 

Name:           k6
# renovate: datasource=github-tags depName=grafana/k6 extractVersion=^v(?<version>.*)$
Version:        2.3.0
Release:        102.koh
Vendor:         koh
Source1:        vendor.tar.gz
BuildRequires:  go1.27

%build
export GOTOOLCHAIN=local GOFLAGS="-mod=vendor -trimpath"
CGO_ENABLED=0 go build -ldflags "-s -w" -o k6 .

%check
./k6 version | grep -F "%{version}"

 

pinup reads the renovate: comment to find new k6 versions. The %check is small, but it catches the case where a build passes and a different version ends up in the binary.

Rebuilds as a job, not a reminder

A new Go release or a module fix triggers the rebuilds. pinup raises Go and the modules, and the packages build on the merge. Nobody has to remember that k6 should be rebuilt after the next Go patch.

Release number and repository priority

Where openSUSE ships a package with the same name, my release numbers start at 100.koh. For the same version, RPM compares the release number segment by segment, and a digit beats any letter. 1.koh would therefore sort below SUSE's 1.1, and a scanner checking my package against the SUSE advisories would treat a SUSE fix in 1.1 as missing. The 100 is a floor, not a count; k6 is at 102.koh today. zypper shows the result of the comparison directly:

 

zypper versioncmp 1.0.0-100.koh 1.0.0-1.2

 

The release number alone is not enough, though. If openSUSE raises the version, its package wins again. That is why my repository also gets a higher priority. With zypper, a smaller number means a higher priority, and the default is 99. A repository with higher priority wins even when another one offers a newer version. So my build tool writes my repository into the target with priority 90, ahead of the default 99, and checks metadata and packages against my own key:

 

[koh]
name=koh (devops/rpm-packages)
baseurl=…/koh/aarch64/snapshots/<id>/
priority=90
gpgcheck=1
repo_gpgcheck=1
pkg_gpgcheck=1

 

The first two builds of my own were k6 and trufflehog. k6 went from 5 HIGH and 13 MEDIUM to not a single finding.

What it costs

In the end it is 40 to 60 specs, and each one is rebuilt for two architectures on every Go release. Most of them follow the same template, so the effort per package is small. Added up, though, it is exactly the maintenance Wolfi used to do for me. Now I do it myself.

A single module can set off more than you would expect. A bump of x/net touched 27 specs, and the build job for each architecture ran into its time limit. Since then every architecture builds in four shards, so a broad bump no longer hangs on a single job.

The target is stated so that it can be measured: no finding with a published fix that is older than its SLA, meaning one day for CRITICAL and seven days for HIGH. Zero at every moment is deliberately not the target. Between an advisory and the rebuilt image, a finding is always open for a few hours, on any base.

Recommendations

  1. Scan before the rollout, split by origin. After a base switch, compare the findings in system libraries and in application programs separately. Otherwise you only see a total and look in the wrong place.
  2. Read the toolchain from every binary.go version -m shows which Go a binary was built with, and govulncheck -mode=binary shows whether that matters.
  3. Check what feeds your scanner. If it judges package files only against the distribution's advisories, your result depends on how quickly those are maintained.
  4. Decide per program who builds it. The distribution package where it keeps up with the fixes, your own build where it does not.
  5. Make the rebuild a job. A Go release has to trigger rebuilds without anyone remembering to. Release number and repository priority have to be set so that both the package manager and the scanner rank your build above the distribution's.

Conclusion

Choosing a base distribution also means choosing who rebuilds what, and when. I had praised Wolfi for clean images and missed that the cleanliness was a service I cancelled the day I switched. So the question for your next base switch is not which distribution has fewer findings. It is who brings the next Go fix into your binary. And how long that takes.

I check who builds which program in your images, and when.

Base images, Go and Rust binaries, toolchain versions and rebuild paths. At the end you know how long each fix takes to reach your image.

Book a call →

About the author

Photo of Kai Ole Hartwig.

Kai Ole Hartwig

Freelance DevSecOps consultant · OnlyOle Consulting

Programming since 2002 – self-taught, set up my own business with KO-Web in 2012. Over 100 projects, with a focus on security, performance, automation and quality. Today freelance: DevSecOps consulting, training and software development.