Filtered, except for the messages

TL;DR

TL;DR:git filter-repo with --path and --replace-text cleans files, but not commit messages and not tag texts. So in one repository internal links sat in the public history, and in the second the names of internal projects would have gone out. The mirror now rewrites the messages too. Then it checks every message and every tag against the same patterns, and it does not push while anything is left.

What happened

git filter-repo removes files from every revision with --path and replaces text in file contents with --replace-text. Both act on blobs, that is, files. Commit messages and the texts of annotated tags are not blobs. They pass through unchanged.

Two finds

In the first repository, the release commits held links to the internal GitLab instance. The release tool also writes the changelog into the commit message, with one link per commit. The file CHANGELOG.md had been cleaned, its copy in the message had not, and that was public.

In the second repository, commit messages named internal projects, because an exception in the configuration concerned exactly them. The configuration itself is not mirrored. The message that explained it would have been. Nothing went out, because the mirror had no token yet.

I only noticed both when I checked on purpose whether really nothing internal was going out. The files were clean, the messages were not.

Rewrite the messages too, then check

The mirror component got a third input, replace-message. It calls git filter-repo --replace-message and so rewrites commit messages and tag texts. Then the job reads every message of the rewritten history and every tag text. It checks every pattern again, literally, as a regular expression or as a glob. If a hit remains, the job stops and does not push.

The pattern list has two layers. At the top are targeted rewordings, so sentences stay readable. Below them are single names as a net. What the targeted rules miss, the net replaces. What even the net misses, the check stops.

The public history rewritten

For the first repository that meant replacing the GitHub history with a force push, rewriting all tags and then checking from outside. A fresh clone from GitHub, every message and tag text searched: zero hits for the internal domain. As of 13 October: 78 commits and 10 tags, all messages clean.

Whoever cloned the old state keeps it, though. A force push deletes nothing from other people's copies.

Two pitfalls

The pattern list goes into a Python string as JSON, and a \w in it broke the JSON. So the list uses character classes instead of escapes, [.] for a dot, for example.

Empty commits also disappear. A commit that only changed removed files is empty after the filter. It drops out together with its message, which is what you want here.

The lesson

A public mirror publishes the whole history: files, messages, tags and release texts. A filter without a check afterwards is a hope, and the check has to be able to stop the push. The best rule comes before any filter, though: in repositories with a public mirror, internal names stay out of commit messages.

Conclusion

I filtered what I had in front of me as files. The messages were not on my list, because I never saw them as content. Since then the mirror checks the result, not my assumption, and a hit stops the push.

If you missed the start, it is in part 7: Grant first, switch second.