sylius-robots-txt — a robots.txt that answers per channel.

A wildcard group says nothing about GPTBot

A crawler evaluates the group that names its own token and ignores the wildcard group as soon as a more specific one exists. So if you try to govern AI crawlers through User-agent: *, you have said nothing about them — you have only written a rule that is followed by exactly those crawlers that carry no token of their own anyway. That is why every product token gets its own group, written out and by name, instead of being lumped together.

The Content Signal stands in the document and also goes out as a response header, because the two have different reach. The robots.txt describes a crawl policy for an entire host — it is a statement of intent that a client fetches before it retrieves anything else. The header describes the specific resource being retrieved, at the moment it is delivered. A client that has never read the robots.txt still sees the signal for the document it is currently holding.

That leaves the trap that most often renders this package ineffective: a static public/robots.txt. The web server delivers it before the request reaches Symfony at all — the route is never called, no log entry is written, no error is thrown. The channel configuration is then entirely correct and entirely without effect. The file belongs deleted during installation, not edited.