Kai Ole Hartwig
Planning a Kubernetes migration?

typo3-indexnow — new pages in the index within minutes.

A new page waits until a crawler happens to come by, which can take days. typo3-indexnow tells search engines through IndexNow which pages were published, changed, scheduled or removed. Bing, Yandex, Seznam, Naver, Yep and Amazon share every submission. Bing's index feeds ChatGPT search and Copilot, so a page can be citable there the same day.

  • Composer package:koh/typo3-indexnow
  • Built on:koh/indexnow, the framework-agnostic client
  • TYPO3: 14 · PHP: 8.5+ · MIT · source on GitHub

TL;DR — Does this fit my setup?

Use it if

Do not use it if

Effort

One Composer package and one line in the site configuration. No key to generate, no file to upload, no cron job to set up.

How it works

Editor saves, moves, deletes               start time passes
workspace is published                            │
   │  DataHandler hook                            │  scan per run
   ▼                                              ▼
Queue tx_kohindexnow_queue   (one row per page and language)
   │
   ▼
indexnow:run   (scheduler, every five minutes)
   │  build the URL, check the rules, group by host
   ▼
one request per host ──► api.indexnow.org ──► Bing, Yandex, Seznam and others
TriggerAnnounced
Page or content element saved, moved, deleted, restoredThe page in the language that changed; every language when visibility, access or SEO flags changed; for a move, the old page as well
Workspace publishedWhat became public
Start or end time passesThe page, although nobody saved anything
First run with an active hostEvery visible, indexable page in every language

A changed page is announced even when it is now hidden, deleted or noindex. The engine has to re-crawl to learn that. Its parent page goes with it, because a new article changes the list it appears in.

No key to manage

The key is derived per host from the installation's encryption key (HMAC-SHA256). Every replica computes the same one, every host gets its own, nothing is stored. A middleware serves the key file at /<key>.txt.

What is never announced

A submission cannot be recalled, and every participating engine receives it. So the extension checks what may go out before every request.

RuleWhat it prevents
Only in the Production application contextLocal installations and test systems stay silent.
The library's host policyIP addresses, .localhost, .test, .local, .internal and hosts with dev, staging or preview in the name are never announced, even when they run in Production. There the key file does not exist either.
Access-restricted pagesA frontend user group on the page, its translation or an ancestor that extends to subpages: the URL stays private, even when the page changes.
Start time in the futureA scheduled page is announced once it is live, even when it was published from a workspace earlier.
Page types that are not documentsLinks, shortcuts, mount points, spacers, folders and the recycler, as in the core XML sitemap.
Workspace draftsOnly what is published is announced.

The site setting indexnow.deniedHostSuffixes blocks further hosts.

New content in Bing, ChatGPT search and Copilot sooner? I help with the setup, detail pages for news or products, and monitoring the submissions. Get in touch →

Installation and activation

Two steps

  1. Install the package: composer require koh/typo3-indexnow, then vendor/bin/typo3 extension:setup. The setup creates the scheduler task that runs indexnow:run every five minutes.
  2. Add the set to every site that should announce, in config/sites/<site>/config.yaml:
dependencies:
  - koh/indexnow

 

The first run with an active host announces the whole site once. That is the go-live; afterwards only changes go out.

Commands

 

vendor/bin/typo3 indexnow:status        # per host: active or why not, key file, queue
vendor/bin/typo3 indexnow:run           # what the scheduler task runs
vendor/bin/typo3 indexnow:submit-all    # queue every indexable page again

 

indexnow:status answers “why is nothing being sent?”.

Site settings

SettingDefaultEffect
indexnow.enabledtrueSwitch for the site
indexnow.keyemptyEmpty: derived per host. Set only to keep an existing key
indexnow.excludedDoktypes3, 4, 6, 7, 199, 254, 255Page types never announced
indexnow.includeParentPagetrueAlso announce the parent of a changed page
indexnow.deniedHostSuffixesemptyFurther hosts never announced

Delivery: queue, retry, give up

Nothing expensive happens while an editor saves. The hook remembers the records and writes the queue once at the end of the DataHandler run. URLs are built later by the worker, from the raw page record, which is why a deleted page still has a URL. Saving a page twenty times before the next run announces it once.

ResponseWhat happens
AcceptedThe rows are removed.
Temporary (429, 5xx, no connection, 403 while the key file cannot be fetched yet)Wait with exponential backoff from one minute to six hours, honouring Retry-After. Given up after eight attempts.
Rejected (400, 422)The rows are removed and the failure is logged, because the same request would be rejected again. The scheduler task exits with status 1.

HTTP goes through TYPO3's configured client: proxy, CA bundle and HTTP.allowed_hosts apply, ten-second timeout, no redirects.

Extending and limits

Three PSR-14 events

Limits

Common pitfalls

Next step

Getting new content into the index?

If new pages take days before Bing and the AI search engines know them, typo3-indexnow helps. Talk to me about the setup, listeners for your detail pages or monitoring the submissions.

Discuss the setup →

Or email me directly: mail@ole-hartwig.eu