I was halfway through a post on SEO versus AEO, typing a confident line about how AI agents now read your website in real time, when a small and very rude question walked into my head.
Has any AI ever read this website?
I did not know. Not “the numbers are low.” I mean I had no numbers at all. Here I was, explaining to the internet how answer engines work, like a cricket coach giving a talk on batting technique who has never once watched his own footage.
So this post comes in two parts. Part 1 is the idea, the blind spot, and what I changed to fix it. Part 2, next Tuesday, is the actual data, including all the places where I turn out to be wrong.
My blog had a visitors’ book only humans could sign
I do have analytics. Google Analytics sits on every page, and it tells me useful things, for example that my most loyal reader is in Wellington. I suspect it is me, checking whether the post published.
But Google Analytics only counts browsers that run its little piece of JavaScript. Humans run it without knowing. Bots mostly do not run it at all. They ask for the page, read the HTML, and leave.
The site itself is hosted on GitHub Pages, which serves every single request and tells you nothing about any of them. No access logs, no visitor list, nothing.
So the bots could walk in, read everything, and walk out without signing the book. For a blog about AI, that is a slightly embarrassing place to be.
SEO waits, AEO answers
Here is the idea I was trying to write about, in plain words.
SEO (search engine optimisation) is about being ranked. You publish a page. Some days or weeks later, Googlebot comes by, reads it, files it in an index, and decides where it sits against every other page on the topic. Much later, a human types a query, gets ten blue links, and maybe clicks yours. The whole thing happens offline, in batches, on the search engine’s schedule. Nobody is waiting.
AEO (answer engine optimisation) is about being used. Someone asks ChatGPT or Claude or Perplexity a question, and the answer gets built right there, while they wait. It comes from what the model already knows plus whatever it goes and reads in that moment. You are not trying to be link number three. You are trying to be the page the agent reads, trusts and quotes. And quite possibly nobody clicks anything, because the answer is already on the screen.
That was my original framing. SEO is offline, AEO is real time. It is mostly right. Only mostly, and the gap turned out to be the interesting part.
It is three clocks, not two
AI companies do not send one bot. They send at least three, and each one runs on a different clock.
| Job | OpenAI | Anthropic | When it affects you |
|---|---|---|---|
| Training | GPTBot | ClaudeBot | Months later, in the next model |
| Search index | OAI-SearchBot | Claude-SearchBot | Ahead of time, a lot like Googlebot |
| Live fetch | ChatGPT-User | Claude-User | Right now, while a person waits |
Perplexity runs the same pattern with PerplexityBot and Perplexity-User.
The first two are basically old-fashioned crawling with a new owner. Slow, scheduled, offline. The third one is the genuinely new thing. Nobody ever sat waiting for Googlebot to finish reading their page. Somebody is always waiting on ChatGPT-User, because it only comes when a real person has just asked something.
So AEO is not real time instead of offline. It is both, stacked on top of each other.
The do not disturb sign has a loophole
You would think robots.txt lets you control each clock separately. Block training, allow search, allow live fetch. Very neat.
Mostly, yes. But OpenAI’s own documentation says that because ChatGPT-User acts on a user’s request, robots.txt rules may not apply to it. Perplexity takes the same view for Perplexity-User. Anthropic draws the line differently and says its bots respect the file.
In other words, the “do not disturb” sign on your gate works on the delivery van. It does not necessarily work on the person ringing the bell on behalf of someone else. Whether that is fair is a debate for another post. For this one, it simply means robots.txt is a request, not a lock.
So I put a front desk in front of the blog
To see any of this, something has to stand between the visitor and GitHub and write down who came in. Cloudflare’s free plan does exactly that.
I moved the domain’s nameservers to Cloudflare. That sounds like heart surgery. It is closer to changing your postal address, with the same nervous feeling that the one important letter will go to the old house. Now every request to kiwigpt.co.nz passes through Cloudflare first, and a screen called AI Crawl Control labels the AI bots by name.
Two honest caveats. On the free plan, Cloudflare recognises bots by the user agent string they announce, and anyone can fake that. And a bot that does not announce itself will not appear at all. So whatever I find will be a floor, not a ceiling.
Two pleasant surprises also. First, Cloudflare now sorts AI traffic into three categories: Search, Agent and Training. Those are my three clocks, more or less, which made me feel very clever for about four minutes. Second, since October, free domains keep 30 days of analytics history. I was ready to screenshot the dashboard every evening like a nervous parent. Not required.
If you try this yourself, do one thing first. Since 15 September 2026, new Cloudflare domains block Training and Agent crawlers by default on pages that show ads. I opened the crawler list and made sure every one was set to Allow. Before you conclude that no AI loves you, check you have not locked the door yourself.
My prediction, in writing, so I cannot wriggle out
A guess written down is a guess you can be wrong about in public. That is the point. So here is what I expect to see for the week of 24 to 30 November.
- Training bots will be the biggest visitors by far. GPTBot and ClaudeBot together will make more requests than every other AI bot combined.
- Live fetches will be tiny. ChatGPT-User, Claude-User and Perplexity-User together will visit fewer than five times in the whole week. Possibly zero. I would very much like to be wrong about this one.
- AI bots will outnumber my human readers. I am not sure whether to feel flattered or used.
- The most-read thing will not be a post. It will be robots.txt or the RSS feed. A bot reading the house rules more carefully than any human ever has.
Part 2 comes on Tuesday 1 December
Next week I will put up the real numbers, a screenshot or two, and a straight scorecard against these four guesses. I will also compare KiwiGPT with Cloudflare’s crawl-to-refer figures, which measure how many pages AI companies read for every human visitor they send back. The industry numbers are not flattering. We will see whether mine are worse.
For as long as this blog has existed, I have been writing for readers I could only half see. Next week I find out who else has been in the room. Even if the answer is one confused bot reading robots.txt three times a day, that is still one more reader than I could prove yesterday. And honestly, anyone who studies the house rules that carefully is my kind of reader.
Written for KiwiGPT.co.nz · Generated, Published and Tinkered with AI by a Kiwi