Three kinds of AI crawler, and the one that decides whether you get credited

On 29 July, one crawler read all 42 pages of my new knowledge base in about an hour. It was Anthropic’s. Two days later I asked Claude a question that those 42 pages answer better than anything else on the internet — because I wrote them, about a component I built — and Claude answered it perfectly and linked my old website.

I had been treating “an AI read my work” and “an AI credits my work” as one event. They are not one event, and they are not even the same crawler.

There are three kinds, and the confusion is not the reader’s fault: they arrive as almost identical lines in a log file, and the one that actually decides whether you get a link is the one nobody talks about.

The short version

• A crawler that reads to train a model pays off slowly, invisibly, and possibly never in the form of a link. GPTBot, ClaudeBot.
• A crawler that reads to build a search index is the one that decides citations, because it feeds the thing the assistant queries while answering you. OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• A fetch that happens because someone just asked a question is the closest thing to a live signal that your page is being used. ChatGPT-User, Claude-User, Perplexity-User.
• Google does not use this shape, and Google-Extended — the one everybody blocks — is not a crawler at all. It fetches nothing.
• robots.txt does not govern all three. The vendors say so themselves, and they disagree with each other about the third.
• The one assistant that cited me correctly is the one whose only documented crawler exists to link websites. That is not a coincidence I can prove, but it is a coincidence with a mechanism.

What was actually in my log

I publish a free knowledge base about GSAP animation in Webflow — 42 pages, written to be read by both people and assistants. Before announcing it I measured whether AI assistants send anyone to my site at all: ten questions, four assistants, scored by hand. 31 points out of 80, and the knowledge base itself scored zero, because nothing had linked to it yet. That’s the previous article.

Then the announcement went out, and the crawl log did this:

Two details matter more than the totals. About 44 of Anthropic’s 47 requests landed in a single hour — the bundle is 42 URLs, so that is one crawler taking the whole thing in one pass. And OpenAI’s zero is a real zero: the six named crawlers sum to exactly 117, so nothing is hiding in a rounding error.

A note on scope, because the table is wider than the argument. This article explains the four assistants I actually measure — ChatGPT, Claude, Perplexity and Google. Amazonbot was the second-largest reader of my knowledge base and I am not going to explain it, because I have no way to see whether Amazon’s reading ever turns into anyone being credited anywhere. It stays in the table because deleting the rows that don’t fit the thesis is how you get a tidy table and a false one.

Three kinds, and only one of them is about the link

Here is what each vendor says its own agents are for. I am quoting them because what an agent is for is a vendor claim, and the only thing I can assert on my own authority is what showed up in my log.

1. Reading to train the model

GPTBot exists, in OpenAI’s words, “to make our generative AI foundation models more useful and safe,” crawling content “that may be used in training our generative AI foundation models.” ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”

Note what neither sentence promises: a link. Training crawlers turn your work into a model’s general competence, and a model’s general competence has no citation list. It is the most flattering visit in your log and the least traceable.

And it is what I got. ClaudeBot swept 42 pages in an hour, and the answer that came out two days later was the best of the four — it warned me about two failure modes that are documented almost nowhere else — attached to a link to a domain I no longer publish on. Nothing malfunctioned. I had been read by the crawler whose job does not include telling anyone where the words came from.

2. Reading to build a search index

This is the one that decides.

• OAI-SearchBot exists “to surface websites in search results in ChatGPT’s search features.”
• Claude-SearchBot “navigates the web to improve search result quality for users… analyzes online content specifically to enhance the relevance and accuracy of search responses.”
• PerplexityBot is “designed to surface and link websites in search results on Perplexity.”

When an assistant answers your question with sources attached, it is usually because it searched, and the index it searched was built by one of these. This is the visit that can become a link. It is also the one that gets discussed least — the public conversation is about training crawlers and whether to block them, which is a conversation about a different agent with a different consequence.

I did not have to argue this from theory, because my own zero moved. On 1 August, at around 14:00 JST, OAI-SearchBot fetched two pages. Two, out of forty-two — and it is the search agent, not GPTBot. Three days after the first real inbound links to the knowledge base went onto the internet.

3. Fetching because someone just asked

ChatGPT-User covers “certain user actions in ChatGPT” — when “users ask ChatGPT or a CustomGPT a question.” Claude-User “supports Claude AI users. When individuals ask questions to Claude, it may access websites.” Perplexity-User “supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer.”

This is the most interesting line in a log file and the one most likely to be invisible to you: it means a human being asked something a few seconds ago and an assistant went to your page to answer them. Not a corpus. Not an index. A person, now.

Google does it differently, and Google-Extended is not a crawler

If you have read one thing about AI crawlers, it is probably that you should decide whether to allow Google-Extended. Worth knowing what it is: it fetches nothing.

Google’s own categories are common crawlers, special-case crawlers, and user-triggered fetchers. Googlebot — one agent — crawls for Search, and Google states plainly that Google-Extended “does not impact a site’s inclusion in Google Search.” What it controls is whether content Google has already crawled may be used “for training future generations of Gemini models.”

So it is not an agent, it is a permission wearing a user-agent’s clothes. One crawl, two questions, and the second one answered by a token that never requests a page. Which means the mental model “each vendor has three bots” is wrong in exactly one direction, and the direction matters: with Google there is no separate visit to look for in your log, so the crawl you see and the use you are consenting to are decoupled.

Google’s user-triggered fetchers are a real and long list, though — Google-GeminiNotebook, Google-Read-Aloud, Google-Agent and six more, each triggered by a person doing something specific.

The correlation I keep staring at

Perplexity documents no training crawler at all. PerplexityBot is explicitly “not used to crawl content for AI foundation models.” There is no Perplexity equivalent of GPTBot.

And Perplexity is the assistant that got my question right and cited three pages of my knowledge base — the recipe, the reference sheet of verified names, and the section index — while the other three sent people somewhere else.

I want to be careful here, because this is exactly the kind of pairing that gets written up as a law. It is one question, on one site, at one moment. What I can say is that the mechanism is coherent: a company whose only documented crawler exists to link websites has no pathway by which my work becomes anonymous competence, and a company that runs both has two — and only one of them ends in a link. My measurement ranked Perplexity the earliest indicator because it searches the most. Its documentation says why.

Two things stop me claiming more. Perplexity’s citations are unstable in a way its facts are not: in three days the same answer went from nine inline links to four grouped ones, so no argument here rests on a citation count. And a single vendor’s behaviour is not a rule about vendors.

Substitution, not hallucination — what Google actually did

Google deserves its own section, because what it did is a distinct failure and the word “hallucination” hides it.

Google found my work. It got the component’s main attribute right. Then it invented two option names that do not exist and attached plausible default values to them.

Look at the citation list and you can see the seam. What the component is came from my pages. The numbers came from generic articles about GSAP staggering — real articles, correct about GSAP, describing a different thing. The answer is a graft: my component’s identity with someone else’s parameters.

That is substitution, and it is worse than invention for a specific reason: invention leaves a trace a careful reader can smell, while substitution arrives fully sourced. There are citations. They are real pages. They are simply not about the thing being described, and no amount of clicking the sources reveals the problem, because each source is individually fine.

The fix is different too. You cannot correct substitution by writing more truth on your own site — I had already written it, and it was already crawled. What is missing is the specificity that makes the generic article the wrong match: the exact names, on the exact page, stated as the values of a named component. Which is a documentation problem, and I wrote about that separately — including the eight attribute names I nearly invented myself, by reasoning correctly.

robots.txt is not one instrument

Here is the practical consequence of the three roles, and it is the part I did not know before reading all four vendors’ documentation in one sitting.

Training and search crawlers obey robots.txt at all four vendors. On-demand fetchers mostly do not — and the vendors say so in almost identical words. Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Google: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

Anthropic is the exception, and as far as I can tell the only one: it states that all its bots, Claude-User included, respect “do not crawl” signals “by honoring industry standard directives in robots.txt.”

So the same file can exclude the crawler that would have trained on your work while doing nothing whatsoever about the fetcher that answers a live question with it — and whether that is true depends on which vendor you are talking about, not on how carefully you wrote the rule. If you have ever written a robots.txt block for “AI” and considered the matter closed, that is the sentence to sit with.

What my log knew before the answers did

The part I did not expect, when I finally put the two measurements side by side: every one of the four answers had already been predicted by a number I had written down days earlier.

Perplexity had crawled the knowledge base with the agent whose purpose is to link websites — it cited the knowledge base. Anthropic had swept all 42 pages with the agent whose purpose is training — Claude answered best and credited the wrong address. OpenAI had fetched nothing when I ran the test — ChatGPT had nothing of mine to link and linked a webflow.io page. Google had reached my work with Googlebot and had no separate agent to show me anything about the second question — and produced the substitution above.

I did not design that experiment. I ran two instruments a week apart for different reasons and only noticed afterwards that one explained the other. Which is the honest description of most useful measurement.

Where these definitions come from

Every quote above is from the vendor’s own crawler documentation, read on 5 August 2026. I am listing them because an article arguing that provenance matters and then paraphrasing four companies from memory would deserve everything it got:

• OpenAI — developers.openai.com/api/docs/bots
• Anthropic — support.claude.com › does Anthropic crawl data from the web
• Perplexity — docs.perplexity.ai/guides/bots
• Google — common crawlers and user-triggered fetchers

Two of those four addresses had moved since I last saw them cited — OpenAI’s bot documentation is no longer on platform.openai.com, and Anthropic’s support article now lives on support.claude.com. If you find an older article on this subject, check whether its links still resolve before you trust its quotes. These pages change, and anything I have quoted is what they said on the date above.

What this cannot tell you

• A user-agent string is a claim, not proof. My per-vendor numbers are what Cloudflare’s own grouping reported; I did not independently verify that every request came from the network it says it did.
• My crawl view is 24 hours wide. So 1 August is the first time I saw OAI-SearchBot, not demonstrably the first time it came. Two days in between are simply unobserved, and I am not going to claim a first.
• Three interventions are now live at once — the posts, the links added to my project descriptions, and the video — so when I take the next reading, on 28 August, I will not be able to say which one moved a number. All three were the intended fix for the same measured problem, and reporting them as one is the honest option.
• This is one site, in one niche. The three roles come from the vendors’ documentation and are general; every number here is mine and is not.

If you make anything, do this in ten minutes

Open your crawl log and group it by role, not by company. “Anthropic visited” is not a finding. “Anthropic’s training crawler visited and its search crawler never has” is a finding, and it tells you what to do next: it is a distribution problem, not a writing problem. No amount of new pages fixes a crawler that has never found the way in.

Stop celebrating the wrong visit. A training crawler reading everything you own feels like the win. It is the least traceable outcome available. The line to watch is the search agent, and after that the on-demand fetcher — the one that means a person asked.

Then check what your robots.txt actually governs, per role and per vendor, rather than per company. It probably governs less than you assumed, and in one case more.

The numbers here, dated and left as they were measured, live on one page that grows every round: Credited →. The knowledge base all of this is about is here, free and with no signup, and the full story of measuring it is in Credited Elsewhere and in the video.

Round 1 is on 28 August. If the search crawlers keep showing up, I will say so. If nothing moved, I will say that instead — that is the whole point of writing the number down first.

Have an awesome journey,

Francesco