How AI Search Works: Memory, Rank, and the Fetch

I don't start with rankings when I want to know how AI search is treating a site. I start with the robots.

GPTBot and the other indexing crawlers show up on their own schedule. They are reading pages that a model may later treat as background knowledge. Then a different agent arrives: ChatGPT-User, Claude-User, Perplexity-User. That one is attached to a person. Someone asked a question, the product decided memory was not enough, and it opened a URL.

I use the two logs to see where people are looking. The indexing crawl tells me what may become the prior. The user bot tells me which questions are already past memory, and which pages got pulled into a live answer.

That split is how I read ChatGPT, Claude, Perplexity, and Google.

A base model understands the question, answers from what it already knows when that knowledge is good enough, and goes to the web when it isn't. Brand presence gets you into the first answer. Traditional ranking gets you into the second.

Understanding comes before retrieval

The base model has to grasp the question before any of the rest matters. If it understands the job, and it already knows a solid answer, it can respond from base knowledge and never open a page. If it understands the job and the knowledge is thin, old, or risky, it goes looking. If it misunderstands the job, both paths fail in a confident voice. It either answers a nearby question it already knew, or it retrieves a neat set of sources for the question it thought it heard.

This is the part answer-engine work keeps skipping, because it isn't a tag. Perplexity can cite the wrong frame and look rigorous. ChatGPT can finish a stable definition with no fetch at all. Google can rank excellent documents and summarize a slightly different task. The quality of the answer starts with comprehension: who it's for, what has to be true, and what would make the obvious answer fail.

A page cannot repair a model that never felt the constraint. It can make the right reading cheap. Name the case. Name the limit. Put the decision in a passage that still makes sense if it is lifted alone.

"What is answer engine optimization?" is a question a current model may already understand and already know. You are one voice in a chorus. "Which URLs should a B2B site refresh first if ChatGPT-User is hitting pricing and alternatives, while GPTBot is mostly crawling the blog?" is a different question. Memory blurs that into a generic answer. The lookup needs a page about that situation.

Four products, one split

They feel unrelated because their defaults differ. The shape is the same.

Product	Answers from memory when	Goes to the web when	Indexing agents	User-triggered agent
ChatGPT	The question is stable and the prior is enough	The fact is current, narrow, or expensive to invent	GPTBot, OAI-SearchBot	ChatGPT-User
Claude	The prior covers the question it understood	The answer needs a current or specific source	ClaudeBot, Claude-SearchBot	Claude-User
Perplexity	The model still has to frame the question first	By default, because the citation is the product	PerplexityBot	Perplexity-User
Google	The overview can lean on a stable fact	AI Overviews and AI Mode start from the index	Googlebot	Usually inside Google, so it never appears as a fetch in your logs

OpenAI publishes the distinction. GPTBot can feed training. OAI-SearchBot builds a search index. ChatGPT-User fetches because a person did something, and OpenAI says it is not an automatic crawl. Perplexity draws the same line between PerplexityBot and Perplexity-User. Claude's published agents follow the pattern: ClaudeBot, Claude-SearchBot, and Claude-User. Indexing has two hands, training and search, and both of them run before the question. The user bot runs because of the question.

Google is the odd default, and the important one. I wrote up that pipeline in What Google's AI Search Patents Reveal About SEO. Google looks, then speaks. ChatGPT often speaks, then looks if it has to. You will rarely see a user-agent line for an AI Overview, because the retrieval happened inside an index Google already owns. Rank, eligibility, and impressions are the instrument. The chat products leave footprints in the log. Google leaves them in Search Console.

Brand is the prior

When a model answers without looking, it is compressing a chorus. Your homepage is not open. The name, the category, and the repeated description of what you do are already in what it learned. That is brand presence on the internet. Same facts, said the same way, in enough places that a model can repeat them without a fetch.

One strong page does not build that prior. Neither does a wide presence that describes you three different ways. The prior likes agreement. If the open web cannot say what you are, the model will average you into the category and move on.

This is why a mention can happen with no session behind it. The model understood the question, knew a name, and finished. Treat that as real. Also treat it as incomplete. A name in a sentence is not a source a careful buyer can check, and it is not a page the product can quote the day the question gets specific.

Rank is the fetch

ChatGPT, Claude, and Perplexity do not publish Google's ranking formula, and they are not Google. They still retrieve from the web traditional search has been crawling, linking, and ordering for decades. A URL that ranks is a document a system can justify. A URL that doesn't is a claim with no neighborhood.

There's a serious argument that mentions have replaced rank, and half of it is right. A model can name you without fetching you, once the web has repeated you often enough. Mentions do work a blue link used to do alone. The half I don't follow is the conclusion. Mentions shape what the model says when it stays inside its prior. Traditional ranking shapes what it can open when the question forces a lookup. Drop the rank and you drop the fetch, including the fetch inside the chat products that made rank look optional.

The tradeoff is the job. Brand presence is slow and distributed. The specific page is focused, and it expires. Brand without the page gets you named. The page without the brand can win a narrow citation while the model describes your category in other people's words.

What the lookup can quote

Freshness matters because the lookup exists for what memory should not trust. A visible date should be a date you mean. Changing the timestamp on a generic explainer mostly refreshes the chorus.

Once a page is eligible to be fetched, the citation goes to something narrower than "good content."

• The specific case. The constraint has to be in the passage: the size of the team, the rule, the budget, the stage of the company.
• A diversity of situations. Ten articles that repeat one definition teach the model the definition, and then it can stop visiting. Distinct cases give the expanded query something to retrieve.
• Evidence of work. What you compared, measured, shipped, or ruled out. The solution should look like something a person did.
• A passage that can travel. The answer engine lifts sentences. Write the solution so it survives without the introduction wrapped around it.

That is a harder standard than classic on-page SEO, and a fairer one. You are writing source material for a voice that keeps the sentence it can stand behind.

Google still protects the query it can sell

For now, Google still has a reason to send the click when the query is worth an auction. An overview can absorb "what is a CRM." The query that smells like a purchase still wants an ad, a product, a map, a link. Use that. Rankings on commercial intent are still how a lot of businesses meet demand.

Don't build the next two years on the protection holding.

OpenAI is the forcing function. ChatGPT does not need the auction. A finished answer that names a vendor, for a stated case, is the whole product. As more people ask that full question in the chat box, Google's reason to withhold the answer gets weaker. AI Mode is Google moving to that altitude because a competitor will occupy it. The protected query is a lead with an expiry.

The practical reading is boring, which is why it's easy to ignore. Keep the Google rankings that still send the commercial click and still seed everyone else's retrieval. At the same time, assume a buying question will be asked in ChatGPT, Claude, and Perplexity, and that the answer will come from the prior plus whatever those user bots choose to open. Leaving rank because a demo felt finished drops you out of both.

I wrote about the measurement side of this separately, when demand is real and the visit is late, in AI Search Is Not Killing Demand, It Is Killing the Visibility of Demand. The mechanism under that scoreboard is the one here. Memory can satisfy the question. The fetch is the question memory refused.

Be known before the lookup

Watch the two clocks separately. Indexing crawlers, GPTBot among them, tell you whether your work is entering the prior and the retrieval index. User bots tell you which questions are already past memory, and which URLs were worth opening. Add those counts together and you will invent a strategy from noise.

Keep the brand description stable everywhere a crawler can learn it. Put the variance on the page: fresh facts, different cases, evidence of the work, and a solution written so it can be quoted without you in the room.

Answer engine optimization is search, plus a model that sometimes finishes before search is called. Be part of what that model understands. Have a ranked page ready for the day understanding runs out.