Where ChatGPT, Gemini, Claude and Perplexity Get Their Information

AI assistants get their information from two places: what the model learned during training, and a live web search run at the moment you ask. For live answers, Gemini draws on Google's search index, Microsoft Copilot on Bing, ChatGPT on OpenAI's own crawler plus outside search providers that include Bing, Perplexity on its own index, and Claude on a web search service that Anthropic's supplier list says includes Brave Search. If you want to be in those answers, each of these sources has to be able to find and read your website.

That's why a business that only ever thought about Google now needs to think about Bing, a handful of new crawlers, and its robots.txt file.

What's the difference between training data and live search?

Training data is the large snapshot of text a model learned from before release, while live search is a fresh lookup done when someone asks a question. They behave very differently for a business.

Training data is old by the time anyone uses it, and you can't edit it. If your business was barely mentioned online when the snapshot was taken, the model may know little or nothing about you. Live search is current. When ChatGPT, Gemini, Claude or Perplexity searches the web, reads a few pages and writes an answer with links, a page you published last month can be one of those links.

Many questions about a business ("best solar installer in Kumasi", "who does tracking setup in Accra", "is X legit?") tend to trigger a live search in the major assistants, because the answer depends on current information. That's the part you can influence fastest.

Which search index does each assistant use?

Each assistant relies on a different index for live answers, and the company documentation is the best guide to which one.

A few details sit behind that table, and they matter.

ChatGPT

OpenAI's crawler documentation says OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features", and that sites which opt out of it "will not be shown in ChatGPT search answers". OpenAI's help centre also says ChatGPT search sends rewritten queries to third-party search providers, and its page on search for Enterprise and Edu names Bing as one of them. OpenAI doesn't publish the exact mix, so the safe approach is to cover both: allow OAI-SearchBot and get indexed in Bing.

Gemini

Google's crawler documentation describes Google-Extended as a control token, not a separate crawler. It manages whether content Google crawls may be used to train future Gemini models and for "grounding (providing content from the Google Search index to the model at prompt time)" in the Gemini app. In other words, Gemini's live answers come from Google's own index, and blocking Google-Extended can stop your pages being used for them.

Claude

Anthropic's support article lists three bots. ClaudeBot collects content that may be used for training. Claude-SearchBot crawls the web "to improve search result quality for users", and blocking it may reduce your visibility in Claude's search results. Claude-User fetches pages when a person asks Claude something that needs them. Anthropic hasn't published a full description of how Claude's web search works, but its list of subprocessors has named Brave Search as a web search provider since March 2025, as TechCrunch reported at the time.

Perplexity

Perplexity runs its own search index. Its engineering team wrote in September 2025 that the index tracks over 200 billion unique URLs. PerplexityBot builds it, and Perplexity's documentation says it "is not used to crawl content for AI foundation models". We cover how it picks sources in how Perplexity chooses its sources.

Copilot

Microsoft's own support documentation says Copilot in Bing grounds its answers in Bing web search results and links to the sites it uses. If Bing can't find you, Copilot can't cite you.

Training crawlers and search crawlers are not the same thing

Most AI companies run separate bots for training and for search, so you can block one without losing the other.

• GPTBot (OpenAI) gathers content that may train its models. OAI-SearchBot powers ChatGPT search.
• ClaudeBot (Anthropic) is for training. Claude-SearchBot is for search.
• Google-Extended controls Gemini training and Gemini app grounding. Googlebot controls Google Search, including AI Overviews and AI Mode.
• PerplexityBot is for Perplexity's search index, and Perplexity says it isn't used for model training.

This is where many businesses go wrong. A website that blocks "all AI bots" with one rule, or a hosting dashboard with an AI-blocking switch turned on, can quietly remove the business from ChatGPT and Perplexity answers. On the other hand, a publisher that doesn't want its articles used for training can block GPTBot and ClaudeBot and still appear in search answers. Google's crawler documentation is explicit that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

For most businesses that want customers, the sensible default is to allow all of them. MediaWura's own robots.txt allows Googlebot, Google-Extended, Bingbot, Applebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User and others by name, and blocks only a few private paths such as /api/.

What about the "user" fetchers?

ChatGPT-User, Claude-User and Perplexity-User visit a page because a person asked a question that needed it. They don't build an index. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says its user fetcher "generally ignores robots.txt rules". Anthropic says its bots respect robots.txt. Either way, these visits are a good sign: someone asked an assistant about something your page answers.

Why Bing Webmaster Tools matters again

Bing now feeds at least two major assistants, which makes Bing Webmaster Tools one of the most useful free tools for AI search.

Copilot is built on Bing, and OpenAI names Bing among ChatGPT's search providers. Yet plenty of businesses set up Google Search Console and never open Bing Webmaster Tools. If Bing hasn't indexed your service pages, you're giving up a route into two widely used assistants.

Setting it up usually takes under an hour:

1. Create a Bing Webmaster Tools account and add your site. You can import it straight from Google Search Console.
2. Submit your XML sitemap.
3. Check the indexing reports for pages Bing has found but not indexed, and fix the reasons.
4. Turn on IndexNow, which Bing supports, so new and updated pages are reported to Bing as soon as you publish them. Many website platforms and SEO plugins have it built in.
5. Open the AI Performance report. Microsoft launched it as a public preview in February 2026, and it shows how often your pages are cited in Copilot, Bing's AI summaries and some partner services.

That last report is one of the few places any AI platform shows you citation data for your own site. Pair it with the monthly checks in how to measure your visibility in ChatGPT, Gemini and Perplexity.

How to make sure every assistant can read your site

The goal is simple: no crawler you care about should be blocked, and every important fact about your business should be readable as plain text.

1. Read your robots.txt. Open yoursite.com/robots.txt. Look for Disallow: / under any of the bots in the table above, or under User-agent: *.
2. Check your hosting and CDN settings. Some security services offer one-click AI bot blocking. Make sure it isn't on unless you meant it to be.
3. Get indexed in Google and Bing. Verify your site in both Search Console and Bing Webmaster Tools.
4. Put key facts in text. Prices, locations, opening hours and service areas inside images or PDFs are harder for any crawler to use.
5. Keep facts consistent. Assistants compare what your site says with your Google Business Profile, directories and press. See entity SEO.
6. Consider an llms.txt file. Google's AI features documentation says you don't need AI text files to appear in AI Overviews or AI Mode, so treat llms.txt as an optional extra for other tools, not a requirement. See what llms.txt is.

None of this guarantees a mention. It removes the reasons an assistant couldn't mention you. Being chosen still depends on what's on the page and what others say about you, which we cover in how to get recommended by ChatGPT and generative engine optimisation.

Common questions

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search. OpenAI's documentation says GPTBot covers training, while OAI-SearchBot decides whether your site can appear in ChatGPT search answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search, but it can keep your content out of Gemini app answers.

Can I submit my site to ChatGPT or Perplexity?

Neither OpenAI's nor Perplexity's crawler documentation describes a submission form. The route in is allowing their crawlers, being indexed in the search engines they draw on, and publishing pages worth citing.

How long do robots.txt changes take to work?

OpenAI says changes for OAI-SearchBot take about 24 hours, and Perplexity says up to 24 hours. Being crawled and then cited can take much longer, and isn't guaranteed.

Why does an AI assistant describe my business wrongly?

Usually because it's working from old training data or from thin, inconsistent pages. Clear, current pages that live search can find are the fix.

If you'd like us to check which assistants can read your site and fix what's blocking them, see our AI search optimisation service or book a free strategy call.