How to check if AI assistants can read your website

Q: Which AI crawlers should I allow if I want to appear in AI answers?

At minimum, do not block the retrieval crawlers assistants use to fetch pages when answering, such as OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), and PerplexityBot. Training-only crawlers like GPTBot or CCBot are a separate decision.

Q: How do I test my site quickly?

Three checks catch most problems: fetch your page with curl to confirm the raw HTML contains your content, check robots.txt for AI user agents, and confirm your key facts appear as text rather than only in images or JavaScript-rendered widgets.

7 min readLast verified 01 Jul 2026audited with seoprog · free check below

More people ask ChatGPT, Claude and Perplexity for recommendations every month, and those assistants can only mention you if they can read you. This guide walks through exactly how AI systems access your site, the four places access usually breaks, and how to test each one in minutes.

How AI assistants actually read your site

Two different things get called "AI reading your site," and they have different rules:

Training crawls collect pages to train future models. OpenAI's GPTBot, Google's Google-Extended and Common Crawl's CCBot are in this group. Blocking them keeps your content out of training data but does not stop assistants from citing you live.
Retrieval fetches happen when an assistant answers a question right now. ChatGPT browsing uses OAI-SearchBot and ChatGPT-User, Anthropic uses ClaudeBot, Perplexity uses PerplexityBot. Block these and you disappear from AI answers - this is the visibility that matters for most small sites.

Retrieval also leans on classic search indexes. ChatGPT's browsing draws heavily on Bing's index, so a site absent from Bing is largely invisible to it regardless of robots rules. If you have not registered in Bing Webmaster Tools, that is step zero.

The four places access breaks

1. robots.txt blocks more than you intended

Check yoursite.com/robots.txt. Look for User-agent: lines naming the crawlers above, and for a blanket User-agent: * Disallow: / left over from a staging setup. Many "why am I not in ChatGPT" cases end here. Decide deliberately: block training bots if you wish, but leave retrieval bots open if you want to appear in answers.

2. Your firewall or CDN challenges bots

Bot-protection settings on Cloudflare and similar services can serve AI crawlers a challenge page instead of your content. The page looks fine in your browser while every crawler sees a wall. Test with a plain fetch:

curl -A "ClaudeBot" -sI https://yoursite.com/ | head -5

A 200 is good. A 403 or a challenge redirect means your protection layer is the problem, not your content.

3. The content only exists after JavaScript runs

Most AI crawlers read raw HTML and do not execute JavaScript the way Google can. If your page is an empty shell that hydrates client-side, assistants see the shell. Test it: curl -s https://yoursite.com/ | grep "a phrase from your page". If your actual copy is missing from the response, AI systems cannot quote you. Server-render or pre-render the pages you care about.

4. Key facts live in images

Prices in a screenshot, opening hours in a banner image, product specs in a PNG table - invisible. Anything you want an assistant to repeat about you must exist as text in the HTML.

Make yourself easy to quote

Passing the access checks gets you read. Getting cited well needs the page itself to answer questions cleanly:

State the important facts in plain sentences near the top: what you do, who it is for, what it costs.
Use real headings that match questions people ask - assistants lift answers from well-structured sections.
Add JSON-LD for your organization and content type, and keep visible dates on articles. Machines trust pages they can parse and date.

Five-minute test: ask an assistant with browsing enabled to "read yoursite.com and summarize what this company offers and what it costs." If the summary is wrong or empty, one of the four breaks above is the reason.

Frequently asked questions

Does blocking AI crawlers hurt my Google rankings?

No. Google web search uses Googlebot, which is separate from AI training crawlers like GPTBot or Google-Extended. You can block AI training and still rank normally. The trade-off is visibility inside AI assistants, not in classic search.

Which AI crawlers should I allow if I want to appear in AI answers?

At minimum, do not block the retrieval crawlers: OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot. Training-only crawlers like GPTBot or CCBot are a separate, values-based decision.

How do I test my site quickly?

Fetch your page with curl and confirm your copy is in the raw HTML, check robots.txt for the AI user agents, and make sure key facts are text, not images. Or run the free check below - it does all of this in one pass.

See how your page scores

Run the free check - it reads your page the way Google and AI assistants do and gives you a plain-English fix list. No sign-up for your first check.

Check a page free