Obi GEO

Get free account

Guide

How to check whether ChatGPT can read your website

jotajota

To check whether ChatGPT can read your website, test three things: whether your robots.txt allows OpenAI’s crawlers, whether your firewall lets them through, and whether your text appears without JavaScript. You can check each by hand in about ten minutes, or ask your AI app to run Obi GEO’s free check.

1. Read your robots.txt

Open https://yoursite.com/robots.txt in your browser. Look for:

  • A group for OAI-SearchBot or ChatGPT-User with Disallow: /. That blocks ChatGPT search or page fetches.
  • A group for GPTBot with Disallow: /. That only blocks training, which is a choice, not a problem.
  • User-agent: * with Disallow: /. That blocks everyone, usually left over from when the site was being built.

If there’s no robots.txt at all, crawlers are allowed by default. Which crawlers to allow explains the choices.

2. Check your firewall

Firewalls decide by more than the crawler’s name, so you can’t fully test them from your own browser. Open your host’s or CDN’s security settings and look for bot protection, “block AI bots” options, or challenge pages for unknown visitors. If verified AI crawlers are challenged, they see a challenge page instead of your content.

3. Look at your page without JavaScript

Many AI crawlers read the page as it arrives, without running scripts. To see what they see in Chrome:

  1. Open DevTools. Open the page and press F12.
  2. Open the command menu. Press Ctrl+Shift+P (Cmd+Shift+P on a Mac).
  3. Turn off JavaScript. Type “Disable JavaScript”, press Enter, and reload the page.

If your services, prices or contact details disappear, crawlers that don’t run JavaScript can’t read them. A quicker test: view the page source and search for a sentence from the page.

4. Ask ChatGPT directly

With web search on, ask ChatGPT to summarize one of your pages by its address. If it can’t open it, or answers from other websites, something is in the way. It’s a quick reality check, not a prediction of what ChatGPT will recommend.

Or run the free check

Obi GEO checks all of this, plus Claude, Perplexity and Google’s crawlers, structured data, speed and how clear each page is. Create a free account, connect your AI app, and ask:

Inspect https://yoursite.com and tell me what stops AI assistants from reading or recommending it.

Sources

Related: what is GEO? ยท the free AI visibility check

See what AI sees on your site

Create a free account, connect Claude or ChatGPT, and ask it to inspect your website. Three checks a day, no card.

Get a free account

More guides