--- title: "AI Crawler Checker | robots.txt test | Croissant & Baguette" description: "Check whether ChatGPT, Claude, Perplexity and Gemini can read your site: robots.txt, firewall blocks, text without JavaScript and llms.txt. Free." url: "https://croissantbaguette.com.au/tools/ai-crawler-checker/" --- # AI crawler checker https://croissantbaguette.com.au/tools/ai-crawler-checker/ Check whether ChatGPT, Claude, Perplexity and Gemini can read your site: robots.txt, firewall blocks, text without JavaScript and llms.txt. Free. ## Can ChatGPT, Claude and Perplexity read your site? Put in a page. We ask for it as twenty AI crawlers and as an ordinary visitor, and show what your robots.txt tells each one beside what your server actually did. Then the text a crawler gets before any JavaScript runs, and the files written for AI: llms.txt, the sitemap, Content-Signal and Markdown. robots.txt, Firewall blocks, Text without JavaScript, llms.txt ## How the check works. - **Two answers for each crawler.** robots.txt is what a site asks crawlers to do; the server's answer is what it does. A site can welcome GPTBot in one and refuse it in the other, through a firewall rule, a security plugin or Cloudflare's AI blocking. We show both, side by side. - **An ordinary visit to compare with.** We ask for the page as a normal browser first. If that is refused too, the site turns robots away in general, and the checker says so instead of blaming the AI crawlers. - **The page as a crawler reads it.** The crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript, so we count the words in the page as it is served. A page built in the browser can show them almost nothing. - **Twenty crawlers, from their makers' own pages.** Every name, user agent and robots.txt promise comes from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, Common Crawl and ByteDance, read on 12 October 2026 and checked again each quarter. - **A strong hint at the server, not proof.** We ask from our servers, not OpenAI's, so a firewall that checks where a crawler comes from can treat the real one differently. Your firewall's log for the crawler's name settles it. ## Ask away. Seven things people ask about AI crawlers and this checker. Still one missing? [Ask us](https://croissantbaguette.com.au/contact/). - **How do I check if my website blocks AI crawlers?** Put the page into the checker above: in about ten seconds it shows, for twenty AI crawlers, what your robots.txt says and what your server did when we asked as each one. By hand, you would read /robots.txt for each crawler's name and for the rules that apply to every crawler, then request the page with each crawler's user agent and compare the answer with an ordinary visit. - **Does blocking GPTBot keep my site out of ChatGPT?** No. GPTBot collects pages that may train OpenAI's models. ChatGPT's search results come from OAI-SearchBot, and ChatGPT-User opens a page when someone asks ChatGPT to, so a site can block GPTBot and still appear in ChatGPT's answers. Block OAI-SearchBot and ChatGPT search stops showing your pages. OpenAI documents the three separately, and each follows its own robots.txt rule. - **Is Cloudflare blocking AI crawlers on my site?** It might be. Since 1 July 2025 Cloudflare has blocked AI crawlers by default on newly added domains, and its managed robots.txt disallows GPTBot, ClaudeBot, Google-Extended and others. The checker shows both: a robots.txt that names those crawlers, and a refusal or a challenge when we ask as one. Both are turned on and off in the Cloudflare dashboard, under AI Crawl Control. - **Do AI crawlers run JavaScript?** Most do not. A study by Vercel and MERJ in December 2024 found that OpenAI's, Anthropic's and Perplexity's crawlers fetch JavaScript files but never run them, while Googlebot and Applebot render pages. So a site whose text appears only after its scripts run can look almost empty to ChatGPT and Claude. The checker counts the words in your page as served, before any JavaScript. - **Will an llms.txt file get my business into AI answers?** Nothing shows that it does. An llms.txt is a short Markdown guide to a site for AI agents, and none of the major assistants has said it uses one to choose answers. It costs little to keep, so the checker reports whether you have one, but letting the crawlers in and serving readable text matter far more. - **What does the checker send to my site?** About twenty-five ordinary requests in a few seconds: one as a normal browser, one as each AI crawler that fetches pages, and one each for robots.txt, llms.txt, the sitemap and a Markdown version of the page. Nothing is posted, no form is filled and no page beyond the one you name is opened. Each request carries Cloudflare's CF-Worker header, naming our site. - **Why does the checker say it can't tell?** Because your site turned our ordinary visit away too, or asked us to slow down. When a server refuses a normal browser request from our servers, its refusal of GPTBot says nothing about AI crawlers in particular, so the checker reports what robots.txt says and leaves the live test open. Your firewall's log, searched for the crawler's name, gives the real answer. ## Sources Every crawler's name, user agent and robots.txt promise was read on its maker's own page on 12 October 2026. - **[Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)** OpenAI · GPTBot, OAI-SearchBot and ChatGPT-User - **[Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)** Anthropic · ClaudeBot, Claude-User and Claude-SearchBot - **[Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)** Perplexity · PerplexityBot and Perplexity-User - **[Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)** Google · Google-Extended - **[AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)** Google · what keeps a page out of AI Overviews and AI Mode - **[About Applebot](https://support.apple.com/en-us/119829)** Apple · Applebot and Applebot-Extended - **[Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)** Meta · meta-webindexer, meta-externalfetcher and meta-externalagent - **[Amazonbot and Amazon's other crawlers](https://developer.amazon.com/amazonbot)** Amazon · Amzn-SearchBot, Amzn-User and Amazonbot - **[DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)** DuckDuckGo · DuckAssistBot - **[CCBot](https://commoncrawl.org/ccbot)** Common Crawl · CCBot - **[The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler)** Vercel and MERJ, December 2024 · which AI crawlers run JavaScript - **[The free report.](https://croissantbaguette.com.au/tools/ai-visibility/)** Whether ChatGPT, Claude, Gemini and Google's AI name you when your customers ask. - **[Check ChatGPT by hand.](https://croissantbaguette.com.au/guides/how-to-check-if-chatgpt-mentions-your-business/)** The questions to ask, the engines to ask them in, and what to write down. - **[Technical SEO.](https://croissantbaguette.com.au/ai-visibility-seo/technical-seo/)** Crawler access, speed, structured data and redirects, fixed in the site's own code. - **[SEO & AI Visibility.](https://croissantbaguette.com.au/ai-visibility-seo/)** The free audit, the monthly programme, and the number that says whether AI names you. ## Kept out by mistake? It is usually one firewall setting or one line in robots.txt. Tell us the site in two lines, and a person replies within a business day.