AI crawler check

free tool · ceeme

If your robots.txt blocks AI crawlers, your site cannot show up in AI answers — no matter how good it is. This check reads your robots.txt and tells you, per AI crawler, whether it may enter, plus whether you already have an llms.txt.

Also try: llms.txt generator · OG & meta preview

Why would a site block AI crawlers?

Often by accident: security plugins and old robots.txt rules block everything they don't recognise. The result: ChatGPT, Claude and Perplexity can't read your content — and what they don't know, they don't recommend.

Which crawlers does this tool check?

GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Bingbot (Copilot) and CCBot (Common Crawl, which feeds many AI models).

My site blocks a crawler — now what?

Adjust your robots.txt so the AI crawlers you want no longer fall under a block, or ask your web builder. The free ceeme scan then measures whether you actually show up in the answers.