Question 1

Where does robots.txt go on a site?

Accepted Answer

Serve it at the site root: https://example.com/robots.txt. Subfolder files are ignored. Use HTTPS and a text/plain content type.

Question 2

Does robots.txt hide pages from search results?

Accepted Answer

No. Disallow stops well-behaved crawlers from fetching a URL. The URL can still appear in results if it is linked elsewhere. Use noindex if you need that.

Question 3

Which AI crawlers does the block preset cover?

Accepted Answer

The preset disallows GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, anthropic-ai, Bytespider, PerplexityBot, Applebot-Extended, cohere-ai, and several others. Crawlers that ignore robots.txt will not honor this file.

Question 4

Can I allow search engines but block AI training bots?

Accepted Answer

Yes. Keep User-agent: * Allow: / for general crawlers, then add separate User-agent groups that Disallow: / for the AI bots you want to refuse.

Robots.txt Generator