CodeKitHubWhat Is This Tool?
robots.txt is a plain-text file at your site's root that tells web crawlers which parts of the site they may or may not access. Search engines like Google and Bing respect it for indexing; in 2026 it's also the standard way sites opt out of having their content used to train AI models, via specific bot names like GPTBot, CCBot and Google-Extended.
Getting the syntax exactly right matters — a wrong Disallow path can accidentally block search engines from your whole site, or an AI-blocking rule placed incorrectly can fail to actually block anything. This generator produces syntactically correct output for each scenario.
After almost three decades as an informal convention dating back to 1994, the Robots Exclusion Protocol was finally standardized as RFC 9309 in 2022 — the rules this generator emits (User-agent, Disallow, Allow, plus the Sitemap extension) follow that specification.
Why Use It?
- A dedicated AI-crawler-blocking preset covering GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, Bytespider and anthropic-ai — while explicitly keeping search engine access.
- Custom disallow paths for content you want hidden from all crawlers.
- Sitemap URL field — a small addition that helps search engines discover your pages faster.
- Correct syntax every time — no risk of an accidental typo blocking your whole site.
- Free, instant, download the file directly or copy the text.
How to Use
- Pick a preset: allow all, block all, or block AI crawlers only.
- Optionally add custom disallowed paths, one per line (e.g. /admin/).
- Optionally add your sitemap URL.
- Click Copy or Download, then upload the file to your site's root as /robots.txt.
Example
Input
Preset: Block AI crawlers, keep search engines. Sitemap: https://example.com/sitemap.xmlOutput
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
...
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlEach AI crawler gets its own explicit block, while the wildcard rule at the end keeps normal search engine access open.
Should you block this crawler?
Not all bots serve the same purpose — blocking the wrong one can cut off search traffic instead of just opting out of AI training.
| Crawler | What it does | Block it if… |
|---|---|---|
| Googlebot, Bingbot | Indexes your site for search results | Never — blocking these removes you from Google/Bing search entirely. |
| GPTBot, ChatGPT-User | OpenAI's crawlers — GPTBot trains models, ChatGPT-User fetches pages a user asked ChatGPT to browse | You don't want your content used for AI training, but note ChatGPT-User browsing a page a user linked is a different case some sites choose to allow. |
| Google-Extended | Controls whether Google uses your content for Gemini/AI training — separate from Googlebot's search indexing | You want to stay in Google Search but opt out of Google's AI training — this directive does exactly that without hurting rankings. |
| CCBot, Bytespider, anthropic-ai, ClaudeBot | Common Crawl (feeds many AI datasets), ByteDance, Anthropic's crawlers | Same reasoning as GPTBot — block if you don't want content in AI training sets. |
Frequently Asked Questions
Does robots.txt actually stop AI from using my content?
It works on an honor system — reputable AI companies' crawlers do respect it (that's why GPTBot, CCBot etc. exist as distinct, documented user-agents specifically so sites can opt out). It does not stop crawlers that ignore the standard, and it has no effect on content already scraped before you added the block.
Will blocking AI crawlers hurt my Google/Bing search ranking?
No — the AI-crawler preset specifically blocks only the named AI bots and leaves a wildcard "Allow: /" for everyone else, so Googlebot and Bingbot are unaffected.
Where do I put the robots.txt file?
At the root of your domain, so it's reachable at yourdomain.com/robots.txt exactly — not in a subfolder. Most website hosts and static site generators have a designated public/root folder for this.
What's the sitemap line for?
It points crawlers to your sitemap.xml, which lists all your indexable pages. It's optional but commonly recommended — one extra line that can speed up how quickly new pages get discovered.
Can I combine custom disallow paths with the AI-blocking preset?
Yes — the AI-blocking preset adds AI bot blocks first, then applies your custom disallow paths (if any) under the general wildcard rule that still allows search engines.