Robots.txt Generator for AI Crawlers
Robots.txt Generator
Build a robots.txt with your blocked paths and sitemaps, choose which AI crawlers may read your site, and test any URL before you upload the file.
- Free, no sign-up
- 21 AI crawlers
- Built-in tester
- Google rule checks
Your robots.txt
Checks against Google’s rules
Runs in your browser. Nothing is sent or stored.
Robots.txt tester
Check if a crawler may visit a URLStep by step
How to use the robots.txt generator
Pick a starting point
Choose a preset such as “Block AI training, keep AI search” or “WordPress site”. Everything it sets can be changed afterwards.
Add the paths to keep crawlers out of
Add Disallow rules for areas such as checkout, account, internal search or a staging directory. Full URLs, missing slashes and spaces are fixed for you.
Choose which AI crawlers to block
Bots are grouped by what they do: training, AI search and user-triggered visits, so you can block training without disappearing from AI answers or search engines.
Check, test and upload
Clear the checks, test a few URLs with the built-in robots.txt tester, then copy or download the file and upload it to the root of your domain.
Syntax
What each line in a robots.txt file does
Every robots.txt generator writes the same handful of lines. Here is what each one tells a search engine crawler.
| Line | Example | What it does |
|---|---|---|
| User-agent | User-agent: GPTBot | Starts a group of rules for one crawler. * means every crawler without a group of its own. |
| Disallow | Disallow: /checkout/ | Asks the crawler not to visit URLs that start with this path, such as a whole directory. An empty Disallow blocks nothing. |
| Allow | Allow: /wp-admin/admin-ajax.php | Re-opens part of a blocked directory. When two rules match, the longer path wins. |
| Sitemap | Sitemap: https://www.example.com/sitemap.xml | Points crawlers to your sitemap. Must be a full URL. |
| Wildcards | Disallow: /*.pdf$ | * matches any characters and $ marks the end of the URL. |
| Crawl-delay | Crawl-delay: 10 | Not supported by Google. Some other crawlers respect it. |
| Noindex | Noindex: /old/ | Not supported. Use a noindex meta tag on the page instead. |
Groups matter more than most people expect. A crawler follows only the most specific group that names it. If you add a group for Googlebot, it stops reading the rules under User-agent: *, and a blank line does not end a group. The generator writes every group so these rules work the way you intend, following Google’s robots.txt specification and RFC 9309.
AI crawlers
How to block AI bots without blocking Google
Most AI companies now run more than one crawler, and each one does a different job. Blocking the wrong one can remove you from AI search answers when you only meant to opt out of training. That is why this robots.txt generator groups them by purpose.
- Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for future models. Google-Extended and Applebot-Extended are not separate crawlers but control tokens: blocking them tells Google and Apple not to use your pages for AI, while Googlebot and Applebot keep crawling for search.
- AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you appear and get cited in those assistants’ answers.
- User-triggered fetchers visit a page because a person asked an assistant to read it. OpenAI, Perplexity, Meta and Amazon say these may not follow robots.txt.
A typical file that keeps search engines in and two training crawlers out looks like this:
User-agent: * Disallow: User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Crawler names change. The list in the generator was checked against each operator’s documentation on 2 Nov 2026, and every name links to its source.
Installing
Upload and test your robots.txt
The file must be called robots.txt and sit at the root of the host, for example https://www.example.com/robots.txt. Each subdomain needs its own file. Save it as plain UTF-8 text and keep it under 500 KiB, because Google ignores anything after that.
WordPress
WordPress serves a virtual robots.txt. Yoast SEO and Rank Math have a file editor, or upload a robots.txt file to the site root by FTP, which replaces the virtual one.
Shopify
Edit the robots.txt.liquid template in your theme code. Shopify’s default rules stay unless you remove them, so add your lines to them.
Other sites
Upload the file to the web root, the same directory as your homepage, then open yoursite.com/robots.txt to check it loads.
After uploading the file from the generator, paste the live file into the tester above and try a few important URLs. In Google Search Console, the robots.txt report under Settings shows which version Google fetched and any errors it found.
FAQ
Questions about the tool
How do I create a robots.txt file?
Choose a preset in the generator, add the paths you want to block, tick any AI crawlers you want to keep out and add your sitemap. Then copy or download the file and upload it to the root of your domain as robots.txt.
Where do I upload robots.txt?
To the root of each host, so it opens at yoursite.com/robots.txt. A file in a subfolder is ignored, and each subdomain, such as shop.yoursite.com, needs its own file.
How do I block GPTBot and other AI bots without blocking Google?
Tick them in the AI training group of the generator. Each one gets its own User-agent group with Disallow: /, and the rules for every other crawler stay as they are. The built-in checks confirm Googlebot and Bingbot can still crawl your homepage.
Will blocking GPTBot remove my site from ChatGPT?
No. According to OpenAI, GPTBot is for training and OAI-SearchBot decides whether your pages appear in ChatGPT search. Anthropic and Perplexity separate their crawlers the same way, so you can block training and still be cited.
Does robots.txt stop a page from appearing in Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed without its content if other sites link to it. To keep a page out of search results, let Google crawl it and add a noindex meta tag.
Is crawl-delay supported?
Google ignores Crawl-delay. Some other crawlers respect it, including ClaudeBot according to Anthropic, while Amazon says Amazonbot does not. The generator only adds it if you choose a value.
How do I test my robots.txt?
Use the robots.txt tester on this page. Paste your live file or test the one you just built, pick a crawler and enter a URL. It shows whether the URL is allowed and which line decided it, using Google’s rules for groups and the longest match.
What happens if a site has no robots.txt?
Search engines and other crawlers treat a missing file as permission to crawl everything. That is fine for many small sites, but a file with your sitemap and any blocked areas gives you more control.