FREE SEO TOOLS · 9 FREE TOOLS · NO SIGNUP · MARKET ME GOOD · MARKET ME GOOD
Robots.txt Generator · Free Tool · Market Me Good
FREE TOOL · ROBOTS.TXT
Robots.txt Generator
Configure crawler access, block private paths, add sitemaps. Includes AI-crawler blocking presets (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider) — 2026 relevant.
Configure options above, then click Generate.
What robots.txt does
A plain-text file at your domain root that tells crawlers which URLs they can and cannot access. Google, Bing, DuckDuckGo, and most AI crawlers respect it. Doesn't block indexation of URLs discovered via backlinks — for that use noindex meta tags.
Common mistakes
- Disallow: / blocks your entire site — used only on staging.
- Blocking CSS/JS — Google needs these to render pages. Never block
/wp-content/blindly. - No sitemap — always declare your sitemap(s) at the bottom.
Want the full toolkit — not just one calculator?
Our SEO Engine ships 20 optimised articles / month with schema, backlinks, and rankings baked in. 13 years of receipts.
Book a call →