- No signup
- Free
- Runs in your browser
How it works
Three steps.
You give it
Tick the AI crawlers you allow, leave the others unticked to block them, and add a sitemap address if you like.
It does
The page builds the file as you choose, entirely in your browser.
You get
A robots.txt file to copy or download, ready for the root of your site.
Why it matters. AI crawler access is the largest check in the score, 26 of 100 points. See how the score is built.
Questions
Common questions.
What is an AI crawler?
An AI crawler is an automated bot that visits web pages to gather content for a large language model, either to train the model or to answer a live user question. This tool covers the 10 most common AI crawlers, including GPTBot, ClaudeBot and PerplexityBot.
Should I block AI crawlers?
It depends on your goals. Blocking a crawler keeps your content out of that system's training data and live answers. Allowing it can put your brand in front of users asking AI tools for recommendations. Most sites benefit from allowing crawlers used for live answers and deciding case by case on training-only crawlers.
Does blocking GPTBot remove me from ChatGPT?
Blocking GPTBot stops OpenAI from using your content to train future models. It does not remove your site from ChatGPT's live answers, which use a separate crawler called ChatGPT-User. To stop ChatGPT from reading your site in real time too, block ChatGPT-User separately.
Next step
Put the file at the root of your site, then scan again to check it. To see where you stand against another site, compare the two.
Use it from Claude. In the MCP server, generate_robots_txt builds the same file from the crawlers you name. See the MCP server.
Not sure what's currently blocking you? Scan your site.
Scan your site free