← All tools
Robots.txt Generator
Generate crawler directives from disallowed paths and an optional sitemap URL.
Loading Robots.txt Generator…
Using Robots.txt Generator
Which inputs does Robots.txt Generator support?
Enter paths one per line, a public sitemap URL, and whether to add the named AI-crawler allow groups.
What is an example of the result?
A disallowed path /private produces Disallow: /private under User-agent: *.
Which limits or failure conditions matter?
robots.txt is a crawler instruction file, not access control. Named AI-crawler allow groups can override the generic restrictions for those agents; publish only the policy you intend.
Where is this task processed?
Your input is processed in this browser. Application libraries may be downloaded on first use; the entered content is not sent to a conversion service.