Free tools
Robots.txt Generator
Create a robots.txt policy with sitemap discovery.
Processed in your browser. Your input is not uploaded.
Input
Paste text or read a local text file. Input is never saved in browser storage.
Ctrl / Cmd + Enter to runResult
Your result will appear here.
About this tool
Prepare crawler instructions for a website as a text file. User-agent is configured separately from the rules, and an optional sitemap URL can be included without editing the generated text manually.
When to use it
- Prepare crawler rules for a new site.
- Review Allow and Disallow paths.
- Include an existing sitemap address.
How to use it
- Set the User-agent and enter rules or comments.
- Add a valid sitemap URL if you have one.
- Generate, review and place robots.txt at your website root.
Input and output example
Input
User-agent: * Rules: Allow: /; Disallow: /private/
Output
User-agent: * Allow: / Disallow: /private/
Important details
- Robots rules are crawler instructions, not authentication or access control.
- Blocking crawling is not a reliable substitute for a noindex directive.
- Crawlers may support different extensions; check your target crawler before using Crawl-delay.
Frequently asked questions
Does robots.txt protect private files?
No. Protect confidential resources with actual access controls rather than relying on crawler compliance.
Will the generator submit the file to a search engine?
No. It generates a local text file for you to deploy and verify.