Robots.txt Generator
Create a robots.txt file for your website. Control which search engines can crawl your pages, block AI bots, and manage sitemaps.
⚠️ Enter URL (https://example.com)
🕷️ Rule
📁 Common Paths to Block
🗺️ Sitemaps
📄 robots.txt
💡 Upload this file to the root of your website: https://yoursite.com/robots.txt
📚 About robots.txt
- • robots.txt tells search engine crawlers which pages to index
- • Place the file at the root of your domain
- • Use
Disallow: /to block all crawling - • Use
Disallow:(empty) to allow all crawling - • Block AI bots like GPTBot and CCBot to protect your content
- • Always add your sitemap URL for better indexing
Robots.txt Generator
Our free Robots.txt Generator helps you create a properly formatted robots.txt file. Configure user-agent rules, disallow/allow paths, set crawl delays, add sitemap URLs, and block AI crawlers like GPTBot and CCBot. Includes presets for WordPress, e-commerce, and AI bot blocking.
🔥 Popular Tools
ℹ️ About Robots.txt Generator
The Robots.txt Generator builds a crawler directive file rule by rule: pick a User-agent, add Allow and Disallow paths, then set a Crawl-delay or one or more Sitemap URLs. Checkboxes cover the paths sites commonly block, such as /wp-admin/, /cart/ and /*.pdf$, and presets handle allow-all, block-all, WordPress, e-commerce, and blocking AI crawlers like GPTBot, CCBot and anthropic-ai. The file is assembled in your browser and can be copied or downloaded as robots.txt.
💡 Use Cases
📋 How to use
Start from a preset if one fits your site: Allow All, Block All, WordPress, Block AI Bots or E-commerce, then edit what it produced.
Fill in Site URL with your domain, for example https://example.com, so the sitemap line can be built from it.
In each rule choose a crawler from the User-Agent dropdown, then use + Add Disallow and + Add Allow to enter paths such as /admin/; add more crawlers with + Add User-Agent Rule.
Tick entries under Common Paths to Block and press Apply to Rule 1, and set Crawl Delay in seconds if you want to slow a crawler down.
Under Sitemaps keep Auto-add /sitemap.xml ticked or use + Add Sitemap to list extra sitemap URLs.
Check the robots.txt preview on the right, then click Download or Copy and upload the file to the root of your domain.
💎 Pro Tips
- Paste your content directly — the Robots Txt Generator detects common input patterns and applies sensible defaults automatically.
- Use keyboard shortcuts (Ctrl/Cmd+A to select all, Ctrl/Cmd+C to copy) to speed up your workflow inside the Robots Txt Generator.
- For very large inputs, break them into chunks — most browser tabs can handle megabytes of text, but parsing huge payloads may freeze the UI briefly.
- The Robots Txt Generator does not log or store your input. If you need history, copy important outputs to a local file or note app.
- Combine the Robots Txt Generator with other tools on this site (formatters, converters, encoders) to build complete data pipelines without leaving your browser.
❓ Frequently Asked Questions
Yes, fully free with no signup, no API key, no watermarks, and no usage limits.
No. The Robots Txt Generator runs entirely client-side in your browser. Your input never touches our servers.
There is no hard limit, but expect slowdowns on inputs larger than a few megabytes, depending on your device's memory.
Yes. Outputs generated with the Robots Txt Generator are yours to use freely in any personal or commercial project.
Most of the logic runs in client-side JavaScript, so after the initial load you can often keep working without a network connection.
None. Because the Robots Txt Generator runs in your browser, you can process as many inputs as your hardware allows without API keys or throttling.