🤖
Robots.txt Generator
Create a robots.txt file for your website. Control which search engines can crawl your pages, block AI bots, and manage sitemaps.
🕷️ Rule
📁 Common Paths to Block
🗺️ Sitemaps
📄 robots.txt
💡 Upload this file to the root of your website: https://yoursite.com/robots.txt
📚 About robots.txt
- • robots.txt tells search engine crawlers which pages to index
- • Place the file at the root of your domain
- • Use
Disallow: /to block all crawling - • Use
Disallow:(empty) to allow all crawling - • Block AI bots like GPTBot and CCBot to protect your content
- • Always add your sitemap URL for better indexing
Robots.txt Generator
Our free Robots.txt Generator helps you create a properly formatted robots.txt file. Configure user-agent rules, disallow/allow paths, set crawl delays, add sitemap URLs, and block AI crawlers like GPTBot and CCBot. Includes presets for WordPress, e-commerce, and AI bot blocking.
🔥 Popular Tools
FreeWebTools AI
Powered by free AI models · Full chat →