Генератор Robots.txt

📚 About robots.txt

• • robots.txt tells search engine crawlers which pages to index
• • Place the file at the root of your domain
• • Use Disallow: / to block all crawling
• • Use Disallow: (empty) to allow all crawling
• • Block AI bots like GPTBot and CCBot to protect your content
• • Always add your sitemap URL for better indexing