SEO ToolsTechnical SEOFree Tool
robots.txt Generator
Enter your domain to generate a starter robots.txt with a sitemap reference — a fixed, standard rule set you can customize further by hand.
How to Use robots.txt Generator
- 1Enter your domain name
- 2Click Generate
- 3Review the Allow/Disallow rules and edit if you need to block additional paths
- 4Save the result as robots.txt at your site root
Frequently Asked Questions
A fixed template: "User-agent: *", "Allow: /", your Sitemap: line, and "Disallow: /admin/". Only the domain (used in the Sitemap: line) is generated from your input.
Not directly from this generator — the Allow/Disallow rules are fixed. Add extra Disallow: lines to the output manually for any other paths you want to block.
Save it as robots.txt in your site's root directory (e.g. https://yoursite.com/robots.txt) so crawlers can find it automatically.
No — the default template allows crawling of everything except /admin/. Double-check it against the Robots.txt Analyzer before publishing if you're unsure.
Related Tools
Sitemap Generator
Paste a list of your page URLs and get back a valid XML sitemap, ready to upload and submit to Search Console.
Sitemap Splitter
Paste a large sitemap or URL list and split it into 500-URL chunks plus a sitemap index, matching Google's recommended per-file limit.
robots.txt Analyzer
Paste in your existing robots.txt content to see its User-agent, Allow, Disallow, and Sitemap rules summarized, with a warning if you're accidentally blocking all crawlers.
.htaccess Redirect Generator
Enter old-path,new-path pairs and get back ready-to-paste Apache mod_rewrite 301 redirect rules for your .htaccess file.