SEO ToolsTechnical SEOFree Tool
robots.txt Analyzer
Paste in your existing robots.txt content to see its User-agent, Allow, Disallow, and Sitemap rules summarized, with a warning if you're accidentally blocking all crawlers.
How to Use robots.txt Analyzer
- 1Visit yoursite.com/robots.txt and copy its full content
- 2Paste it into the input box here
- 3Click Analyze
- 4Review the parsed rules and fix any accidental "Disallow: /" you didn't intend
Frequently Asked Questions
No — visit yoursite.com/robots.txt in a browser, copy the content, and paste it in here. The analyzer works on the text you provide, not a live fetch.
It parses every User-agent:, Allow:, Disallow:, and Sitemap: line and lists them, then flags the single most damaging mistake: a bare "Disallow: /" that blocks your entire site from being crawled.
Not required by the standard, but adding a Sitemap: line to robots.txt is a widely recommended best practice so crawlers can discover it easily.
No — this only checks robots.txt syntax and rules, not your actual indexing status, meta robots tags, or canonical setup.
Related Tools
Sitemap Generator
Paste a list of your page URLs and get back a valid XML sitemap, ready to upload and submit to Search Console.
Sitemap Splitter
Paste a large sitemap or URL list and split it into 500-URL chunks plus a sitemap index, matching Google's recommended per-file limit.
robots.txt Generator
Enter your domain to generate a starter robots.txt with a sitemap reference — a fixed, standard rule set you can customize further by hand.
.htaccess Redirect Generator
Enter old-path,new-path pairs and get back ready-to-paste Apache mod_rewrite 301 redirect rules for your .htaccess file.