robots.txt Input
Validation Results
Paste your robots.txt and click Validate to see results.
Paste your robots.txt file to validate syntax, check directives, and inspect crawl rules — runs entirely in your browser.
Paste your robots.txt and click Validate to see results.
https://yoursite.com/robots.txt) and copy its contents.A robots.txt file tells search engine crawlers which pages or sections of your site they should or should not request. Syntax errors or misconfigured rules can accidentally block Google or Bing from indexing important pages — or allow them to crawl pages you meant to hide. This validator parses your file line by line, checks for common mistakes like missing User-agent declarations, invalid path patterns, and unreachable Allow rules, and displays each group of directives in a structured view. All validation runs locally in your browser; your file is never uploaded anywhere.
No — it is a polite request, not a technical barrier. Compliant crawlers like Googlebot respect it, but a malicious bot can still visit disallowed URLs. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead.
Disallow: with an empty value mean?An empty Disallow: value (just Disallow: or Disallow: ) means the crawler is allowed to access all pages — it is equivalent to no restriction. This is the opposite of Disallow: /, which blocks the entire site.
When a crawler matches both a specific User-agent group and the wildcard User-agent: * group, the most specific rule wins. Google's crawlers follow this convention; always prefer a named group for precise control.