Robots.txt Checker

robots.txt is a public text file a website can publish to tell crawlers which paths they may request. This checker summarizes that file. It does not crawl the site.

It lives at the site origin, as /robots.txt. A page such as https://example.com/blog/page is checked at https://example.com/robots.txt, not at a path under /blog.

User-agent names the crawler a group of rules applies to, such as * for all crawlers or Googlebot for Google’s crawler.

Disallow lists paths the named crawler is asked not to request. An empty Disallow value is a common way to allow all paths for that group.

Allow lists paths that remain permitted inside a group that also has Disallow rules. This checker reports the declared values. It does not apply crawler precedence.

Crawl-delay is an optional directive some crawlers treat as a pause between requests, shown here in seconds when the value is numeric.

A Sitemap line declares a sitemap URL. This checker lists those declarations. It does not fetch or validate the sitemap files.

No. robots.txt can ask crawlers not to request a path. It does not by itself hide a page from search results.

If the server returns 404 or 410 for /robots.txt, this checker reports Not found. That is a normal website outcome, not a tool failure.

No. This version summarizes the robots.txt file. A path-specific robots tester is a separate capability.