Robots.txt Checker

What is a robots.txt checker?

A robots.txt checker reads the robots.txt file published at the root of a website and helps you understand which paths a crawler is allowed or blocked from accessing. It can be useful when troubleshooting crawling, technical SEO and unexpected search-engine access.

Test major search-engine crawlers

This checker lets you test rules for major crawler user agents, including Googlebot, Googlebot-Image, Googlebot-News, Bingbot, MicrosoftPreview, YandexBot, Baiduspider, DuckDuckBot, Slurp, Yeti, Applebot, PetalBot, SeznamBot and the wildcard *. Microsoft legacy msnbot is also available for compatibility testing. Bing documents Bingbot and other Microsoft crawlers, while Google documents its crawler user-agent groups and robots.txt interpretation.

Check whether a URL is allowed or blocked

Enter your website and the path you want to test, then select the crawler. The checker identifies the applicable user-agent group and evaluates its allow and disallow rules against the requested path. A specific crawler group can take precedence over the wildcard group, so testing the crawler you care about can reveal a different result from a general User-agent: * rule.

Allow and disallow rules

Robots.txt commonly uses User-agent, Allow and Disallow fields. If a matching disallow rule blocks the requested path, the result is shown as blocked. If an allow rule is the more specific matching rule, the path is shown as allowed. A path with no matching restriction is treated as allowed for the selected group.

Sitemaps in robots.txt

Robots.txt files can also contain absolute Sitemap URLs. The checker lists sitemap entries separately so you can quickly see whether the file advertises one or more XML sitemaps or sitemap indexes. For response-level details, the HTTP Header CheckerCanonical URL CheckerURL Indexability CheckerSERP Preview Checker can inspect the HTTP headers returned by the robots.txt request. Sitemap declarations are not tied to a particular user-agent group.

Robots.txt and technical SEO

A robots.txt rule controls crawling access; it is not the same as a noindex directive. If you need to inspect page-level indexing instructions, use the Meta Noindex Checker. A page can be disallowed from crawling and still have its URL known to a search engine. If you need to control whether a page is indexed, review the page's meta robots directives and relevant HTTP headers as well.

How this robots.txt checker works

RedirectionChecker requests the site's root /robots.txt over HTTP or HTTPS, follows supported redirects within safety limits, resolves the destination to a public IP, parses the returned rules and reports the result for the selected crawler and path. The checker does not require an account and does not intentionally store the submitted URL.

Check robots.txt online

Use the checker above to inspect a public robots.txt file and test crawler access. If the tested URL is redirected, the Redirect Chain Checker can help you inspect the intermediate responses. For standards-based behavior, the tool follows the Robots Exclusion Protocol model and Google's documented robots.txt interpretation for user-agent groups and allow/disallow matching.