SEO CRAWLER DIRECTIVE TOOL
Meta Robots Noindex Checker
Check HTML meta robots tags and X-Robots-Tag headers to find noindex and crawler directives returned by a URL.
HTML meta robots directives
X-Robots-Tag HTTP header
Page and response details
What is a meta robots checker?
A meta robots checker examines a webpage for crawler directives that are declared in HTML and in the HTTP response. These directives can tell search engine crawlers whether a page should be indexed, whether links should be followed, and how certain content should be handled. Checking both locations is useful because an indexing directive may be present in the HTML source, the HTTP headers, or both.
Check noindex in HTML
The HTML <meta name="robots"> tag is one common way to communicate crawler instructions. A page can use directives such as noindex, nofollow or noarchive. The checker also looks for crawler-specific names such as googlebot and bingbot when they are present. This helps website owners identify page-level directives that may not be obvious from the visible content.
Check X-Robots-Tag headers
Search crawler directives can also be returned through the X-Robots-Tag HTTP response header. This is especially useful for resources that are not ordinary HTML pages, such as PDFs and other files, because an HTTP header can communicate indexing instructions without adding an HTML meta tag. The checker reports the header value exactly as returned by the response. For a broader view of server response headers, see the HTTP Header CheckerCanonical URL CheckerURL Indexability CheckerSERP Preview Checker.
Why check both locations?
Checking only the page source can miss an indexing directive supplied by the server. Checking only the HTTP headers can miss a meta robots tag in the HTML. Reviewing both gives a more complete picture of the crawler directives returned by the URL. Remember that robots.txt controls crawling access and is a separate mechanism from page-level noindex directives. If multiple directives exist, the results show each detected directive so you can compare the page-level and response-level configuration.
How to use the Meta Robots Checker
Enter a public HTTP or HTTPS URL and select Check Robots. RedirectionChecker follows supported redirects within its safety limits and analyzes the final response. The results show the HTTP status, final URL, detected HTML crawler directives, the X-Robots-Tag value, response timing and resolved IP information. Headers or meta directives that are not present are clearly labeled as Not present.
Meta robots and technical SEO
Unexpected noindex directives can be an important technical SEO issue when a page is intended to appear in search results. A checker can help confirm whether the instruction is coming from HTML markup or an HTTP response header. It is still important to review the site's intended indexing policy and the complete page configuration before changing a directive. You may also want to check the URL's HTTP status and redirect path before deciding whether an indexing instruction is intentional.
Check meta robots directives online
Use the checker above to inspect crawler directives for a public URL. The tool is designed to make the difference between HTML meta robots instructions and server-level X-Robots-Tag instructions easy to see in one report.
Understand the difference between crawling and indexing
Meta robots directives and the X-Robots-Tag response header can tell search engines whether content should be indexed or links followed. A robots.txt block is different: it can prevent a crawler from fetching the page and seeing a noindex directive. Check both mechanisms and avoid treating a single tool result as a guarantee about what will appear in search.