Test Website Crawlability

SEOptimer’s free crawlability test checks whether search engines and AI crawlers can reach, fetch and index your page - robots.txt rules, noindex directives, status code and how much content actually renders.

SEO Audit Any Site in Seconds
+ Comprehensive SEO / GEO Tools for Every Function

Run over 100 checks on your website across On-Page SEO, Usability, Performance, Social and Links. Now with GEO in every report. SEOptimer also offers keyword research, rank tracking, backlinks and a full site crawler.

Start 14 Day Free Trial

About Website Crawlability

What does crawlability mean?

A page is crawlable when a search engine can reach it, is permitted to fetch it, and finds something worth keeping once it arrives. Fail any one of those and the page is effectively invisible, no matter how good the content is.

It is the step before everything else. Ranking, snippets, AI citations - all of it assumes the page made it into an index in the first place, so the sensible first move when a page simply is not appearing is to test website crawlability rather than start rewriting the content.

The awkward part is that every one of these failures is silent. Nothing breaks, nothing errors, the page looks perfect to you - it just never shows up.

The four things that stop a page being indexed

Almost every crawlability problem is one of these, and they fail in a specific order:

The URL does not resolve - a 404, a server error, or a redirect chain that gives up.
Robots.txt disallows the crawler, so the page is never fetched at all.
A noindex directive tells the engine to fetch the page but keep it out of results.
The page renders almost nothing without JavaScript, so there is little to index.

The order matters when you fix them. A robots.txt block means the noindex tag beneath it is never even read, so pairing the two does the opposite of what people expect - the page stays crawl-blocked and its removal instruction goes unseen.

Robots.txt and noindex are not the same thing

This is the confusion that causes the most damage. Robots.txt controls whether a crawler may fetch a page; noindex controls whether it may keep it. They operate at different stages and are not interchangeable.

To keep a page out of search results you want noindex, and the crawler has to be allowed to fetch the page to see it. Blocking in robots.txt instead can leave a URL listed with no description at all, because the engine knows the address but has never been permitted to look at it.

For the reverse - a page you do want found - both need to be clear, and the noindex can arrive in either the HTML or an HTTP header. The header version is the one that catches people out, since it is invisible in the page source.

AI crawlers are a separate decision in 2026

The AI companies run their own named crawlers, and blocking them is a different choice from blocking search engines. GPTBot, ClaudeBot, PerplexityBot and the rest each take their own line in robots.txt.

Plenty of sites have blocked them without quite meaning to, often by copying a robots.txt from somewhere else. If being cited in AI answers matters to you, that is worth knowing about rather than discovering later.

The opposite is also a legitimate position. If you would rather your content were not used for training or summarising, blocking those agents is how you say so - the point is that it should be a decision rather than an accident.

How to test website crawlability with this tool

Enter a URL and the tool runs eleven checks against it, reporting each one separately, in the order a crawler meets them:

Page reachable - the status code the URL returns.
Search engines blocked - whether robots.txt disallows Googlebot.
AI crawlers blocked - how many of the major AI user agents are disallowed.
Noindex tag - a noindex directive in the page HTML.
Noindex header - the same directive delivered as an X-Robots-Tag.
Robots.txt - whether the file exists, and what it contains.
XML sitemap - whether one exists for crawlers to discover the rest of the site.
Llms.txt - whether the file AI assistants look for is present.
Canonical tag - which URL the page nominates as the one to index.
Rendered content - how much of the page a crawler sees without running JavaScript.
JavaScript errors - scripts failing on the page, which is often why the rendered content is thin.

It checks the single URL you give it rather than crawling the whole site, so test website crawlability on the page that is actually missing rather than on the homepage - the two often differ, and the homepage is rarely the one with the problem.

Where to go next

If everything here passes and the page still is not appearing, discoverability is the next thing to look at. Check the page is listed in your XML sitemap, review the whole robots.txt file rather than the rules for one agent, and see whether anything on the site links to it with the internal link checker.

Comprehensive SEO Toolbox with over 55 Tools

Check Titles, Meta Descriptions, Headings, Schema, Core Web Vitals, SSL and Robots.txt. Generate Meta Tags, Sitemaps and .htaccess Rules, minify CSS, JS and HTML, and draft copy with our AI Writing Tools - instantly and for free.