Test Website Crawlability
SEOptimer’s free crawlability test checks whether search engines and AI crawlers can reach, fetch and index your page - robots.txt rules, noindex directives, status code and how much content actually renders.
SEO Audit Any Site in Seconds
+ Comprehensive SEO / GEO Tools for Every Function
Run over 100 checks on your website across On-Page SEO, Usability, Performance, Social and Links. Now with GEO in every report. SEOptimer also offers keyword research, rank tracking, backlinks and a full site crawler.
About Website Crawlability
What does crawlability mean?
A page is crawlable when a search engine can reach it, is permitted to fetch it, and finds something worth keeping once it arrives. Fail any one of those and the page is effectively invisible, no matter how good the content is.
It is the step before everything else. Ranking, snippets, AI citations - all of it assumes the page made it into an index in the first place, so the sensible first move when a page simply is not appearing is to test website crawlability rather than start rewriting the content.
The awkward part is that every one of these failures is silent. Nothing breaks, nothing errors, the page looks perfect to you - it just never shows up.
The four things that stop a page being indexed
Almost every crawlability problem is one of these, and they fail in a specific order:
- The URL does not resolve - a 404, a server error, or a redirect chain that gives up.
- Robots.txt disallows the crawler, so the page is never fetched at all.
- A noindex directive tells the engine to fetch the page but keep it out of results.
- The page renders almost nothing without JavaScript, so there is little to index.
The order matters when you fix them. A robots.txt block means the noindex tag beneath it is never even read, so pairing the two does the opposite of what people expect - the page stays crawl-blocked and its removal instruction goes unseen.
Robots.txt and noindex are not the same thing
This is the confusion that causes the most damage. Robots.txt controls whether a crawler may fetch a page; noindex controls whether it may keep it. They operate at different stages and are not interchangeable.
To keep a page out of search results you want noindex, and the crawler has to be allowed to fetch the page to see it. Blocking in robots.txt instead can leave a URL listed with no description at all, because the engine knows the address but has never been permitted to look at it.
For the reverse - a page you do want found - both need to be clear, and the noindex can arrive in either the HTML or an HTTP header. The header version is the one that catches people out, since it is invisible in the page source.
AI crawlers are a separate decision in 2026
The AI companies run their own named crawlers, and blocking them is a different choice from blocking search engines. GPTBot, ClaudeBot, PerplexityBot and the rest each take their own line in robots.txt.
Plenty of sites have blocked them without quite meaning to, often by copying a robots.txt from somewhere else. If being cited in AI answers matters to you, that is worth knowing about rather than discovering later.
The opposite is also a legitimate position. If you would rather your content were not used for training or summarising, blocking those agents is how you say so - the point is that it should be a decision rather than an accident.
How to test website crawlability with this tool
Enter a URL and the tool runs eleven checks against it, reporting each one separately, in the order a crawler meets them:
- Page reachable - the status code the URL returns.
- Search engines blocked - whether robots.txt disallows Googlebot.
- AI crawlers blocked - how many of the major AI user agents are disallowed.
- Noindex tag - a noindex directive in the page HTML.
- Noindex header - the same directive delivered as an X-Robots-Tag.
- Robots.txt - whether the file exists, and what it contains.
- XML sitemap - whether one exists for crawlers to discover the rest of the site.
- Llms.txt - whether the file AI assistants look for is present.
- Canonical tag - which URL the page nominates as the one to index.
- Rendered content - how much of the page a crawler sees without running JavaScript.
- JavaScript errors - scripts failing on the page, which is often why the rendered content is thin.
It checks the single URL you give it rather than crawling the whole site, so test website crawlability on the page that is actually missing rather than on the homepage - the two often differ, and the homepage is rarely the one with the problem.
Where to go next
If everything here passes and the page still is not appearing, discoverability is the next thing to look at. Check the page is listed in your XML sitemap, review the whole robots.txt file rather than the rules for one agent, and see whether anything on the site links to it with the internal link checker.
Comprehensive SEO Toolbox with over 55 Tools
Check Titles, Meta Descriptions, Headings, Schema, Core Web Vitals, SSL and Robots.txt. Generate Meta Tags, Sitemaps and .htaccess Rules, minify CSS, JS and HTML, and draft copy with our AI Writing Tools - instantly and for free.
Further Reading
もっと見る
Google Crawlerとは何ですか？
Googleを使ってサービスを検索したり、情報を見つけたりする時のことを知っていますか？ページが読み込まれると、常に一番上にあるウェブサイトがあります。 Google検索エンジンの結果ページ（SERP）で1位に位置するサイトは、すべてのクリックの大部分を獲得します。...
AI Crawlability: Should You Let AI Bots Access Your Site and How Can You Check?
As more people use ChatGPT, Claude, Perplexity, and Google’s AI search features, AI crawlability is becoming increasingly important for webs...
Robots.txt - 究極のガイド
Robots.txtとは何ですか？ Robots.txtは、特定のページをインデックスに登録するかしないかをボットクローラーに指示するテキスト形式のファイルです。これは、あなたのサイト全体のゲートキーパーとしても知られています。ボットクローラーの最初の目的は、サイトマッ...
Google Search Consoleで「Indexed, though blocked by robots.txt」を修正する方法
Google Search Consoleで「Indexed, though blocked by robots.txt」という警告通知を受け取った場合、検索エンジンの結果ページ（SERPS）でページがランク付けされる能力に影響を与えている可能性があるため、できるだけ早く修正する...
What is Rendered Content and How it Affects AI Crawlers?
Key Takeaway
Rendered content refers to the version of a webpage that a browser displays after executing HTML, CSS, and JavaScript....
SEOにおけるクロールの深さ: それは何ですか & どうやって改善するのですか？
クロールの深さは、Googleがコンテンツをどれだけ効率的にインデックスできるかに影響します。 Googlebotは限られた時間とサーバーリソースを持っています。したがって、クロール予算、つまり特定の時間枠内でGooglebotがあなたのサイトでクロールできるページ...