Search Engines Blocked Checker
SEOptimer’s Search Engines Blocked Checker reads your robots.txt and tells you whether Google and other search engines are being blocked from crawling your site.
SEO Audit Any Site in Seconds
+ Comprehensive SEO / GEO Tools for Every Function
Run over 100 checks on your website across On-Page SEO, Usability, Performance, Social and Links. Now with GEO in every report. SEOptimer also offers keyword research, rank tracking, backlinks and a full site crawler.
About Blocking Search Engines in Robots.txt
What does being blocked actually mean?
A robots.txt rule tells a crawler not to fetch a path. Applied to the whole site, it stops search engines reading anything, and everything downstream of that stops working: no fresh content is discovered, and existing pages go stale and eventually drop out.
This checker reads your robots.txt and reports whether search engines are being blocked from your site.
It is the single most damaging line a site can carry, and it is one line long.
How it happens
Almost always by accident, and almost always the same way: a staging site is built with a site-wide disallow to keep it out of results, and the file goes live with the rest of the deployment.
Content management systems have their own version of this. A "discourage search engines" setting exists in most of them, ticked during development and rarely thought about again.
Nothing breaks visibly when it happens. The site works perfectly for every human who visits, which is why it often runs for weeks before anyone connects the traffic decline to the cause.
Blocking is not the same as removing
This is the part that surprises people. A blocked page can still appear in results, listed without a description, because the crawler was told not to read it but not told to leave it out.
To keep a page out of results properly you need a noindex directive, which means the crawler has to be allowed to fetch the page to see it. Blocking and deindexing pull in opposite directions.
So robots.txt is the wrong tool for privacy. It controls crawling, not visibility, and the file itself is public.
Does robots.txt still matter in 2026?
It is upstream of everything. Content, structure, speed and links all assume a crawler can reach the page in the first place, and none of them help if it cannot.
It has taken on a second job as well. The same file now governs whether AI crawlers can read your site, which decides whether your pages can be cited in generated answers.
For a file that is usually a handful of lines, it carries an unreasonable amount of weight.
What this tool checks
It fetches robots.txt from the domain you enter and reports whether the rules would prevent search engines from crawling your site.
A pass means nothing is blocking them at the site level. It does not confirm that individual paths are open, so a page that will not index despite this passing is worth checking on its own.
Where to go next
If crawling is open, the next questions are what is discoverable and what is deliberately excluded. Review the whole robots.txt file, check your XML sitemap is present and current, and confirm no noindex tag is doing quietly what robots.txt is not.
Comprehensive SEO Toolbox with over 55 Tools
Check Titles, Meta Descriptions, Headings, Schema, Core Web Vitals, SSL and Robots.txt. Generate Meta Tags, Sitemaps and .htaccess Rules, minify CSS, JS and HTML, and draft copy with our AI Writing Tools - instantly and for free.
Further Reading
もっと見る
Robots.txt - 究極のガイド
Robots.txtとは何ですか？ Robots.txtは、特定のページをインデックスに登録するかしないかをボットクローラーに指示するテキスト形式のファイルです。これは、あなたのサイト全体のゲートキーパーとしても知られています。ボットクローラーの最初の目的は、サイトマッ...
Google Search Consoleで「Indexed, though blocked by robots.txt」を修正する方法
Google Search Consoleで「Indexed, though blocked by robots.txt」という警告通知を受け取った場合、検索エンジンの結果ページ（SERPS）でページがランク付けされる能力に影響を与えている可能性があるため、できるだけ早く修正する...
Google Crawlerとは何ですか？
Googleを使ってサービスを検索したり、情報を見つけたりする時のことを知っていますか？ページが読み込まれると、常に一番上にあるウェブサイトがあります。 Google検索エンジンの結果ページ（SERP）で1位に位置するサイトは、すべてのクリックの大部分を獲得します。...
SEOにおけるクロールの深さ: それは何ですか & どうやって改善するのですか？
クロールの深さは、Googleがコンテンツをどれだけ効率的にインデックスできるかに影響します。 Googlebotは限られた時間とサーバーリソースを持っています。したがって、クロール予算、つまり特定の時間枠内でGooglebotがあなたのサイトでクロールできるページ...