Comprobador de robots.txt

El comprobador de Robots.txt de SEOptimer verifica si tu sitio web tiene un archivo robots.txt configurado, ayudándote a gestionar el acceso de los rastreadores de motores de búsqueda a tu sitio.

SEO Audit Any Site in Seconds
+ Comprehensive SEO / GEO Tools for Every Function

Run over 100 checks on your website across On-Page SEO, Usability, Performance, Social and Links. Now with GEO in every report. SEOptimer also offers keyword research, rank tracking, backlinks and a full site crawler.

Comienza tu prueba gratuita de 14 días

About Robots.txt

What is a robots.txt file?

Robots.txt is a plain text file at the root of a domain that tells automated visitors which parts of the site they may request. It is the first thing most crawlers ask for, before any page, and the oldest convention on the web for saying what a machine should leave alone.

It controls crawling, not indexing, and the distinction catches people out constantly. Blocking a URL stops a crawler fetching it; it does not reliably keep that URL out of search results, because a page can be indexed on the strength of links pointing at it without ever being read.

A robots.txt checker is worth running because a single misplaced character changes the meaning of the whole file, and nothing about the site indicates a problem until traffic disappears.

Does robots.txt still matter in 2026?

Its traditional job is unchanged: keeping crawlers out of admin paths, internal search results, faceted navigation and the endless parameter combinations that would otherwise consume crawl budget on pages nobody should land on.

Its newer job is larger. The file has become the place where site owners decide which AI crawlers may access their content at all, a question most had never had to answer before. Every major AI company now operates a named crawler, and most respect the file.

That makes it one of the few genuinely consequential files on a site, and one worth reviewing deliberately rather than inheriting whatever a platform shipped by default.

How robots.txt relates to AI search

You have a real decision to make, and there is no universally correct answer. Blocking AI crawlers protects your content from being used as training or answer material. It also removes any possibility of being cited in the answers a growing share of people now rely on.

The two effects are not symmetrical. Blocking is close to absolute, while being cited is uncertain, so the choice depends on whether your content is the product or the marketing for it.

A publisher selling subscriptions may reasonably block; a business whose pages exist to be found usually will not. What matters is that the file says what you actually intend.

Robots.txt best practices

Keep the file at the domain root. Anywhere else and it is ignored entirely.
Never use it to hide sensitive content. The file is public, and reading it is the fastest way to find the paths you wanted hidden.
Use a noindex directive rather than a crawl block when the goal is keeping a page out of results.
Do not combine the two on the same page. A blocked page can never be read, so its noindex is never seen.
Reference your XML sitemap in the file so crawlers find it without being told separately.
Give subdomains their own file; a rule on the main domain does not cover them.

Common mistakes

Leaving a development-era rule blocking the whole site after launch, which is the most damaging version of this error.
Blocking CSS or JavaScript, which stops search engines rendering the page as a visitor sees it.
Assuming a blocked URL drops out of search results, when it may remain listed without a description.
Writing rules for a user agent string that does not exist, so nothing is affected.
Blocking crawl paths that your own sitemap then submits, sending contradictory instructions.

What this tool checks

The robots.txt checker requests the file from the domain root and reports whether one is present.

It confirms existence rather than evaluating the rules, so a file that accidentally blocks your entire site passes exactly as a correct one does. Reading what it actually says is the part that matters, and it takes about a minute.

Where to go next

Robots.txt is one of three instructions that decide whether a page is found. Build or revise your file with the robots.txt generator, check the noindex tags checker that keep a fetched page out of results, and make sure your XML sitemap generator lists the pages you do want crawled.

Comprehensive SEO Toolbox with over 55 Tools

Check Titles, Meta Descriptions, Headings, Schema, Core Web Vitals, SSL and Robots.txt. Generate Meta Tags, Sitemaps and .htaccess Rules, minify CSS, JS and HTML, and draft copy with our AI Writing Tools - instantly and for free.