Free Robots.txt Generator

The Free robots.txt file generator allows you to easily product a robots.txt file for your website based on inputs.

robots.txt is a file that can be placed in the root folder of your website to help search engines index your site more appropriately. Search engines such as Google use website crawlers, or robots that review all the content on your website. There may be parts of your website that you do not want them to crawl to include in user search results, such as admin page. You can add these pages to the file to be explicitly ignored. Robots.txt files use something called the Robots Exclusion Protocol. This website will easily generate the file for you with inputs of pages to be excluded.

Default - All Robots are:

Crawl-Delay:

Sitemap:

Search engines:

Blocking these removes you from the results they serve.

Google

Google Images

Bing

Yandex

Baidu

Naver

Apple

DuckDuckGo

Yahoo

AI crawlers:

Think before blocking these. OAI-SearchBot is what indexes you for ChatGPT search, so disallowing it removes you from those answers. Google-Extended covers Gemini training only and does not affect Google Search or AI Overviews.

GPTBot

OAI-SearchBot

ChatGPT-User

Google-Extended

ClaudeBot

anthropic-ai

PerplexityBot

Perplexity-User

CCBot

Bytespider

Amazonbot

Applebot-Extended

Meta-ExternalAgent

cohere-ai

SEO and commercial crawlers:

These gather link and keyword data. Blocking them saves bandwidth and hides your site from competitors using those tools - it has no effect on search rankings.

Ahrefs

Semrush

Majestic

Moz

BLEXBot

DataForSEO

Internet Archive

Restricted Directories:

The path is relative to root and must contain a trailing slash "/"

Output

Your Generated Robots.txt File:

SEO Audit Any Site in Seconds
+ Comprehensive SEO / GEO Tools for Every Function

Run over 100 checks on your website across On-Page SEO, Usability, Performance, Social and Links. Now with GEO in every report. SEOptimer also offers keyword research, rank tracking, backlinks and a full site crawler.

Start 14 Day Free Trial

About Robots.txt Files

What is a robots.txt file?

Robots.txt is a plain text file living at the root of a domain that tells automated visitors which parts of the site they may request. It is the first thing most crawlers ask for, before any page, and it is the oldest convention on the web for saying what a machine should leave alone.

The syntax is deliberately simple: name a user agent, then list the paths it may or may not fetch. That simplicity is also the trap, because a single misplaced slash can move from blocking one folder to blocking an entire site. A robots.txt generator matters chiefly because it produces the file in a valid, predictable shape rather than leaving you to remember the exact spelling of directives you write once a year.

It controls crawling, not indexing, and that distinction catches people out constantly. Blocking a URL stops a crawler fetching it; it does not reliably keep the URL out of search results, because a page can be indexed on the strength of links pointing at it without ever being read.

Does robots.txt still matter in 2026?

Its traditional job is unchanged: keeping crawlers out of admin paths, internal search results, faceted navigation and the endless parameter combinations that would otherwise consume crawl budget on pages nobody should land on.

Its newer job is larger. The file has become the place where site owners decide which AI crawlers may access their content at all, a question most had never had to answer before. Every major AI company now operates a named crawler, and most respect the file.

That makes robots.txt one of the few genuinely consequential files on a site, and one worth reviewing deliberately rather than inheriting whatever a platform shipped by default.

How robots.txt relates to AI search

You have a real decision to make, and there is no universally correct answer. Blocking AI crawlers protects your content from being used as training or answer material. It also removes any possibility of your site being cited in the answers a growing share of people now rely on instead of clicking through to results.

The two effects are not symmetrical. Blocking is close to absolute, while being cited is uncertain, so the choice depends on whether your content is the product or the marketing for it. A publisher selling subscriptions may reasonably block; a business whose pages exist to be found will usually not.

What matters most is that the file says what you actually intend. A robots.txt generator makes it straightforward to state those rules explicitly rather than leaving them implied by omission.

Robots.txt best practices

Keep the file at the domain root. Anywhere else and it is ignored entirely.
Never use it to hide sensitive content. The file is public, and reading it is the fastest way to find the paths you wanted hidden.
Use a noindex directive rather than a crawl block when the goal is keeping a page out of results.
Do not combine the two on the same page. A blocked page can never be read, so its noindex is never seen.
Reference your XML sitemap in the file so crawlers find it without being told separately.
Review the AI crawler rules as a deliberate decision, not a default.

Common mistakes

Blocking CSS or JavaScript, which prevents search engines rendering the page as a visitor sees it.
Leaving a development-era rule blocking the whole site after launch. This is the single most damaging robots.txt error.
Assuming a blocked URL will drop out of search results, when it may remain listed without a description.
Writing rules for a user agent string that does not exist, so nothing is affected.
Forgetting that subdomains need their own file; a rule on the main domain does not cover them.

Using the robots.txt generator

Choose which crawlers to allow or refuse, add any directories to keep out of, set a crawl delay if your server needs one, and point to your sitemap. The generator assembles a correctly formatted file you can save to your domain root.

Once it is live, verify it rather than assuming. Fetch the file in a browser to confirm it is being served, and check in Search Console that the pages you expect to be crawlable still are. A robots.txt file that silently blocks more than intended can take weeks to notice through traffic alone.

Where to go next

Once the file is live, confirm it is being served and that it is not blocking more than you intended. Check it with the robots.txt checker, review the noindex tags checker that keep a fetched page out of results, and make sure the XML sitemap checker you referenced actually exists.

Comprehensive SEO Toolbox with over 55 Tools

Check Titles, Meta Descriptions, Headings, Schema, Core Web Vitals, SSL and Robots.txt. Generate Meta Tags, Sitemaps and .htaccess Rules, minify CSS, JS and HTML, and draft copy with our AI Writing Tools - instantly and for free.