Back to Free Tools

Free robots.txt File Generator

robots.txt is a file that can be placed in the root folder of your website to help search engines index your site more appropriately. Search engines such as Google use website crawlers, or robots that review all the content on your website. There may be parts of your website that you do not want them to crawl to include in user search results, such as admin page. You can add these pages to the file to be explicitly ignored. Robots.txt files use something called the Robots Exclusion Protocol. This website will easily generate the file for you with inputs of pages to be excluded.


Default - All Robots are:

Crawl-Delay:

Sitemap:

Search Robots:


Google

Google Image

Google Mobile

MSN Search

Yahoo

Yahoo MM

Yahoo Blogs

Ask/Teoma

GigaBlast

DMOZ Checker

Nutch

Alexa/Wayback

Baidu

Naver

MSN PicSearch

Restricted Directories:
The path is relative to root and must contain a trailing slash "/"











SEOptimer Report Preview

SEOptimer - SEO Audit & Reporting Tool.
Improve Your Website. Win More Customers.

Get a Free Website Audit Instantly