Robots.txt is a file in text form that instructs bot crawlers to index or not index certain pages. It is also known as the gatekeeper for your entire site. Bot crawlers’ first objective is to find and read the robots.txt file, before accessing your sitemap or any pages or folders.
Create a robots file for your site using a utility, Wordpress plugin, or your CMS's robots.txt creation process. Follow our guides below.