Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Whenever a search engine crawls the website, they look for the robots.txt files on the domain roots. The files’ list of directives is read to see if they are blocked from crawling. If you want to create a custom robots.txt file instantly, FreeSEOToolBox has the robots.txt file generator.
This is basically a plain and simple text format file that prevents search engine crawlers from indexing content and crawling. In case you are uncertain if the website has a robots.txt file, you can check it by typing yourdomain.com/robots.txt in the browser. If the format page appears, you have the file. Also, if you are using Yoast, it can be used for creating the robots.txt file.
In case you are wondering, “how do I create a robots.txt file?” or “how do I get a robots.txt file from a website?” FreeSEOToolBox has got you covered. This tool is designed to help SEO experts and website owners to generate robots.txt files even if they don’t have technical skills. This generator tool is extremely easy to use, but it’s best to read through the Google instructions.
No, there is no need to buy any subscription. This is because FreeSEOToolBox has designed the robots.txt generator tool for everyone and is available for free. The best thing is that you don’t need to sign up for an account either. As far as the costs are concerned, this is a free tool which makes it available for everyone and doesn’t have hidden charges.
No, we do not save or store your robots.txt file on our server. This is because we believe in ensuring the security and protection of our customers’ data.


RELATED TOOLS