This tool allows you to generate your robots.txt.
Be careful, you use the robots.txt generator at your own risk. It is advisable to validate that your robots file does not exclude pages by mistake. There are many tools to test this like the Google tool.
The robots.txt file is a text file which contains commands for search engine indexing robots (or crawlers) to specify which pages can or can not be indexed. It is a ASCII file located at the root of the website, and must be written in lowercase and plural. The Meta tag "robots" (in the header of pages) can also be used to forbid the indexed of a page.
This file allows you to leave instructions to the indexing robots:
It can contain the following directives:
Be careful: The robots.txt file is not treated in the same way by all search engine.
For certains robots the first directive takes the upper hand, for others robots the most specific directives that take the upper hand.
Exclusion of all pages for all search engines / crawlers:
Robots txt Generator makes it easy to create a robots file for your website.
You can easily and quickly choose which pages are allowed to be crawled by search engines.