@Steve_SmithJun 24.2013 — #The robots.txt file is just a simple text file as the file extension suggests. It’s created using a simple text editor like Notepad or WordPad, complicated word processors such as Microsoft Word will only corrupt the file.
@Tim_ReedJun 24.2013 — #Robots.txt is a text file which instructs the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.
@joewarn8Jun 24.2013 — #A robots.txt file is used to restricts access to your site by search engine robots that crawl the web.
The simplest robots.txt file uses two rules:
User-agent: the robot the following rule applies to Disallow: the URL you want to block Basically it is used to tell the crawler which one page or file not to be crawled .
@aileennicole13Jun 24.2013 — #Robots.txt is a text file whichis used to guide the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.