@ahk2chanAug 16.2006 — #It basically tells which search engine what to crawl and what not to crawl. There is a site that gives information about it. [url=http://www.robotstxt.org/wc/robots.html]http://www.robotstxt.org/wc/robots.html[/url]
Since search engines are looking at this file... so it is important to set it up right! For example, if you have a few backup copies of an HTML page, and if the search engine crawls those pages according to your robots.txt, then you may get duplicate content indexed... (which is not a good thing because the search engine tends to remove duplicate content, and you don't know if the live one or the backup ones get removed...)
I am afraid I have not worried about the robot.txt file - I just put "robots follow" in the head section of the pages and everything worked - i.e. google has registered more then a thousand pages of my site. Personally I think one just follows the rules and let the search engines get on with what they do best - providing/recommending pages to the viewer that seems most likely to provide the results for the search string that was submitted. What makes one think that their pages are what the viewer is looking for??
The search engines are designed to do a specific job and trying to cheat and make your page come on page one when it is not the best page for the string will result in the page being penalized and dropped. I think one can just provide the information that one has (i.e. page content) and hope that it is what people are looking for and that search engines will recognize this and recommend/suggest the page on page one of their search engine results. When I am inserting keywords, I just think about the contents of the page and type in what I think one would use for strings if they wanted this particular page - it seems to work.