If I have an HTML file that has links to .pdf files, will search engines follow these links and index the .pdf files? And if so is there an equivalent META ( robots : nofollow, noindex ) tag to stop them doing this?
@LeeUNov 13.2006 — #Yes, that's exactly what I meant. It is meant to stop the bot on the page it is listed on or, if in the robots.txt file, at the index. However, be aware that not all bots follow the instructions in the robots.txt file or the metatags.