What is the easiest way to have google and other search engines spider all parts of my website but only show specific pages in search results? I’ve read of a few ways, but also read that they aren’t effective.
@SpiderlineJul 19.2005 — #Or the same cna be on a robots.txt file to limit robots you do not want to index your site.
Useing the Meta tag above will lower the chance of you comming at the top of the search results for anything listed in the page. "follow" will keep it running to all other pages.
A good trick is to use a sitemap.html with noindex and follow and have it attached to the splashpage with a hidden or small link. have this page linking everything on your site. It also make for easy indexing for other crawlers.
@TechEvangelistJul 22.2005 — #Hidden links are a violation of search engine guidelines and may get your site penalized. The site map is a good idea, but a hidden link may not be.