If your site is modified regularly, then the spiders will monitor your site closely.
If your site was left dormant for years, chances are that spiders will pay visit it once a year.
The downloaded pages are not processed immediately. Basically the urls of the old pages are fetched in one cycle. New links present on downloaded pages are not downloaded.
This cycle can repeat itself, with requests for dead pages and no requests for new pages. Ultimately the new links on the pages are parsed and included in a new round.
Meta tags and keywords are not important anymore. Concentrate on relevant content, not on keywords.
@LeeUOct 27.2005 — #Very simply it tells the search engine robot how to act. See more about it [URL=http://www.webreference.com/authoring/robots/]here[/URL]
@soundsolutionOct 29.2005 — #I have been waiting a while for my site to appear in google search, i noticed a change a few days ago where site came up in search results for these terms. [B]"Vegetarian caterers london uk"[/B]. Only now seems to have dissapeared again, i accidentally mispelt when trying search again [B]"vegetarian caterers londonuk"[/B] then site appears in search results in same position, any idea why this is happening.