/    Sign up×
Community /Pin to ProfileBookmark

supressing search engine spiders

Copy linkTweet thisAlerts:
Nov 21.2002

hi everybody,

is there a way to prevent being listed by search engines?

thanks ?

to post a comment
Full-stack Developer

8 Comments(s) ↴

Copy linkTweet thisAlerts:
@Zach_ElfersNov 21.2002 — Yes there is. In the <head> section of your page, type:

<meta name="robots" content="noindex, nofollow">

noindex means that the search engine robots will NOT index your page. nofollow means that they will not follow the links on your page.
Copy linkTweet thisAlerts:
@teekaNov 21.2002 — Hello everybody!! Just signed on.....

Also just published my first web site and, of course, am full of questions!


Concerning this:

<meta name="robots" content="noindex, nofollow">

why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?

Is it not a good thing that robots index a page or follow links?

Can they mess up my pages somehow?

Thanks

?
Copy linkTweet thisAlerts:
@cxemaNov 21.2002 — Here I give you a link to an old article (october 2001) that is most explanatory on the robots topic.

Hope you enjoy it

http://www.searchenginewatch.com/searchday/01/sd1024-robots.html
Copy linkTweet thisAlerts:
@Zach_ElfersNov 22.2002 — [i]Originally posted by teeka [/i]

[B]Hello everybody!! Just signed on.....



Also just published my first web site and, of course, am full of questions!





Concerning this:



<meta name="robots" content="noindex, nofollow">



why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?



Is it not a good thing that robots index a page or follow links?

Can they mess up my pages somehow?



Thanks

? [/B][/QUOTE]


Sometimes you just don't want certain pages listed.
Copy linkTweet thisAlerts:
@teekaNov 22.2002 — Thanks cxema....lots of reading for me now!!
Copy linkTweet thisAlerts:
@Dr_5150Nov 22.2002 — I don't like them indexing some of my pages.

ie.

Pages setup for private use. (My MP3 pages that change dynamically ? )

Not alowing them to index the individual pages of a site other than the home page.
Copy linkTweet thisAlerts:
@johnbullasNov 23.2002 — If you have site running in FRAMES you might not want the search engines to list pages INSIDE FRAMESETS without thier own navigation

Similarly if you set up HTML to go into frames as lists of links or headers & footers you might not want them listed

AND if a page is a list of links but of no intrinsic value you might want it NOINDEX but allow FOLLOW

As rightly said robots.txt can be used to explicitly exclude certain file types or directories

search robots.txt at google for more info:

http://www.robotstxt.org/wc/norobots.html

http://www.robotstxt.org/wc/meta-user.html

There is no guarantee that ALL search engines will abey these rules THE RULE IS ,... if you don't want ANYONE to find it DON'T put it on any webpages unless you believe you can keep it safe with non java .htaccess password control ! ?

John
Copy linkTweet thisAlerts:
@teekaNov 26.2002 — Thanks to all...I appreciate all the input. Being as this is my first web page and it is pretty simple, I don't think I have to worry about this right now, but good to know for the future!

As an aside...great forum! I've learned more from this site alone than I have in multiple other sites combined!

Thanks...
×

Success!

Help @laria spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.28,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...