/    Sign up×
Community /Pin to ProfileBookmark

Blacklisted search engines for robots.txt?

Copy linkTweet thisAlerts:
Apr 13.2006

Are there any “evil” search engines or general IPs that I’d want to block from accessing my site?

Not really search engines, but any particular known IPs that may harm my site

to post a comment
SEO

3 Comments(s) ↴

Copy linkTweet thisAlerts:
@KeveyApr 13.2006 — New one's can come up every day...you can google for bad bots.

Here's an example of a bad bot list. I don't know if it's current.

http://www.kloth.net/internet/badbots.php
Copy linkTweet thisAlerts:
@omgwtfauthorApr 13.2006 — Thanks!
Copy linkTweet thisAlerts:
@felgallApr 14.2006 — All the bad ones ignore the robots.txt file so listing them there wont do you any good. You can only use robots.txt to control what parts of your site that the good bots visit.

The way to ban the bad bots is via entries in the .htaccess file.
×

Success!

Help @omgwtf spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 6.1,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...