/    Sign up×
Community /Pin to ProfileBookmark

Blocking search bots from site stats?

Copy linkTweet thisAlerts:
Oct 31.2005

I have a php counter/statistics script running on my site and it’s storing the info in an xml file, and I was wondering if there was any way to block the search bots from being recorded as I get many each day visiting my site. They use many different ip’s so I can’t simply block a specific ip, which would be the easiest way, any other ideas?

Thanks in advance ?

to post a comment
PHP

11 Comments(s) ↴

Copy linkTweet thisAlerts:
@bokehOct 31.2005 — Most of these can be recognised by user_agent. All you need is a rough list of the bots by user agent and then just do eregi on the user agent. For example if the user agent contains "slurp" or "yahoo" or "googlebot" don't increment your records.
Copy linkTweet thisAlerts:
@SnowCrashNov 01.2005 — Additionally you could also set a [URL=http://www.searchengineworld.com/robots/robots_tutorial.htm]robots.txt file[/URL]. Although it's not always respected...
Copy linkTweet thisAlerts:
@bokehNov 01.2005 — Additionally you could also set a [URL=http://www.searchengineworld.com/robots/robots_tutorial.htm]robots.txt file[/URL]. Although it's not always respected...[/QUOTE]I can't see how that would stop bots from being included in the sites logging system.
Copy linkTweet thisAlerts:
@SnowCrashNov 01.2005 — Lets say your script logs requests to the page "main.php". If you put "Disallow: main.php" into your robots.txt file, then robots shouldn't access that page and thus create no request on that page.
Copy linkTweet thisAlerts:
@bokehNov 01.2005 — If you ban bots from reading you pages you will not be listed at any search engines and I'm sure that is not the desired effect.
Copy linkTweet thisAlerts:
@SnowCrashNov 01.2005 — Your'e right...
Copy linkTweet thisAlerts:
@Brad_ArmitageauthorNov 01.2005 — Thanks for the replies guys, and yeah that option is not what I had in mind! :rolleyes:

bokeh:

about your original post, could you give me an example, I can provide an example of my code if you need it, thanks ?
Copy linkTweet thisAlerts:
@bokehNov 01.2005 — Here is a start:
[code=php]if(!eregi('Slurp|Googlebot|msnbot|search', $_SERVER['HTTP_USER_AGENT'])){
//make a log entry
}[/code]
Copy linkTweet thisAlerts:
@Brad_ArmitageauthorNov 08.2005 — Sorry I couldn't reply till now, got caught up in another problem!!

bokeh -

this is a snipit of my code that shows an example of what I have in my current "banned" array blocking certain IP's, my question for you is, how could I fit that if statement you provided before into my current if statement or do I have to make a brand new one? (I still need to ban a few IP's that aren't bots)

[code=php]
$banned = array('24.25.26.27', '142.141.140.139');
if (!in_array($user_ip, $banned)) {

if($OpenFile){

fwrite($OpenFile, $Count);

fclose($OpenFile);

}
}
[/code]

?
Copy linkTweet thisAlerts:
@bokehNov 08.2005 — You just add it as a second condition of the first IF statement.[code=php]$banned = array('24.25.26.27', '142.141.140.139');
if ((!in_array($user_ip, $banned)) and (!eregi('Slurp|Googlebot|msnbot|search', $_SERVER['HTTP_USER_AGENT']))){

if($OpenFile){

fwrite($OpenFile, $Count);

fclose($OpenFile);

}
}[/code]
Copy linkTweet thisAlerts:
@Brad_ArmitageauthorNov 08.2005 — I thought that was how, but I just got the syntax a little wrong for adding a second condition! Thanks and it appears to be working, I'll let you know later on if any bots re-appear, but I imagine that should work!! ?
×

Success!

Help @Brad_Armitage spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.28,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...