There was an article on *i think* drupal.org about how to test for bad robots (i.e. email harvesters etc) by seeing if they obey a fake dir in robots.txt, then giving the robots a 412 precondition failed if they don’t.
Anyone help me find this article? I’ve searched for ages!!
@MindzaiMay 09.2009 — #I dont know about the article, but you could do this by setting an exclude in robots.txt for a randomly named and un-linked php script. Any accesses of that script must mean the robot is purposefully following excludes and it can take the steps you desire.