[Catalyst] Catalyst and robots.txt

[Karl.Moens at marsh.com]

Brandon Black wrote:

> On 5/16/06, Bernhard Graf <catalyst at augensalat.de> wrote:
> > On Tuesday 16 May 2006 18:32, Bill Moseley wrote:
> >
> > > > Don't use robots.txt or you'll shoot into your own feet. Some
> > > > Spambots check for robots.txt to find the interesting pages.
> > >
> > > Do you have a reference that supports that claim?
> >
> > My web server log files. ;-)
> It might be interesting for someone to make a Catalyst module for
> bot-trapping, and stick that in a robots.txt Disallow: path.  Infinite
> recursion into randomly-generated seemingly-real content pages full of
> links to more of the same and fake nonsensical email addresses.

Or make it link to all these on-line pharmacies or mortgage-brokers.

Karl 



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This message and any attachments are confidential. If you have received 
this message in error please delete it from your system. If you require 
any assistance please notify the sender. Thank You.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~