[Catalyst] Catalyst and robots.txt

[Brandon Black]

On 5/16/06, Bernhard Graf <catalyst at augensalat.de> wrote:
> On Tuesday 16 May 2006 18:32, Bill Moseley wrote:
>
> > > Don't use robots.txt or you'll shoot into your own feet. Some
> > > Spambots check for robots.txt to find the interesting pages.
> >
> > Do you have a reference that supports that claim?
>
> My web server log files. ;-)

It might be interesting for someone to make a Catalyst module for
bot-trapping, and stick that in a robots.txt Disallow: path.  Infinite
recursion into randomly-generated seemingly-real content pages full of
links to more of the same and fake nonsensical email addresses.

-- Brandon