[Catalyst] Catalyst and robots.txt

Bernhard Graf catalyst at augensalat.de
Tue May 16 22:25:04 CEST 2006


On Tuesday 16 May 2006 18:32, Bill Moseley wrote:

> > Don't use robots.txt or you'll shoot into your own feet. Some
> > Spambots check for robots.txt to find the interesting pages.
>
> Do you have a reference that supports that claim?

My web server log files. ;-)
-- 
Bernhard Graf



More information about the Catalyst mailing list