At 6:42 AM -0800 2/4/98, Tim Bowden wrote:
> There's a very satisfactory middle ground. Our archive Attic is
> closed to the bots but wide open to everyone else. Doesn't
> robots.txt accomplish the mission of blocking the search
> engines?
I don't know that this is middle ground -- that's what I'm doing. Yes,
robots cuts out the search engines and other crawlers. Everyone else
can get to my archives. And for those who need search engine
capability, I'm adding it, but privately, so that this information
doesn't end up in the global crawlers.
--
Chuq Von Rospach (Hockey fan? <http://www.plaidworks.com/hockey/>)
Apple Mail List Gnome (mailto:chuq@apple.com)
Plaidworks Consulting (mailto:chuqui@plaidworks.com)
<http://www.plaidworks.com/> + <http://www.lists.apple.com/>
References:
-
Re: Archives
From: Chuq Von Rospach <chuqui@plaidworks.com>
-
Re: Archives
From: Tim Bowden <tcbowden@clovis.nerdnosh.org>
|
|
