On Fri, 6 Jun 1997, Eric Schult wrote:
> > There is no file or image that is related to this file name on our web
> > page. Could not even find it on the server anywhere. Am I unduly
> > concerned?
>
> No, don't be concerned. Many (well behaved) web indexers (and other "robots")
> will look for this file, which may contain instructions to NOT index
> a particular site.
>
> I don't know what you'd put in it, I only know that its use is not
> a threat.
It may not be a direct threat but it can be an indirect one.
There was a very good article last year about how not using robots.txt in
some cases can bring your system to a crawling halt. I believe the
article was in Web Techniques and it told how one web site was completely
swamped by requests from the crawlers that it forces the CPU usage up to
near 100% and thus made the site inaccessible. It was not an attack or
anything but it had the same effect as a Denial of Service attack.
---
"It's a damn poor mind that can only think of one way to spell a word."
-- Andrew Jackson
Follow-Ups:
References:
|
|
