At 6:41 PM -0800 2/4/98, Cyndi Norman wrote:
> Are there global search engines I have to worry about that do FTP sites?
They don't search FTP sites per-se, but if you have urls pointing to
stuff on an FTP site, then when the search engine wanders across your
links, the data on the FTP site will be added by many of them. This
isn't a protection from the search engines, unless you put a robots.txt
on your ftp site (and do so via an HTTP daemon).
> I don't really understand this though. If I have a custom domain that's
> really a virtual domain, why would robots.txt work?
It should work if (and only if) <http:foo.bar.baz/robots.txt> is
readable. If your virtual hosting doesn't allow for that, it won't. And
it's also possible that places like Alta vista aren't honoring the
Host: fields, and that could be causing problems....
--
Chuq Von Rospach (Hockey fan? <http://www.plaidworks.com/hockey/>)
Apple Mail List Gnome (mailto:chuq@apple.com)
Plaidworks Consulting (mailto:chuqui@plaidworks.com)
<http://www.plaidworks.com/> + <http://www.lists.apple.com/>
References:
|
|
