Subject: Re: Security Issues...
From: Michael Tucker (mtucker@eecs.harvard.edu)
Date: Tue Aug 14 2001 - 09:21:33 MDT
Hey Brian,
Sites like AltaVista and Google rely on automated "spiders" that crawl
through through the Internet indexing the URLs they visit. Although the
hit on your site may have been one of any number of web-saerch sites, it
is probably harmless. The robots.txt file allows you to direct the
search-engine's spider in its task of indexing your content. For
well-behaved "robots" (some just ignore them), this will let you keep your
site off of search engines, or better yet only get listed the pages that
you want people coming in through (ie front door, instead of coming in the
side ;) For more info, take a look on Yahoo or Google for more information
about robots.txt, there is PLENTY of it out there!
Hope that helps.
Mike
On Tue, 14 Aug 2001, Brian Watson wrote:
> What purpose does the robots.txt file serve? The bot that requested
> it was trying to index my website?
This archive was generated by hypermail 2a24 : Tue Aug 14 2001 - 08:30:30 MDT