Subject: Re: Apache on YDL question
From: Keary Suska (hierophant@pcisys.net)
Date: Sun Feb 03 2002 - 16:36:22 MST
on 2/3/02 3:57 PM, tjmahan@earthlink.net purportedly said:
> How do I use robots.txt keeps the spiders out of a site? I have
> development files up that I don't want
> indexed or seen just yet. I'ts a trickly URL that you couldn't find by
> accident, probably, but I don't want it indexed for another few weeks
> yet.
>
> Any ideas and suggestions would be greatly appreciated.
Look here:
http://www.robotstxt.org/wc/robots.html
Keep in mind that the robots exclusion protocol is advisory--which means
that all robots *should* obey it but they don't *have* to. I think most
reputable search engines do, but there may be aggregation services that
don't.
If you want to use it for security, don't. You are really just telling
potential hackers where to look.
Keary Suska
Esoteritech, Inc.
"Leveraging Open Source for a better Internet"
This archive was generated by hypermail 2a24 : Sun Feb 03 2002 - 16:52:07 MST