Subject: Re: Apache on YDL question
From: Trevor J. Mahan (tjmahan@earthlink.net)
Date: Sun Feb 03 2002 - 16:49:05 MST
Thanks Keary.
No I'm not planning on using it for security. I just need to keep the
bots away from indexing for a few weeks while I finish the project:)
Trevor
----- Original Message -----
From: "Keary Suska" <hierophant@pcisys.net>
To: "Yellow Dog Linux" <yellowdog-general@lists.yellowdoglinux.com>
Sent: Sunday, February 03, 2002 4:36 PM
Subject: Re: Apache on YDL question
> on 2/3/02 3:57 PM, tjmahan@earthlink.net purportedly said:
>
> > How do I use robots.txt keeps the spiders out of a site? I have
> > development files up that I don't want
> > indexed or seen just yet. I'ts a trickly URL that you couldn't find
by
> > accident, probably, but I don't want it indexed for another few
weeks
> > yet.
> >
> > Any ideas and suggestions would be greatly appreciated.
>
> Look here:
> http://www.robotstxt.org/wc/robots.html
>
> Keep in mind that the robots exclusion protocol is advisory--which
means
> that all robots *should* obey it but they don't *have* to. I think
most
> reputable search engines do, but there may be aggregation services
that
> don't.
>
> If you want to use it for security, don't. You are really just telling
> potential hackers where to look.
>
> Keary Suska
> Esoteritech, Inc.
> "Leveraging Open Source for a better Internet"
>
This archive was generated by hypermail 2a24 : Sun Feb 03 2002 - 17:03:30 MST