Closed Thread Icon

Topic awaiting preservation: advice on robots.txt Pages that link to <a href="" title="Pages that link to Topic awaiting preservation: advice on robots.txt" rel="nofollow" >Topic awaiting preservation: advice on robots.txt\

Author Thread
Bipolar (III) Inmate

From: Chicago
Insane since: Jan 2003

posted posted 06-05-2003 06:20

Hi all,

just wondering...

As I have been building my website, I knew it would be slow going, so I decided to block all robots from crawling my site in my robots.txt file. I didnt want to get listed on google unntil I was ready. I guess now Im ready to open it up, even thought the site will always go through revisions. but I was wondering: Do you suggest opening a site for all bots to index it, or do you list only certain ones that you want? Arent some bots not so nice?

Also I was wondering... Can bots find pages that are hidden or isolated? Ones that are not linked to the rest of the site? I have my resume on one page which no one should be able to find unless I were to personally give them the link to get to it. I also have a ton of old pages that are older versions of pages and trials, experiments- I just have them around because I didnt want to delete them.

thanks mucho


Maniac (V) Inmate

From: the Asylum ghetto
Insane since: Oct 2002

posted posted 06-05-2003 06:46

hmmm... i don't really have an answer to your question... BUT... i do have a link that may help. i haven't really checked any of this out, as i didn't have the i just bookmaked it for later... and later never has come yet

anyways: The Web Robots Pages

hope that helps

Cell 1007::SST

« BackwardsOnwards »

Show Forum Drop Down Menu