Topic awaiting preservation: advice on robots.txt (Page 1 of 1) $Pages that link to <a href="https://ozoneasylum.com/backlink?for=6348" title="Pages that link to Topic awaiting preservation: advice on robots.txt (Page 1 of 1)" rel="nofollow" >Topic awaiting preservation: advice on robots.txt <span class="small">(Page 1 of 1)</span>\$

Eric001 Bipolar (III) Inmate From: Chicago Insane since: Jan 2003	posted 06-05-2003 06:20 Hi all, just wondering... As I have been building my website, I knew it would be slow going, so I decided to block all robots from crawling my site in my robots.txt file. I didnt want to get listed on google unntil I was ready. I guess now Im ready to open it up, even thought the site will always go through revisions. but I was wondering: Do you suggest opening a site for all bots to index it, or do you list only certain ones that you want? Arent some bots not so nice? Also I was wondering... Can bots find pages that are hidden or isolated? Ones that are not linked to the rest of the site? I have my resume on one page which no one should be able to find unless I were to personally give them the link to get to it. I also have a ton of old pages that are older versions of pages and trials, experiments- I just have them around because I didnt want to delete them. thanks mucho Eric
Lacuna Maniac (V) Inmate From: the Asylum ghetto Insane since: Oct 2002	posted 06-05-2003 06:46 hmmm... i don't really have an answer to your question... BUT... i do have a link that may help. i haven't really checked any of this out, as i didn't have the time..so i just bookmaked it for later... and later never has come yet anyways: The Web Robots Pages hope that helps __________________________ Cell 1007::SST

Topic awaiting preservation: advice on robots.txt (Page 1 of 1) $Pages that link to <a href="https://ozoneasylum.com/backlink?for=6348" title="Pages that link to Topic awaiting preservation: advice on robots.txt (Page 1 of 1)" rel="nofollow" >Topic awaiting preservation: advice on robots.txt <span class="small">(Page 1 of 1)</span>\$

Eric001
Bipolar (III) Inmate

From: Chicago
Insane since: Jan 2003

posted 06-05-2003 06:20

Hi all,

just wondering...

As I have been building my website, I knew it would be slow going, so I decided to block all robots from crawling my site in my robots.txt file. I didnt want to get listed on google unntil I was ready. I guess now Im ready to open it up, even thought the site will always go through revisions. but I was wondering: Do you suggest opening a site for all bots to index it, or do you list only certain ones that you want? Arent some bots not so nice?

Also I was wondering... Can bots find pages that are hidden or isolated? Ones that are not linked to the rest of the site? I have my resume on one page which no one should be able to find unless I were to personally give them the link to get to it. I also have a ton of old pages that are older versions of pages and trials, experiments- I just have them around because I didnt want to delete them.

thanks mucho

Eric

Lacuna
Maniac (V) Inmate

From: the Asylum ghetto
Insane since: Oct 2002

posted 06-05-2003 06:46

hmmm... i don't really have an answer to your question... BUT... i do have a link that may help. i haven't really checked any of this out, as i didn't have the time..so i just bookmaked it for later... and later never has come yet

anyways: The Web Robots Pages

hope that helps

__________________________
Cell 1007::SST