Topic awaiting preservation: Dynamic Pages & Search Engines

Hiyas all, just wondering if anyone knows what the go is with dynamic pages and search engines.

I guess the problem is with dynamic pages, they dont really exist until you goto them, so is it possible for a search engine to find the data on that page?? I think this is a problem i have heard a bit about, so im a bit reluctant to make dynamic pages. If it werent a problem i would go ahead and make my entire site dynamic...

I have made a dynamic database product catalogue however, which is what many people do. Will this mean that people wont be able to find products on my page from a search engine?

The only thing i can think of is if search engines follow links through and cache all pages, but my product catalogue is for about 10 companies where u choose the company via a pull down list <select>. Im not sure if search engines will follow these through.

So does anyone have any idea the facts behind it all, what would you suggest and are there any alternative solutions (say text files that the search engine can use which point to dynamic sections of your site). Any help would be great, thanks all.

H][RO: I'm currently sketching a couple of tutorials out for the GN which touch on this subject.

Basically search engines used to go nowhere near a page with a dynamic URL (e.g. my_page.php?id=13) but in the last year or so some (like Google) have tended to index the first dynamic page and go no further. I also suspect they tend to rate 'static' URLs higher than dynamic ones (everything else being equal).

The trick is to do something to present a seemingly static URL to the browser whilst the server process it server-side and pases variables through to the dynamic pages. We are going this in the FAQ for example and over at the GN. It all depends on what your platform is - Apache is really good as it has the mod_rewrite module (and if you are using that then drop the dynamic style URLs in and we'll throw back a quick expression which you can use) but I know there are things in ASP and ColdFusion that can also be used.

I should start an FAQ on this now I think about it.

[edit: No sooner said than done:

:FAQ:

I'll throw more stuff in there later]

___________________
Emps

FAQs: Emperor

Thanks for that emp thats great. Im a little confused: so something that had my.php?ID=13 is treated as dynamic, something with my.php or my/13/ is treated as static tho? /*edit: i forgot to mention its a windows server also (does that mean its likely to be apache?) i dont know enough about the server unfortunately and wouldnt be able to change anything. I think im going to change hosts tho they are still running php 4.1.1 and wont update


In regards to following links through, any idea how that works? do they follow through links if they are from a pull down menu? My product page lets you select a product category from a pull down, and pulls the relevant data from a database to produce the page, is it a problem that its from a pull down?

Thanks again



[This message has been edited by H][RO (edited 01-12-2003).]

H][RO: Use you phpinfo() page to see if it is running on Apache (it is perfectly possible to run Apache on Windows).

However, if the only links into your product catalogue are from a pull down menu then yes you could be in trouble - its possible that some might but such features would (I believe) stop a robot dead. Its also a bad idea to use such a thing as your principal product navigation anyway.

___________________
Emps

FAQs: Emperor

Its a bad idea? doh, i thought pull down menu's would be a good way, what do you think is a better way? Im not using it for product searching or anything but to choose which companies products you want to veiw and which line of products.


here is what i was going to do:

http://jt-design.net/aae/productPages/products.php

any suggestions then?

Hmm doesnt seem to be apache either =/

http://jt-design.net/aae/php/phpinfo.php

While a Drop Down Menu is a perfectly nice back-up navigation system (for instance, after you choose a company, the DDM could be placed on the subsequent pages in case the viewer wants to pick a different company), for the primary page where you select the company, I would personally just put the options in a list menu.

If the list isn't going to get much longer than it is now, it won't be a problem with space at all. If it is going to be significantly longer, then you may need other options.

Yeh i guess it would probably take up roughly the same room, although the list is going to get a bit bigger, probably another 5 items i would say. mmm....

Do robots only follow through physical links? if i had a link specified in the code with nothing to actually click on, would the robots follow it through? the only problem i have with a pull down personally is search engines. I guess it could also cause issues with text based browsers tho =/

Google seems to have no problems indexing or following dynamic pages or query strings. Although Google does limit the amount of dynamic pages it indexes. I gather that many search engines don't index dynamic pages with query strings or session IDs because of the danger that the spider/bot gets trapped in a loop.

quote:

---

We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index.

---

Yeh i've decided for now to just make separate static pages, with the layout its not much work anyhow. I will still use dynamic product pages but not much i can do about that, i might make another static page with a simple product listing so they can be searched,