When I talk to my clients about spiderability, I'm generally referring to two things...



1) Are all the links in the site true hyperlinks that can be picked up properly by the search engines.

2) Are all the pages within the site reachable within 2-3 clicks from the homepage.

So let's go over the above two areas of concern.

1) Hyperlinks. This may seem almost silly, but you would be amazed at the number of sites I run into when doing consultations and website analyses that have non-standard hyperlinks. By "non-standard", I'm referring to javascript generated hyperlinks or hyperlinks embedded within flash files.

There is nothing inherently wrong with javascript or flash when used properly, but that simple fact is that javascript and flash are NOT search engine friendly. Google is pretty much the only engine that is able to pick up links within javascript or flash code. At this time, I have seen no evidence that either Yahoo or MSN have this ability.

While Google may be able to pick up links, it is unclear as to whether or not Google places any VALUE on the links it finds in this manner. Remember, much of a pages ranking in Google is determined by links so you want to be absolutely sure that each and every link is valued.

So, be absolutely sure that your links are true hyperlinks (by "true" hyperlinks, I'm talking about hyperlinks coded with the normal href tags) if you want to make sure they are found, followed, and counted by all the engines.

2) Distance from Homepage. Ideally, you want your visitors and the search engines to be able to reach any page within your site within a maximum of three (3) clicks and preferably two clicks. The more clicks it takes to reach a page, the less chance there is that the search engines will index that page.

It is for this reason, that site maps have become so popular. By utilizing a sitemap, you are able to link from your homepage to a page that lists all or most of the links to the various pages of your site. The search engines (and visitors) are then able to get to virtually any page of your site within just a couple of clicks.

You'll notice I've mentioned not only the search engines but the visitors as well in the above paragraphs. By reducing the number of clicks it takes to get from your homepage to any page on your site, you will find that you also increase the overall usability of your site.

While site maps can definitely help to increase the spiderability of a site, it is important to remember that they are not a total fix for bad navigational structure within a site. As mentioned, all of the search engines utilize a pages link popularity in one way or another in their algorithms.

In general, the homepage of a site will have the highest link popularity of any page within the site. This is because most inbound links to a site are pointing to the homepage. It's from the homepage that all the internal pages derive their link popularity from a sort of "trickle down" affect.

A site map will only derive a certain amount of link popularity that it can pass on to the pages it links to. To understand this best, think of the homepage as a large river with each link on the homepage a smaller river branching off from the main river. Each river will be fed a similar amount of water by the main river. Alone one branch of the river will never be able to deliver as much water to the various areas as all the branches of the river can combined.

The site map is one branch of your sites link popularity river and it has value, but it will never have the same impact as a well throughout and implemented links structure that makes use of all the rivers of link popularity within your site.

To make the most use of the link popularity of your site, you should try and setup your sites navigational structure so that even without a site map, the search engines and visitors are STILL able to reach any page within your site within 2-3 clicks.

So...to make a long story short...Always be sure to utilize true, standard hyperlinks throughout your site and be sure that your sites navigational structure allows any page of your site to be reached within no more than 3 clicks.

See you at the top!