How Search Engine Robots Work ?

Search Engines do not actually search the internet each time somebody types in a search query. This would take far too long. Instead, what they do is search through their databases of web sites they have already indexed. The Search Engine robots find pages that are linked to from other pages that they already know about or pages that are submitted to them. When a web page is submitted to a search engine, the url is added to the search engine bots queue of websites to visit. Even if you don't directly submit a website, or the web pages within a website, most robots will find the content within your website if other websites link to it. That is a part of the process referred to as building reciprocal links. This is one of the reasons why it is crucial to build the link popularity for a website, and to get links from other relevant sites back to yours. It should be part of any website marketing strategy you opt in for.

The search engine databases update at varying times. Once a website is in the search engine database, the bots will keep visiting it regularly, so as to pick up any changes that are made to the websites pages, and to ensure they have the most current data. The number of times a website is visited will depend on how the search engine sets up its visits, which can vary per search engine. However, the more active a website, the more often if will get visited. If a website varies frequently, the search engine will send bots more often. This is also true if the website is extremely popular, or heavily trafficked.

Sometimes bots are unable to access the website they are visiting. If a website is down, the bot may not be able to access the website. When this happens, the website may not be re-indexed, and if it happens repeatedly, the website may drop in the rankings.

Types of Search Engines

There are basically two types of search engines which gather their listings in different ways.

Crawler Based Search Engines - Crawler based search engines, such as Google, create their listings automatically. The Google Robot crawls or spiders the web, then people searches through what they have found.

How do you identify a Search Engine Robot?

Search engines send out what are called spiders, crawlers or robots to visit your site and gather web pages. These Search Engine robots leave traces behind in your access logs, just as an ordinary person does. If you know what to look for, you can tell when a spider has come to call. That can save you worrying that you haven't been visited. You can tell exactly what a robot has recorded or failed to record. You can also spot robots that may be making a large number of requests, which can affect your page impression statistics or even burden your server.

A better way of spotting spiders is to look for their agent names, or what some people call browser names. Spiders or search engine robots have their own names, just like browsers. For example, Netscape identifies itself by saying Mozilla. Alta Vista's spider says Scooter, while yahoo's spider is named Slurp.

Search Engine Robots Crawling Problems

Search engine robots follow standard links with slashes, but dynamic pages, generated from databases or content management systems, have dynamic URLs with question marks (?) and other command punctuation such as &, %, + and $. Search Engine Robots find difficult to crawl such dynamic sites as they include these blocking parameters in url's.

The simplest Search Engine Optimisation solution is to generate static pages from your dynamic data and store them in the file system, linking to them using simple URLs. Site visitors and robots can access these files easily. This also removes a load from your back end database, as it does not have to gather content every time someone wants to view a page.

Search engine robots sometimes have problems finding pages on the web. Spidering issues can be caused by Macromedia Flash sites which are coded with image based language, rather than a text based language, which the search engine robots can't read. Search engine robots can have difficulty in penetrating Javascript navigation menus as well.