Web masters (usually website owners) and content providers began optimizing sites for search engines in the mid-1990s, at first this merely meant you built a website, checked it for spelling errors and then submit it to search engines to be cataloged. Much like a self service library might be run. This was the earlyWorld Wide Web.



Once a web master had submitted a page, or URL, to the various engines they would send a spider to “crawl” that page, extract links to other pages from it, and return information found on the page to be indexed.

The process has evolved so that now a Search Engine Spider downloads a target page and stores it on the Search Engine's own server, where a second program, known as an indexer, breaks down the information, extracts various elements that make up the page, such as the words it contains, the coding information, META tags, description, title, keywords, link text, and where

these are located, as well as any weight or added significance for specific words and all links the page contains, which are then placed into a scheduler for crawling at a later date. The Indexer then gives each element a value according to

the search engine algorithm, the mathematic formula used to determine the overall relevance of the website in regard to certain search terms.

As site owners started to recognize the value of having their sites highly ranked and visible in search engine results webmasters began to try ways to influence the Search Engines. This was the birth of SEO, Search Engine Optimization. As more users began to search for goods and services online, it was possible to track how the search results were used. It became evident that searchers would not often go through many pages of results to find what they wanted. The competition for those high search results became more intense and SEO became and remains vital to website promotion.

Since early versions of search algorithms relied on web master-provided information such as the keyword meta tag, or index files in. Meta-tags provided a guide to each page's content. But using meta data to index pages was found to be less than reliable because the web master's account of keywords in the meta tag were not always relevant to the site's actual keywords.

In short using the information provided by the web master's to determine the search result was like using the fox to guard the hen house. Inaccurate, incomplete, and inconsistent or manipulated data in meta tags caused pages to rank for irrelevant searches. Irrelevant search results began to frustrate searchers. They began to look to other search engines to provide more relevant searches. The search engines that used an algorithm based on information that involved elements actually in the content on the page and how it related to the search.

While they were graduate students at Stanford University, Larry Page and Sergey Brin developed “backrub”, a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links. This means that some links are stronger than others, or give more weight to the PageRank because the contents of the page is more likely to be relevant to the searcher.

In 1998 the grad students founded Google. Google attracted a loyal following among Internet users, who liked the results their system produced. Off-page factors such as PageRank and hyper link analysis were considered, as well as on-page factors, to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more complex, web masters went to work using link building schemes to manipulate search results to their favor. Web masters focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.

To counter the adverse impact of link schemes, as of 2007, search engines again had to evolve to consider a wider range of undisclosed factors for their ranking algorithms. Google has since disclosed that it now uses more than 200 different elements to rank pages. The three leading search engines, Google, Yahoo and Microsoft's Live Search, closely guard the algorithms they use to rank pages.