Back in the days around 3 B.G (Before Google) AltaVista was the new search engine on the block. In an effort to show off the power of their minicomputers, the AltaVista team at Digital decided to crawl and index the entire web. This was at the time a new concept. Many web masters didn't relish the idea of a "robot" program accessing every page on their web site as this would add more load to their web servers and increase their bandwidth costs. So in 1996 the Robots Exclusion Standard was created to address these web master concerns.
You can use a simple text file called robots.txt to keep search engines out of a directory. Here is a very simple example that will prevent all search engines (user-agents) from accessing the /images directory.
User-agent: * Disallow: /images
When you block the /images directory, you also block all subdirectories. For example, the directory /images/logos and the file /images.html will also be disallowed.
Strange enough, the first draft of this standard did not contain an "Allow" directive. Later on this has been added, yet without a guarantee of support by all search engines. This implies that anything not specifically disallowed has to be seen as a target for web crawlers.
If you choose to disallow access to your entire web site, you can use a robots.txt like this:
User-agent: * Disallow: /
The next lines apply to every search robot when the User-agent is *. Through the specification of the signature of a web crawler as User-agent specific instructions can be given to such a search robot.
User-agent: Googlebot Disallow: /google-secrets
The protocol has been changed since the initial spec was put into place. Wildcards have proven to be an extension that has proved popular.
User-agent: Slurp Disallow: /*.gif$
This prevents Yahoo! (whose web crawler is called Slurp) from indexing any files on your site that end with ".gif". Keep in mind that wildcard matches are not supported by all search engines so you have to preface these lines with the appropriate User-agent line.
You can combine several of the above techniques in one robots.txt file. Here's a theoretical example.
Computer applications work great when it comes to following well defined instructions. The human brain however is less efficient at these functions, so the best advice is to keep things simple.
For us mortals there is a robots.txt analysis tool in Google's webmaster tools. Highly recommended. Another good resource for more information on the Robots Exclusion Standard is www.robotstxt.org.
Today when companies are spending a lot of money to be included in search engine listings, the idea of excluding your content may seem quaint. But from a security perspective there are many valid reasons for limiting what a search engine indexes on your site.
Thinking out of the box might be a term that has been done to the death. But the truth is that when it comes to search engine optimization, it is important to think out of the box on a regular basis. In fact, the concept of thinking out of the box holds true for a number of search engine optimization concepts. One of the most demanding and time consuming aspects of search engine optimization is link building. Link building has become such a complex and vast field that link building is now considered a specialized field of search engine optimization. Contrary to popular belief, link building is more than just searching for high Page Rank directories. A good link building campaign utilizes the following components: Link Baiting Link baiting is the process of offering webmasters useful scripts, documentation or research material for free and in return, the website or webpage offering the widget receives an incoming link. Link baiting is one of the most hassle free yet creative ways of generating incoming links to a website. One of the most popular link baiting techniques utilized by SEO websites is offering a small widget that displays the Page Rank of a website with an incoming link to the SEO website. Status: Any website serious about increasing their Page Rank above 5 should consider link baiting. Article submission Article submission services have matured considerably over a period of time, and webmasters have started paying search engine optimization firms to write meaningful articles and post them on various Article websites. Depending on the quality of the article and its usefulness, a single article can be displayed on nearly 100 article websites. Each article has a footnote knows as the resource box that is designed to carry outgoing links to a website thus allowing for effective link building. Status: The degree of acceptance varies from site to site, but articles are a great way to carry out link building with LSI in mind. Website Reviews Website reviews posted on various blogs are a fantastic source of link building. Unlike the sources mentioned above, a single website review can bring multiple text rich links to a website. In addition, the Page Rank of the blog can vary considerably and popular blogs can offer outgoing links from a webpage that is Page Rank 4. Although there has been considerable criticism of website reviews and Google in particular is keeping close tabs on website review blogs, there is not denying that website reviews are a great way to increase the Page Rank of a website. In addition, website reviews are also being compared to paid links as the webmaster does pay the blogger money to post a website review, but as it stands there is very little stop webmasters from harnessing the power of this new phenomenon. Status: Is a relatively new concept that will require time to pick up and the true impact will only be visible over a period of time.
Both Nick Dalton. & Ginfocv are contributors for EditorialToday. The above articles have been edited for relevancy and timeliness. All write-ups, reviews, tips and guides published by EditorialToday.com and its partners or affiliates are for informational purposes only. They should not be used for any legal or any other type of advice. We do not endorse any author, contributor, writer or article posted by our team.
Nick Dalton. has sinced written about articles on various topics from . Nick Dalton is an expert who regularly writes articles for. Nick Dalton.'s top article . to your Favourites.
Ginfocv has sinced written about articles on various topics from Blogging, Increase Adsense Revenue and Adsense. For more useful tips & hints, please browse for more information at our website:-. Ginfocv's top article generates over 5400 views. to your Favourites.