Passwords, scripts, personal and other private information are a few examples of information that you might not want Google to google with. Or maybe, you have sales and promotional pages in your database that need to remain exclusive to your clients. These are good reasons for keeping the spider "bots" out of these files. However, since the goal is for your website to achieve the highest rank possible when potential buyers conduct an online search, you do want Google's or other search engine's robots to have access to most of your website's information and pages. So how do you go about keeping your secret stuff secret?



One of the most powerful and popular methods is the use of a robots.txt file that is stored in the root directory of your site. This system, if used properly allows search engine robots to access your website pages without accessing private files. This text file acts like a security guard for your private files. It will only allow the "bots" to see what you want them to see. It also gives you the power to block a particular search engine or engines from gaining any access at all if that is what you wish. When a "bot" starts crawling your site, it begins by checking to see if you have a robot.txt file that will prevent it from accessing certain pages of your site. The "bot" understands code language that you will use when setting up your robot.txt file.

Of course, for many website owners, the main reason they have a website is to benefit from Adsense dollars on as many pages as possible. So, you might be wondering that if you block certain pages, how you will be able to generate revenue from advertising sources. Adsense is a program whereby you can allow advertisers to display their ads on your site. When someone clicks on one of these ads, you will be paid for that click. Usually, it is only a few cents per click, but they add up. Many individuals make a nice side income strictly from Adsense revenue. To sign up for this program, visit http://www.google.com/adsense. But, back to the question about how this will work, if certain pages are blocked. The answer is again in the robot.txt file. Simply allow the user-agent MediaPartners-Google to gain access to these pages. This agent does not share pages with the other "bots", so your information will remain private.