Generally in case of small websites webmasters are under false assumption that they do not require to create a robot.txt file. But it is required. First of all let us define the robot.txt file. Even prior to it we need to define what a web robot is. A web robot is also called spider or crawler which should not be confused with the normal web browser as web browser is not a robot.

The main use of robots.txt file for webmasters in is to give instructions to the robot what they crawl and what should not be crawled. This can give you some control over the robots. This give you little more control over the robots and this indicates that you can issue indexing instructions to various search engines.

Robots.txt invites the search engines. Some of the good bots can also step away from your website in case you have not created robot.txt in the top level of your website. Some time there is requirement to exclude some pages from search engines. These are those pages that are still under construction and those directories that you do not want to get them indexed. You may also want to exclude those search engines whose main aim is to collect email addresses.

Robot.txt file is a simple text file created in notepad. This is required to be saved to the root directory of your website. It means that directory where your home page or index page is stored. In order to create a simple robot.txt file with the specification that allows all robots to spider your website, write the following info:

User-agent: *

Disallow:

This will allow all robots to index your pages.

In case you don't want a specific robot to have access any of your website pages. Then do the following:

User-agent:specificbot

Disallow: /

In case you do not want a specific robot to access any of your web pages, then do the following. Suppose you do not want Googlebot to index a page names as ?abc? and you directory name is newdir. In the disallow section you will be required to put:

User-Agent: Googlebot

Disallow:/newdir/abc.html

IN case you do not want to get indexed the complete directory then you would put:

Now if it's a complete directory you do not want indexed you would put:

User-Agent: Googlebot

Disallow:/newdir/abc.html/

By putting forwarding slash in the beginning and in the end, search engines are informed that not to include any of the directories.

Thus create a robot. Text is an important part in SEO services and it can not be ignored at all costs.