Search engine robots are sometime also called spiders or crawlers and their job is to find web pages. The question is what is they really do and is what they do all that important. They are essentially ‘data retrieval programs’.

If your site cannot be found it may be because the search engine robot cannot find your site, or it may be more complicated such as the robot not being able to crawl your site, or they cannot understand what is on your web page.

There are certain things which block robots from doing their work, perhaps a site is built using frames or has flash movies. Robots are not able to enter areas protected by passwords and they cannot click on buttons. Their search they can be impeded if they come across JavaScript navigation and if they can be stopped in their tracks if they come across a dynamically generated URL.

How do they work?

Robots spend their day by travelling and finding links and information. A robot has a list of websites which it has to ‘troll’; submission of a new URL is added to a queue on this list. A website can also be added to this queue via a link which is another reason to Build Inbound Links with popularity in mind.

When it’s your websites turn, the robot checks to see if you have a robot.txt file which tells it which areas of your site are off limits, such are directories which contains information that does not need to be indexed. The robot then starts to collect information and follow links from page to page which is the only way a robot can find its way around the World Wide Web.

When a robot reaches your site it first looks at all visible text and the content contained within various Meta Tags in the pages source code. It also looks at hyperlinks. The information it finds in the text and from links helps the search engine understand what your site is all about. It is the responsibility of the search engine to decide what matters and it does this by means of algorithms.

Once the information has been evaluated it is accorded a certain value, and is ranked according to this value which is then indexed and sent to the database where it is available for searchers looking for a specific item. Once a searcher enters their query, a calculation is done and the result is that the right set of relevant results are given to the searcher. Only those pages which have been visited by a robot and in turn sent back to the database for indexing are shown.

It is possible to track where a robot has visited by means of server logs or results from log statistic programs. This information will tell you when they visited your website, which pages they visited and how often they visited. Search Engine Spiders are identified by names given them by their ‘master’ search engines. Inktomi calls their robot ‘Slurp’.

Get Involved