What do they do?

Understanding how search engine robots, or spiders, work is not only for webmasters, it is imperative for anyone undertaking an SEO Campaign.  Knowledge goes a long way in helping optimize your website and eek out the best from it, especially in the age of so many search engines.

Search engines use spiders to index websites.  In other words, they create an index which lists a website so that those searching for a particular product or service can find them via their search facilities.  The more correct and accurate a search engines index, the more popular they are amongst users.

For instance, a search engine indexes the first words that are found on a website.  They use Meta Tags do this, that is why it is so important to make sure that you have good Meta Tags on your pages.  Meta Tags must include very important information first because it what describes your product or service which is the reason for your website.  It is this information that is listed on the search engine index.  Most Meta Tags can only contain about 200 words, or characters.

However, search engines do not only look at Meta Tags, and some don’t even look at them and because of this fact it is important that you mirror the information you have in the body of your website in your Meta Tags.

It takes time for a search engine spider to do its job. Some spiders will only look at between 2 and 10 pages per week, this is why it takes longer.  In the case of Alta Vista this is three months whereas other spiders on average take only about six weeks or so to index your entire website.  Alta Vista calls their spider ‘Scooter’.

How do Spiders work?

Search engine spiders are automated programs which are run by the search engine.  They visit a particular website with the purpose of reading its content, including what are called Meta Tags as well as Title Tags and HTML Tags.  They also follow all the links that are connected to the site and will index them as well.

As soon as you submit your website to a search engine, and after you complete their submission page, the spider commences its job.  It goes out, reads your site and then returns its findings to a central repository which is where the data is indexed.   They return to your site periodically to check of any information has changed, one cannot decide the frequency as this is determined by moderators.

You could also look at a spider as something like a table of contents.  A spider can actually index up to a million pages per day which includes content, links and references of all websites it investigates.

Different search engines produce different ranking results.   This is because search engines use different algorithms and database searches.

As a spider visits each webpage it reads, indexes and stores information.   The process that goes into converting a webpage to an entry on a results page is very complex and consists of very sophisticated ‘data warehousing’ and ‘information retrieval processes’ which vary from search engine to search engine.

Get Involved