Spiders are not always eight legged horrors that give most people the creeps. Search engine spiders are a website’s best friend. These spiders go by many names; search bots, web crawlers, index robots to name a few. The job of the search engine spiders is to come and crawl around your website. They do this to see what you have to offer the internet population and if it is any good.

Search engine spiders never sleep or rest; day in and day out they crawl around billions of web pages around the internet looking for fresh information. When they find fresh and interesting content they will index it.

This is what you want the spiders to do, because you want them to index your web pages so that the people in your target audience can find you. The spiders are not limited to only crawling specific websites or web pages.

Indexing

They follow links from page to page and index as they go along; what they do is commonly known as ‘spidering’. As the spiders hop from link to link, they store all the information they find on your index page. From there they follow the links to the other pages and so on. In a nutshell, this is how search engines are able to keep indexing the billions of web pages on the internet.

Issues that spiders have

Search engine spiders come across problems when they try to crawl web pages within a website; they battle to find the pages. There are a few reasons for this happening such as the page not being linked to any other pages on the website. Thus the spiders have no link to follow, so they cannot get to the page.

Spiders cannot read images; to them it is invisible so if an entire page is Flash based, the page becomes invisible to them. If you absolutely have to use Flash, you have to use optimised text as descriptions for the spiders to read. The spiders also have problems with a Javascript navigation menu. It can prevent the spiders from being able to penetrate the menu.

Deal with spider problems promptly

Avoid using Javascript and Macromedia Flash if it is at all possible. If you absolutely have to use Flash, it is imperative to provide the spiders with an alternative navigation system. You can do this in the site map as well as the page footer. Also replace your Javascript navigation menu with the far spider friendlier CCS layout. Spiders absolutely love CCS and can crawl it with ease.

Finally, make doubly sure that each and every one of your web pages are linked and make sure that you regularly check that each of these links is working. The search engine spiders are your best friends on the internet. Without them your potential clients will never see your website. It is incredibly easy to please the spiders by removing the obstacles that make it difficult to navigate through your website.

Get Involved