Search engine spiders are robots; they are programmed to crawl day and night without stopping through the vast space of the Internet. When they find content that they can spider, they go over it and then index those pages. They are not intelligent and you cannot debate with them about why your links are broken or argue the reasons why your code is huge.

They will simply go on their way, they do not wait around for you correct mistakes and fix problems. That is simply not part of their programming. Your website and the crawlability of your web pages is your responsibility.

If your visitors cannot easily navigate your website or be taken to an external site or link because of broken links it means spiders cannot crawl your website. When this starts happening the chances become high that you will eventually be sifted out of the SERPs. When this happens, you have basically killed all your SEO efforts.

Spider killers

Flash links and JavaScript make the spiders extremely unhappy, as they are not able to read any text inside Flash or execute JavaScript. So keep to HTML links to give the spiders access. Bloated code is a major problem because although the spiders are excellent at separating content from code, bloated code make it extremely difficult for them. Another spider killer is the addition of session IDs in URLs as spiders will not crawl these URLs.

Use cookies to store your session Ids. Another URL problem for spiders is the excessively long URLs. The spiders can crawl these long links, but shorter URLs see far more click activity than the long ones. Robot text files should be handled with care. These files tell the spiders how they should view your website. Making even one mistake with the robot text files will kill your crawlability completely and can affect your search engine visibility permanently.

Restore your site crawlability

Decide on your preferred domain; canonicalization. Do not use more than one domain as this dilutes the inbound links. Keep to the primary URL and if a secondary domain is in use, it is imperative that a 301 redirect is included for your primary domain. Keep your URLs short and do not change them unless you are fixing a broken link. URL variability is not a good idea and be fully aware of the problems that session Ids create. Remember that the spiders see hyphens as spaces, so use keywords separated by hyphens in your URLs. Make sure that spiders can crawl your website and do it well. Use the appropriate tools to check.

Sitemaps

By making it difficult or impossible for the search engine spiders to gain access to your web pages you are sabotaging your entire SEO campaign and jeopardising your online success. The best way the search engines will find your website is by building a completely efficient navigating system for your site. Have a logical sitemap that easily guide the spiders to each web page.

Get Involved