The search engine optimisation of a site involves a lot of different things. Some of these are very technical aspects, while others of them are substantially more creative. The site’s architecture, code and content all have a very important role to play in where the search engines place its link in the search engine results pages.

In the rush to use all available resources of a site, some things can be forgotten. It’s not always necessary or even desirable to show the search engines everything you’ve got. Sometimes, it’s a good idea to keep some files clear of the search engine spiders. The files you designate in your site’s robots.txt are a part of this. The whole subject of robots.txt often gets overlooked in SEO explanations because it sounds too technical for the average person to understand. It is a technical subject, but robots.txt is something it’s important for every site owner to have some knowledge of.

To begin with, robots.txt file can be accessed by typing in domainname.co.uk/robots.txt, substituting the domain of the site in the first half of the address. This will list the directories of the site that the webmaster wants search engines to disallow. This means that the search engine spiders will skip over the information contained in these files.

Robots.txt files can come in very handy beyond the search engines. It is possible to use robots.txt files to protect your site from malevolent web crawlers, which is useful to say the least. However, robots.txt files need to be handled carefully, as it is easy to make mistakes within these tiny little text files. Just because robots.txt files can be used to make the search engines overlook information, it doesn’t mean that the information is invisible. Many site owners make the mistake of including confidential information within a robots.txt file, which can lead to disaster if it is sampled by a malbot. Robots.txt files are also easily visible to the site’s users. Site owners also need to be careful when editing robots.txt files. The disallow feature within a robots.txt file is easy to write, and just as easy to muck up. The difference between the search engines only crawling desired files in your site and failing to crawl the site completely can be a simple ‘/’.

It’s important to note that robots.txt should not be used when swapping pages on your site. Some companies make the mistake of listing old pages in their robots.txt file when the changeover occurs. This can result in any old indexing power being lost. A simple 301 redirect should be used when you want to pass the search power of your old pages onto new ones.

Your robots.txt files can be useful in terms of SEO, preventing duplicate content and directing search engine attention to the important pages of your site. The understanding of proper use of robots.txt is one of the many reasons why it is usually worthwhile consulting an expert when approaching your site’s SEO. You can talk to our experts at SEO Consult for further advice.

Get Involved