I know that we have talked about robots on past blogs but this time I would like to bring forth the topic of undesirable robots, crawlers and spiders. I shall be referring to them collectively as robots. Firstly and most importantly, not all robots are bad. There are malicious robots out there but I won’t Read Blog…
What makes a robot undesirable?
July 9th, 2010 by Ray
Avoid messing up your robots.txt
December 14th, 2009 by Jon
Robots.txt isn’t something that’s often talked about in articles about SEO, but it is an important consideration. Most sites will have some files that they don’t want indexed for various reasons. The function of your robots.txt files, however, isn’t a simple matter of ‘allow’ and ‘disallow’. Your site’s code plays an important role in the Read Blog…









