The Robots Text File Or How To Get Your Site Properly Spidered, Crawled, Indexed By Bots
So you heard about someone stressing the importance of the robots.txt file, or noticed in your website's logs that the robots.txt file is causing an error, or somehow it is on the very top of the top visited pages, or, you read some article about the death of the robots.txt file and about how you should not bother with it ever again. Or maybe you never heard of the robots.txt file but are intrigued by all that talk about spiders, robots and crawlers. In this article, I will hopefully make some sense out of all of the above. There are many folks out there who vehemently insist on the uselessness of the robots.txt file, proclaiming it obsolete, a thing of the past, plain dead. I disagree. The robots.txt file is probably not in the top ten methods to promote your get-rich-fast affiliate website in 24 hours or less, but still plays a major role in the long run. First of all, the robots.txt file is still a very important factor in promoting and maintaining a site, and I will s...