Should I Use Robot TXT On MY Website
I was asked a good question today by one of my readers asking if they should use a robot txt file on there website.
Most sites now days use a robot txt file as a lot of content management scripts require a bit of tweaking for search engine optimisation purposes. The main cms that comes to mind is wordpress it’s a good idea to create a robot txt file if you use wordpress as there are some duplicate content issues.
Should I Use Robot TXT On MY Website
A robot txt is mainly used to tell the search engines which parts of your site you don’t want crawled and indexed. For example most peoples “contact us” page doesn’t hold any real weight and isn’t used for sales or a landing page so what people tend to do is block it from the search engines, this also helps to control your pagerank more.
Some sites wont need a robot txt file but the ones that have been around for a few will probably need to use one just to keep their site tidy in the eyes of google.
Should I Use Robot TXT On MY Website