robots.txt is a text file that tells search engine robots which pages on your website to index and which to ignore. The robots meta tag can also be used to achieve the same thing, but it is generally less effective. The canonical tag is another option for controlling...
