Robots.txt is a small text file with set of instructions for search engine spiders and bots. Contents of robots.txt file can help website owners, webmasters block or grant access to different parts of website to bots for the purpose of indexing on search results. Though not mandatory, it