Robots.txt Is often a file that exists on the basis Listing of every Web site and can be utilized to instruct search engines like google on which directories/data files of the website they are able to crawl and involve of their index. This is often what search engines like yahoo https://news.rafeeg.ae