Analysis of the robots.txt file


Hello Engines! analyzes the robots.txt file of your website. Crawlers look for this file and read it. You can use this file to leave instructions for crawlers (e.g. to exclude certain pages of the website that are still under construction).