This shows you the differences between two versions of the page.

Link to this comparison view

260 [2014/08/11 10:43] (current)
stephan created
Line 1: Line 1:
 +====== Analysis of the robots.txt file ======
 +Hello Engines! analyzes the robots.txt file of your website. [[crawler|Crawlers]] look for this file and read it. You can use this file to leave instructions for crawlers (e.g. to exclude certain pages of the website that are still under construction).