Differences
This shows you the differences between two versions of the page.
| — |
260 [2014/08/11 10:43] (current) stephan created |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | ====== Analysis of the robots.txt file ====== | ||
| + | ---- | ||
| + | |||
| + | Hello Engines! analyzes the robots.txt file of your website. [[crawler|Crawlers]] look for this file and read it. You can use this file to leave instructions for crawlers (e.g. to exclude certain pages of the website that are still under construction). | ||
