Differences
This shows you the differences between two versions of the page.
| — |
chkl-robots.txt [2014/08/11 12:01] (current) stephan created |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | ====== Use a robots.txt ====== | ||
| + | ---- | ||
| + | {{:wiki:seo3.png?nolink&110|Big Influence}} {{ :wiki:difficulty2.png?nolink&110|Hard to implement}} | ||
| + | In a [[robots.txt]] you can instruct [[crawler]]s to exclude certain areas of your website from the index or not to follow the links on your website. | ||
| + | |||
| + | Additionally you can add a link to your [[sitemap]] in the robots.txt to make sure that the crawlers will find it. | ||
| + | |||
| + | You can use the //Robots Generator// to create a robots.txt and upload it to the root directory of your website. | ||
| + | |||
| + | **Tip:** In the [[Google Webmaster Tools]] you can check which URLs you excluded from the index by clicking on **Crawling** > **Blocked URLs**. | ||
| + | |||
| + | ====== Helpful Links ====== | ||
| + | |||
| + | [[http://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec|Bing Webmaster Help: How to Create a Robots.txt file]] | ||
| + | |||
| + | [[https://support.google.com/webmasters/answer/156449|Google Webmaster Help: Block or remove pages using a robots.txt file]] | ||
| + | |||
| + | **GoogleWebmasters**: If I don’t need to block crawlers, chould I create a robots.txt? | ||
| + | |||
| + | {{youtube>P7GY1fE5JQQ?384x216}} | ||
