Differences

This shows you the differences between two versions of the page.

Link to this comparison view

crawler [2014/08/11 13:15] (current)
stephan created
Line 1: Line 1:
 +====== Crawler ======
  
 +----
 +
 +===== Definition =====
 +
 +A **crawler**,​ also called //robot//, //bot//, //search bot// or //spider//, is a software that analyzes [[website|websites]] automatically and repeatedly in order to gather information and save it to the index. They find websites by visiting directories and following the links from websites.
 +
 +You can check how often crawlers visited your website in the log file of the internet provider.
 +
 +----
 +
 +===== Importance for SEO =====
 +
 +Analyzed websites are saved in the search engine'​s index and will be visited regularly by crawlers and analyzed again.