Crawler


A crawler, also called robot, bot, search bot or spider, is a software that analyzes websites automatically and repeatedly in order to gather information and save it to the index. They find websites by visiting directories and following the links from websites.

You can check how often crawlers visited your website in the log file of the internet provider.


Analyzed websites are saved in the search engine's index and will be visited regularly by crawlers and analyzed again.