Show page Old revisions Backlinks This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== Crawler ====== ---- ===== Definition ===== A **crawler**, also called //robot//, //bot//, //search bot// or //spider//, is a software that analyzes [[website|websites]] automatically and repeatedly in order to gather information and save it to the index. They find websites by visiting directories and following the links from websites. You can check how often crawlers visited your website in the log file of the internet provider. ---- ===== Importance for SEO ===== Analyzed websites are saved in the search engine's index and will be visited regularly by crawlers and analyzed again.