A web crawler, also called a spider or bot, is a program that systematically browses the web by following links, fetching pages so they can be processed and indexed.
Crawlers begin from a set of known URLs, fetch each one, extract the links it contains, and queue those for fetching in turn. Search engine crawlers such as Googlebot and Bingbot obey robots.txt and rate limits. Many other crawlers exist — SEO tools, archivers, AI training scrapers — with varying degrees of good behaviour.
Crawling is the first stage of everything. A page that is never fetched is never indexed and never ranks. Crawlers also consume real server resources, and an aggressive or badly behaved one can degrade performance for actual visitors.
Ensure important pages are reachable through crawlable anchor links rather than JavaScript handlers. Monitor server logs to see which crawlers are consuming capacity, verify claimed search engine bots by reverse DNS, and block or rate-limit abusive crawlers that provide no value.
Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.