A web crawler is a program, often called a bot or robot, which systematically browses the to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

See also

Related glossary terms:

• Web crawler on Wikipedia