MDN self-hosted library · e01.org

Crawler

A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

no runnable cells glossary-definition upstream source

A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

See also

  • Related glossary terms:
  • Search engine
  • Web crawler on Wikipedia