Web crawler

A web crawler is software that automatically visits web pages by following links or other discovery methods. It collects information for tasks such as search indexing, archiving, and site analysis.

URL frontier: A queue or set of URLs awaiting retrieval by a crawler. It holds discovered pages until the crawler is ready to fetch them.
Web search engine: A system that discovers, organizes, and retrieves information from the web in response to queries. Search engines use crawlers to find pages for their indexes.
URL: A standardized address that identifies a resource on a network. A crawler uses URLs to identify pages it can request.
Web server: Software or hardware that responds to requests for web resources. Crawler traffic consumes server capacity and can affect site performance.
Web browser: Software that retrieves and presents web content for interactive use. Browsers serve direct human interaction; crawlers automate systematic page visits.
Robots exclusion standard: A convention that lets website operators specify which paths automated crawlers should access. Crawlers consult robots.txt rules before requesting many pages.
Search engine indexing: The process of analyzing and storing web content so it can be retrieved by a search engine. Crawling supplies pages that indexing systems can analyze.
Hyperlink: A reference in a document that points to another resource. Links are a major source of new URLs for a crawler.
Rate limiting: A control that restricts how frequently a client can make requests. Sites use it to manage or constrain crawler request volume.
Web scraper: Software that extracts specific information from web pages. A scraper targets fields to extract, whereas a crawler focuses on discovering pages.

Linked from 12 pages