Web archiving

Web archiving is the collection and preservation of websites and other online material for future access. It captures changing digital content so it can be studied after the original material disappears or changes.

Web crawler: A program that automatically discovers and retrieves web pages by following links or other rules. Crawlers traverse sites and gather the pages that an archive can preserve.
Internet Archive: A nonprofit digital library that preserves and provides access to books, media, software, and archived web pages. Its large-scale web collection demonstrates public access to preserved online material.
Brewster Kahle: An American computer engineer and entrepreneur who founded the Internet Archive. His project made systematic preservation of the public web an institutional goal.
Digital obsolescence: The loss of access to digital information when its formats, software, or hardware become unusable. Preserved captures can still become unreadable as the tools needed to interpret them disappear.
WARC: A file format for storing aggregated web resources and metadata from web crawls. WARC files package captured pages and related records for storage and exchange.
Archive-It: A subscription web-archiving service operated by the Internet Archive for libraries and other institutions. It enables institutions to build and manage collections of selected websites.
Alexa Internet: An internet analytics company that supplied web-crawling data and technology to the Internet Archive in its early years. Its crawling infrastructure supported some of the earliest Internet Archive captures.
Robots exclusion standard: A convention through which websites request that automated crawlers avoid specified content. Respecting crawl restrictions can leave gaps in the historical record.
Memento (protocol): A protocol that connects present-day web resources with archived versions available at particular times. Memento lets systems discover archived versions by requesting a resource's time-specific history.
UK Web Archive: A collection of websites preserved by the British Library and other UK legal deposit libraries. It shows how a national collection captures websites within a defined jurisdiction.

Linked from 7 pages