This illustration was inspired by Jenny List.

Context

The internet is fragile.

Pipeline

Initial Acquisition

  1. Crawl
  2. Scrape
  3. Verify

Each stage typically involes an extraction operation - during crawling, this is extraction of links and during scraping it is the extraction of content. Network requests are expensive, so extraction operations are better suited on local resources.

Updates

Once content has been aquired, keeping it up to date is a challenge. At scale, some sort of automation is required. Automation logic is fragile, so monitoring is critical.

For certain sources, RSS is available and may be adapted to meet the needs of the archive.

Content

Blogs

Source Available (git)

Git can be used for both initial acquisition and updates for source available websites. This is edit safe (the history of published content that is later modified is preserved).

One difficulty that needs to be considered is if builds are not included in the source - additional software may be required to build the website; for example, a particular version of Hugo may be needed to convert markdown to html (this is not ideal).

Update Available (RSS)

RSS can be used for updates. Initial aquisition will still require a custom crawler and scraper. This scraper can be reused for new targets identified by RSS. This is not edit safe.

Site Available

For blogs where the published site is the only source and which does not contain RSS capabilites, a custom recrawler needs to be crafted. The recrawler is very suseptible to errors as a result of layout changes and should be monitored for indications of such errors. This is also not edit safe.

YouTube

A asset uploaded to YouTube is re-encoded to various formats. TCP guarentees that the delivery of a particular asset is perfect - the source and destination are bitexact. However, various attributes of the asset (including metadata and description) may result in slight variances. For a given quality, it should technically be possible to extract a bitexact copy of the source, regardless of when it was acquired.