How Much of the Web Is Archived?

1. INTRODUCTION

• Public web archives have grown alongside falling preservation-resource costs, production-quality crawling tools, and increased attention to digital preservation.Archives differ in scale, ingest models, collection policies, and crawling and archiving software.Source: §1 · p. 1
• The paper asks how much of the Web is archived, a question not previously addressed despite known archive sizes and an estimate of at least one trillion unique Web URIs.The trillion-URI estimate is attributed to Google, which did not claim to index all of them.Source: §1 · p. 1
• Coverage is estimated from URI samples from DMOZ, Delicious, Bitly, and search-engine indexes, counting archived URIs, timestamped versions, their frequency, and overlap between archives.The paper extrapolates the percentage of the surface Web archived rather than measuring archives’ absolute sizes.Source: §1 · p. 1

2. RELATED WORK

• Coverage research was limited: Day surveyed archiving methods without addressing coverage, Thelwall examined international bias, and McCown and Nelson [13] studied search-engine caches only.The paper notes that little research had examined coverage provided by existing archives.Source: §2 · pp. 1–2
• The question of what should be archived remains unsettled, while the Web’s volume puts complete archiving out of reach.National-archive design involves compromises, and consensus on what to archive has not been reached.Source: §2 · p. 2
• Memento [21] [22] is an HTTP-based framework that lets clients identify and dereference archived representations of an original resource through datetime negotiation.Each URI-R may have zero or more URI-Mi representations, each capturing the URI-R’s state at time ti.Source: §2 · p. 2

3. EXPERIMENT

• The late-2010 to early-2011 experiment aimed to estimate the percentage of publicly visible URIs with copies in public archives and secondarily evaluate archiving quality.The work was organized into three parts: URI sampling, checking current URI status and estimated age, and discovering mementos.Source: §3 · p. 2

3.1 Sample URI Selection

• Each of the four source samples contained 1,000 URIs; Table 1 reports mementos per URI, including means with and without zero-memento URIs, standard deviations, and 95% confidence-level standard errors.The experiment used DMOZ, Delicious, Bitly, and search-engine sources to approximate the Web.Source: Table 1 · p. 2, §3.1 · p. 2

3.1.1 Open Directory Project (DMOZ) Sampling

• DMOZ was included for comparability with prior studies and access to historical directories, but its contents may favor commercial sites and the Internet Archive uses it as a crawling seed.The sample merged 100 snapshots from 2000–2010 into 9,415,486 valid unique URIs, recording when each first appeared.Source: §3.1.1 · p. 2

3.1.2 Delicious Sampling

• The Delicious sample used recently added bookmarks; its Popular random generator was excluded because its set of distinct URIs was too small.Delicious is a social bookmarking service where users tag, save, manage, and share web pages.Source: §3.1.2 · pp. 2–3

3.1.3 Bitly Sampling

• Bitly sampling generated random hashes, dereferenced their short URIs, and retained the first 1,000 that returned HTTP 301 responses.The study recorded creation times from Bitly “+” pages and assumed a Bitly URI was created no earlier than its target URI.Source: §3.1.3 · p. 3

3.1.4 Search Engine Sampling

• Search-engine sampling used a phrase-pool approach from Bar-Yossef and Gurevich to reduce ranking and popularity bias, drawing random URIs from a pool of five-gram queries.The query pool contained 1,176,470,663 queries, from which 1,000 URIs were randomly selected.Source: §3.1.4 · p. 3

• Current-state checks covered whether each resource existed and whether its URI appeared in Google, Bing, and Yahoo! indexes.Source: §3.2 · p. 3
• 95% of Delicious and search-engine sample URIs were live (status 200), compared with 50% of DMOZ and Bitly sample URIs.The authors attribute the difference to recent bookmarks in Delicious and search engines purging caches after discovering 404 responses.Source: §3.2 · p. 3, Table 2 · p. 4
• Google indexing was assessed through Research Search Program APIs; Table 4 reports API results separately from their union with Google-cache URIs found by Memento proxies.The APIs returned results that differed from Google's web interface.Source: §3.2 · pp. 3–4, Table 4 · p. 4

• For each sampled URI, archive-specific proxies queried archives for available Mementos and returned the results to the aggregator.Source: §3.3 · p. 4
• The aggregator merged proxy results, sorted the Mementos by date, and returned a Timemap for the URI.Source: §3.3 · p. 4
• Memento's TimeBundle aggregates the available Mementos for a URI from an archive, using an HTTP-based approach for navigating current and prior resource versions.Source: §3.3 · p. 4

• URI creation dates are usually unavailable, so the study used the earliest available record date as an estimate of URI age.Source: §3.4 · p. 4
• The first Memento date was one candidate proxy for a URI's creation date.Source: §3.4 · p. 4
• The first DMOZ archive report and the date the URI was added to Delicious were also candidate age indicators.Source: §3.4 · p. 4

Temporal distribution

• Most mementos before 2008 came from the Internet Archive, while search-engine caches provided very recent copies.DMOZ and Delicious had similar distributions; coverage was low before 2000, and search-engine caches led at the end of 2010.Source: §4 · p. 4

Mementos per URI

• Bitly had many URIs with no mementos, whereas DMOZ had many with more than 10.The authors attribute the Bitly gap to limited crawler discovery and DMOZ’s high counts to its role as an Internet Archive crawler input and its historical sampling.Source: §4 · p. 4

Memento density

• Memento density was compared with URI-R age using guidelines of 0.5, 1, and 2 mementos per month.Figure 4 plots estimated URI creation date against mementos per URI-R.Source: §4 · p. 4, Figure 4 · p. 7

Backlinks

• Backlink counts had a weak positive relationship with memento counts.Backlinks were treated as one indication of website or page popularity or importance; the relationship was calculated with Kendall’s τ.Source: §4 · p. 4, Table 6 · p. 7

Mementos by source

• Table 7 reports, by source, URI-M counts, covered URI-R counts, and mean URI-Ms per URI-R with standard deviations.The supplied passage describes the table’s measures but gives no values.Source: §4 · pp. 4, 7

5.1 Internet Archive

• The Internet Archive had the best coverage and depth, spanning URIs from 1996 to the present and including dead URIs with 404 responses.Less than 0.1% of its archived versions appeared after 2008, suggesting its 6–24-month interface delay may be longer.Source: §5.1 · p. 7

5.2 Search engine caches

• Search engines provide good coverage but only one cached copy per URI; Google and Bing caches last at most one month.Yahoo’s cached versions have no date, and the authors estimated their age, which may be several years.Source: §5.2 · p. 7

5.3 Other archives

• Other archives focus on particular countries, subscriber collections, or user preferences, and their copy ages depend on the archive’s age.The passage names UK Web Archive, The National Archives, NARA, Archive-It, ArchiefWeb, CDLIB, WebCite, and Diigo.Source: §5.3 · p. 7

5.4 Archive overlap

• The overlap visualization encodes coverage by circle diameter, depth by color density, and archive intersections by edge width.The analysis compares sources across coverage, depth, and age.Source: §5 · p. 7, §5.4 · p. 8

6. CONCLUSION AND FUTURE WORK

• DMOZ and Delicious URIs had much higher archival rates than search-engine URIs (about 2/3 archived) and Bitly URIs (just under 1/3).The authors conclude that URI source significantly drives archival status.Source: §6 · p. 8
• Active submission characterizes DMOZ and Delicious, while search-engine discovery depends on crawl heuristics; the authors suggest publicity interest may drive archival rates.They did not investigate Bitly URI use, but speculate its low archival rate may reflect private, stand-alone, or temporary resources.Source: §6 · p. 8
• Future work will examine how page-change rates relate to archiving rates, assess archive quality, and study archived URIs by language beyond English.The current work used a general sample of URIs.Source: §6 · p. 8