Source-linked AI summary
Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset
Ryan Amos, Gunes Acar, Eli Lucherini, Mihir Kshirsagar, Arvind Narayanan, Jonathan Mayer
TL;DR
Earlier privacy-policy analysis lacked a large-scale curated dataset spanning time, despite policies being a lens on businesses’ handling of personal information. The authors curated archived policies and analyzed their trends, finding substantial underreporting of tracking and third parties as well as longer, harder-to-read policies.
Problem
Privacy policies offer a lens on how businesses interact with personal information, but the paper’s longitudinal analysis addresses policy trends across a large scale and span of time.
Source: §1 · p. 1Method
The authors built a crawler to collect archived policies and used automated trend detection to analyze changes across a curated dataset spanning over two decades.
Source: §1 · p. 1, Privacy policy updates · p. 8Results
Privacy policies severely underreport third parties and tracking technologies, have become longer and harder to read, and show the GDPR’s monumental impact.
Source: §9 · p. 10Takeaways & Limitations
The findings add evidence about the inadequacy of notice and choice and demonstrate the GDPR’s impact on privacy policies.
Source: §9 · p. 10Handcrafted regular expressions may capture unintended terms, include terms outside the targeted concept, or miss terms that fit it.
Source: §9 · p. 11
Abstract
from arXiv · showhide
Automated analysis of privacy policies has proved a fruitful research direction, with developments such as automated policy summarization, question answering systems, and compliance detection. Prior research has been limited to analysis of privacy policies from a single point in time or from short spans of time, as researchers did not have access to a large-scale, longitudinal, curated dataset. To address this gap, we developed a crawler that discovers, downloads, and extracts archived privacy policies from the Internet Archive's Wayback Machine. Using the crawler and following a series of validation and quality control steps, we curated a dataset of 1,071,488 English language privacy policies, spanning over two decades and over 130,000 distinct websites. Our analyses of the data paint a troubling picture of the transparency and accessibility of privacy policies. By comparing the occurrence of tracking-related terminology in our dataset to prior web privacy measurements, we find that privacy policies have consistently failed to disclose the presence of common tracking technologies and third parties. We also find that over the last twenty years privacy policies have become even more difficult to read, doubling in length and increasing a full grade in the median reading level. Our data indicate that self-regulation for first-party websites has stagnated, while self-regulation for third parties has increased but is dominated by online advertising trade associations. Finally, we contribute to the literature on privacy regulation by demonstrating the historic impact of the GDPR on privacy policies.
1 INTRODUCTION
The paper builds a large longitudinal privacy-policy dataset and uses it to examine disclosure, readability, self-regulation, and regulatory change. Its findings include persistent underreporting of tracking technologies and third parties, alongside widespread policy changes prompted by the GDPR.
Dataset
- A crawler and validation process produced a dataset of over one million archived privacy policies from more than 130,000 websites, spanning over two decades.The classifier used to filter non-policies achieved 98% precision and 93% recall.Source: §1 · p. 1
Longitudinal analysis
- The dataset and automated trend detection tool support longitudinal analysis of privacy policies and trends in regulation, self-regulation, third parties, and tracking technologies.The authors describe the analysis as, to their knowledge, the longest-spanning and largest-scale longitudinal analysis of privacy policies.Source: §1 · p. 1
Accessibility
- More than 20% of websites have a privacy policy that links to one or more additional policies, increasing the burden of understanding applicable policies.Source: §1 · p. 2
Longitudinal analysis
- Compared with empirical measurements of web privacy practices, privacy policies vastly underreport tracking technologies and third parties.The authors say these findings undermine the idea that users can reasonably make informed decisions from policy disclosures.Source: §1 · p. 2
Regulatory change
- The GDPR prompted the most widespread changes to privacy policies in the last decade.Source: §1 · p. 2
Self-regulation
- The paper examines which self-regulatory bodies have grown and which have stagnated, and aims to help research on the connections among regulation, self-regulation, and user privacy.The authors say evidence about past actions can help drive evidence-based methods for creating effective privacy-policy legislation.Source: §1 · p. 2
2 LEGAL BACKGROUND
The legal background contrasts the U.S. notice-and-choice approach with Europe’s more direct data-protection model. It also traces U.S. laws that required privacy policies or disclosures and the GDPR’s specific policy-disclosure requirements.
Privacy policies in the United States: notice and choice
- The U.S. notice-and-choice model relied mainly on voluntary company disclosures, with the FTC policing their accuracy and enforcement intended to guide private-sector practices.The policy rested on consumer awareness and choice, with regulation in the background if the model failed.Source: §2 · p. 2
Children’s Online Privacy Protection Act
- COPPA regulated information collection and use on websites directed at children under 13, and its passage and the threat of further regulation led many popular websites to adopt privacy policies.Source: §2 · p. 2
California Online Privacy Protection Act
- California’s CalOPPA required commercial websites to post privacy policies and quickly became a de facto national requirement after taking effect in 2004.The California Consumer Privacy Act and California Privacy Rights Act guarantee consumers specific privacy rights and represent a break from notice and choice.Source: §2 · p. 2
Privacy policies in the European Union: data protection
- Europe’s approach specifies what businesses must disclose and guarantees individual privacy rights, while the GDPR requires specific disclosures about data processing and individuals’ rights.Online services typically provide these disclosures in privacy policies.Source: §2 · p. 2
3 RELATED WORK
Prior privacy-policy research spans marketplace and longitudinal studies, comprehension, machine learning, compliance, and dataset development. This paper positions its contribution as a larger-scale, longer-span longitudinal analysis and dataset that supports future research on changes over time.
Marketplace and longitudinal studies
- Earlier marketplace and longitudinal studies documented policy adoption, content, readability, and length trends, while more recent GDPR analyses examined changes around 2016–2019.The cited GDPR studies reported changes in adoption, GDPR-related phrases, update practices, policy length, and disclosure specificity.Source: §3 · pp. 2–3
- This work extends longitudinal analysis across over two decades and more than 130,000 websites, characterizing document-level and popularity-tier trends and shifts associated with the GDPR.It also provides the first longitudinal analysis of specific trends, according to the authors.Source: §3 · p. 3
Research datasets
Compliance checking
- The paper contributes to compliance research by evaluating representations of regulatory and self-regulatory programs over time and comparing policy disclosures about advanced tracking with known technology adoption.Source: §3 · p. 3
Consumer comprehension
- Prior comprehension research found that consumers inconsistently read policies, have limited comprehension, and struggle to exercise choices; this paper tracks policy length and readability over two decades and across website ranking tiers.Source: §3 · p. 3
Applied machine learning
- Prior machine-learning work grades policies, identifies choices, and supports queries about policy attributes; this paper relies on heuristics and offers its dataset for future longitudinal machine-learning research.Source: §3 · p. 3
4 COLLECTING HISTORICAL PRIVACY POLICIES
The section describes collecting archived privacy policies over time and evaluating how well the collection process worked.
4.1 Building the list of target websites
- The researchers combined domains appearing in Alexa’s top 100K lists from 2009–2019, yielding 541,616 websites across 22 half-year lists.Each interval was a six-month period, with one interval in January–June and one in July–December.Source: §4.1 · pp. 3–4
4.2 Building the list of snapshots
- For each website and six-month interval, the crawler selected the available Wayback Machine homepage snapshot closest to the interval midpoint, including snapshots outside the website’s ranking years.The archive queries were not restricted to the period when a website appeared in Alexa’s top 100K.Source: §4.2 · p. 4
4.3 Downloading privacy policies
- The crawler screened English homepages, matched policy-link text, fetched same-interval archived pages, rechecked language, and extracted readable policy text while blocking requests to live websites.The link-detection method favored comprehensive matching, so non-policy pages could be filtered after the crawl.Source: §4.3 · p. 4
4.4 Practical and ethical considerations
- Researchers notified the Internet Archive, limited bandwidth and parallel workers, paused on server errors, and monitored server load to reduce crawl impact.The crawl disabled image downloads and used up to 256 parallel workers.Source: §4.4 · pp. 4–5
4.5 Evaluating data quality
- Only about 2% of missing policies were attributable to crawler limitations; absent policy links were the most common failure cause.The crawler downloaded 1,292,420 policies from 5,223,228 homepage snapshots, a 24% success rate.Source: §4.5 · p. 5
5 FILTERING THE CORPUS
The authors defined which documents counted as apparent privacy policies, then trained and evaluated a classifier to filter the crawled corpus.
Labeling definition
- The labeling target was an apparent privacy policy, not legal compliance; documents had to concern privacy and be legal in nature.The authors considered determining whether documents satisfied jurisdiction-specific legal requirements outside the project’s scope.Source: §5 · p. 5
- The authors excluded incomplete, cookie-only, security-only, mixed, privacy-center, and summary documents from the policy labels.These categories were excluded because their contents were dissimilar or could pose problems for automated textual analysis.Source: §5 · p. 5
Labeling the sample
- Two researchers independently labeled 100 randomly collected documents using the stated criteria, achieving inter-annotator agreement κ=0.93; the final labeled dataset contained 1,173 positives (∼81.5%) and 266 negatives (∼18.5%).Source: §5 · p. 5
Model training and validation
- The authors selected a random forest over logistic regression based on mean AUC, prioritized precision when choosing its threshold, and used word n-gram features.Link-text and URL features were excluded because they did not improve classification performance.Source: §5 · pp. 5–6
Classifier evaluation
- 97.9% precision and 94.2% recall were achieved by the final classifier on 749 manually labeled held-out documents.Figure 2 presents predictive performance on the held-out documents.Source: §5 · p. 6, Figure 2 · p. 6
Document classification
- 17% (220,932) of the documents were classified as negatives, leaving a final dataset of 1,071,488 policies.The dataset totals are reported after document classification.Source: §5 · p. 6
6 CORPUS COMPOSITION
The corpus was cleaned to reduce bias and duplicates, then assessed for temporal coverage, category representation, and download rates across website popularity. These checks reveal gaps in archival coverage and an underrepresentation of uncategorized sites.
6.1 Cleaning the corpus
- 160,941 policies were removed, leaving 910,546 policies from 108,499 sites for analysis.The removals were intended to reduce potential analysis bias.Source: §6.1 · p. 6
- Cleaning removed 24,289 parking-provider snapshots, excluded cross-origin redirects except likely same-owner sites, and removed 61,220 duplicate policy URLs.The duplicate-removal method retained one site per shared policy URL and preferred a homepage whose domain matched the policy domain.Source: §6.1 · p. 6
6.2 Corpus overview
- Policy and homepage snapshot counts both fell in 2009B and 2018A, suggesting archiving changes or incomplete archives caused those dips.Source: §6.2 · p. 6
- Most policies came from sites ranked 100K and above, while policy-download success was much higher for the top 10K.The >1M rank bin also includes snapshots without an Alexa rank or captured before 2009.Source: Table 2 · p. 7, §6.2 · p. 7
- The category comparison covers present and missing sites; categories were assumed constant over time, and uncategorized sites were strongly underrepresented in the dataset.Websites in multiple categories count once in each category.Source: §6.2 · p. 7, Figure 4 · p. 7
7 DOCUMENT-LEVEL TRENDS
Privacy policies grew longer and harder to read, while updates and vocabulary shifts clustered around the GDPR’s introduction. The analyses associate that period with widespread policy changes across website popularity groups.
7.1 Comprehension
- Median policy length doubled from 876 words in 2009A to 1,522 in 2019B, with a sharper rise in recent years after the GDPR’s introduction.The increase held for both popular and less popular websites; most corpus policies date from 2009–2019, the period the analysis emphasizes.Source: Figure 5 · p. 7, §7.1 · p. 7, §7 · p. 7
- Median Flesch-Kincaid grade level rose from 11.9 in 2000A to 13.2 in 2019B, and more popular websites had less readable policies.Source: Figure 6 · p. 7, §7.1 · p. 7
7.2 Privacy policy updates
- In 2018, significant policy updates spiked across all rank buckets, although popular websites updated more frequently overall.Significant updates were defined by a fuzzy similarity ratio of 95% or less between consecutive snapshots.Source: Figure 7 · p. 8, Privacy policy updates · p. 8
- In 2018A, policies containing GDPR-related phrases averaged 136 new lines per update, versus 15 for policies without those phrases.The authors interpret this comparison as supporting a link between the GDPR’s introduction and the increase in updates.Source: Privacy policy updates · p. 8
- Update timing excludes snapshots with a gap in the prior interval because the exact update time could not be determined.Source: Privacy policy updates · p. 8
- N-gram frequency change points clustered around the GDPR’s introduction, tracking a similar pattern to document-level updates.The analysis counted change points for lemmatized 1-grams and 8-grams with document frequency of at least 0.01 in any interval.Source: Figure 8 · p. 8, Privacy policy updates · p. 8
8 TERM-LEVEL TRENDS
Automated trend surfacing guided analysis of self-regulation, tracking technologies, and third parties. The analysis found substantial gaps between policy disclosures and observed tracking, alongside growth in some third-party and advertising-industry references.
8.1 Automated trend surfacing
- The trend-surfacing tool ranked terms by scoring functions such as frequency gain, surfacing ambiguous phrases and growth in GDPR-related, Google Analytics, and Facebook terms.The pipeline replaced URLs, named entities, numbers, and emails with placeholders before extracting and ranking terms.Source: §8.1 · p. 8, §8 · p. 8
8.2 Trends
- Online advertising groups NAI, DAA, and EDAA gained mentions, while TrustArc and BBB references remained relatively stagnant.EDAA mentions more than doubled after GDPR’s introduction, and NAI mentions rose around four-fold in 10 years.Source: §8.2 · p. 9
- 94.6% of Alexa top 10K websites had observed web beacons, compared with 25.8% of 2019 policies mentioning related terms; fingerprinting terms were also underreported.Cookie mentions, by contrast, were close to prior web measurements.Source: §8.2 · p. 9
- Third parties appeared in policies far less often than in web measurements, though policy reporting increased and the reason—greater presence or transparency—is unclear.A gap between observation and disclosure suggests that policies may omit present third parties.Source: §8.2 · p. 10
- Manual labeling of 100 positive examples per matching category found that the matching criteria had high precision.For third parties and tracking technologies, positives were terms used in a tracking-related context; for self-regulatory initiatives, positives indicated a relationship with the initiative.Source: §8.2 · p. 10, Table 4 · p. 10
- 20.3% of websites linked to additional privacy policies, increasing the reading burden and potentially explaining why some technologies are absent from first-party policies.The linked policy may describe a technology instead of the first-party policy.Source: §8.2 · p. 10
9 CONCLUSION
The paper presents a curated, longitudinal dataset of privacy policies and uses it to examine changing policy practices. Its analyses find underreporting of tracking and third parties, while noting limits in the data and analysis methods.
9 CONCLUSION
- The curated dataset contains over one million privacy policies spanning more than two decades, and the analysis finds that policies are becoming longer and harder to read.The dataset and source code are publicly available.Source: §9 · p. 10
- Privacy policies severely underreport the use of third parties and tracking technologies, adding evidence against the adequacy of the “notice and choice” model.The authors also report a historic impact of GDPR on privacy policies.Source: §9 · p. 10
- A chronologically accurate GitHub repository and separate web interface let users search, compare policy versions, and explore the dataset.The repository uses archive timestamps and includes policy text, HTML sources, and metadata.Source: §9 · p. 10
- The study analyzes privacy policies, not how archived websites actually track users or share personal data with third parties.Source: §9 · p. 11
- For internationalized websites, the Wayback Machine crawler’s location could affect the language of an archived site and its policy, but the location of a particular capture is unknown.Source: §9 · p. 11
- Readability metrics do not account for document organization, formatting, length, or semantic difficulty.Source: §9 · p. 11
- Trend analysis relied on regular expressions that may match unintended terms or miss terms relevant to the targeted concepts.The authors note that this risk remains despite manual validation.Source: §9 · p. 11