Privacy at Scale: Introducing the PrivaSeer Corpus of Web Privacy Policies

Introduction

• Privacy policies are legally required, yet most users fail to understand them; reading all policies an average person encounters yearly is estimated to take about 200 hours.The passage cites Madden (2017), Meiselwitz (2013), Obar and Oeldorf-Hirsch (2018), and McDonald and Cranor (2008) for these concerns and estimates.Source: §1 · p. 1
• Existing privacy-policy NLP research relies on expert-annotated corpora of hundreds or thousands of policies, while annotation is costly and scalability and generalization remain concerns.Tasks requiring skilled legal experts include practice classification, question answering, vague-sentence detection, and compliance-issue detection.Source: §1 · p. 1
• 1,005,380 English-language website privacy policies make up PrivaSeer, representing about ten times more unique websites than the next-largest public collection.The corpus also exceeds the aggregate of unique websites in all other publicly available web privacy policy corpora combined.Source: §1 · p. 2
• PrivaSeer’s creation pipeline combines web crawling, language detection, document classification, duplicate and near-duplicate removal, and content extraction.Source: §1 · p. 2
• The paper analyses policy lengths and top-level distribution and explores corpus topics through topic modeling.Source: §1 · p. 2
• The authors pretrain PrivBERT on PrivaSeer and evaluate it on data-practice classification and question-answering tasks, releasing the corpus, search engine, collection pipeline, and language model.The motivation is that in-domain fine-tuning of pretrained language models has improved performance across domains (Gururangan et al., 2020).Source: §1 · p. 2

2 Related Work

• OPP-115 and PrivacyQA supported research on privacy-policy classification, practice extraction, compliance, question answering, and opt-out choices (Harkous et al., 2018; Ravichander et al., 2019).OPP-115 includes 115 policies with manual annotations of 23k fine-grained data practices; PrivacyQA contains 1,750 questions with expert-annotated answers.Source: §2 · p. 2
• Other datasets examined vague language and opt-outs or collected policy URLs and policies from app stores, 130,000 websites, and DMOZ domains (Lebanoff and Liu, 2018; Sathyendra et al., 2017; Zimmeck et al., 2019; Amos et al., 2020; Nokhbeh Zaeem and Barber, 2021).The collected sets included over 400k Android app policy URLs and around 100k policies from DMOZ domains.Source: §2 · p. 2
• Although most internet users are concerned about privacy, a significant number do not read notices because they seem time-consuming or complicated, and concise website privacy information remains an open problem.Source: §2 · p. 2

Document Collection

• The collection pipeline focused on privacy-policy URLs from Common Crawl, filtering likely links before crawling them to obtain candidate documents.Privacy-policy links are commonly placed in website footers under labels such as “Privacy Policy,” “Privacy Notice,” and “Data Protection.”Source: §3 · pp. 2–3
• The researchers selected 3.9 million URLs containing “privacy” or both “data” and “protection” from the May 2019 Common Crawl archive.The archive comprised 2.65 billion web pages, or 220 TB of uncompressed content, crawled May 19–27, 2019.Source: §3 · p. 3
• The URL-selection technique was estimated to cover 92.17% ± 6.51% of English privacy policies on the web, with a 95% confidence interval.The estimate came from manually examining 115 English-language websites in the OPP-115 Corpus; 4 lacked a policy link on or one hop from the landing page, and 5 failed the URL-selection criteria.Source: §3 · p. 3
• Using Scrapy, they crawled the 3.9 million selected URLs and obtained 3.2 million candidate privacy policies; 0.4 million produced errors and 0.3 million were discarded as duplicates.Crawling took about 48 hours across a few hours each day between August 4 and 10, 2019.Source: §3 · p. 3

Language Detection

• Langid classified candidate documents into 97 languages plus an unknown category; non-English documents were set aside, leaving 2.1 million candidates.The authors focused on English to enable comparison with prior privacy-policy corpora.Source: §4 · p. 3

Content Extraction

• Boilerpipe was selected after manual tool comparison to strip headers, footers, navigation menus, banners, and other boilerplate from English pages.It uses shallow text, structural, and density-based features.Source: §4 · pp. 3–4

Document Classification

• The team labeled 1,600 candidates and trained three random-forest models using URL, document, or combined features, plus a fine-tuned RoBERTa classifier.RoBERTa used only the first 512 tokens of each document; URL and document models used tf-idf features, and SMOTE addressed class imbalance.Source: §4 · p. 4
• The best-performing transformer-based model, evaluated by averaged held-out test results after repeated sampling and training with 5-fold cross-validation, classified 1.54 million of 2.1 million English candidate policies as privacy policies.Source: §4 · p. 4

URL Cross Verification

• The team cross-verified policy URLs against links found by crawling domain homepages and pages one hop away, leaving 1.1 million web pages.The landing-page crawl took place between 8 and 10 November 2019.Source: §4 · p. 4

Duplicate and Near-Duplicate Detection

• After removing exact duplicates, the team used 64-bit Simhashes of three-word shingles and Hamming-distance thresholds to remove near-duplicates only within the same domain, retaining longer policies.This preserved similar policies across different domains, which might reflect shared templates, while removing same-domain versions that might be archived policies.Source: §4 · p. 5

Corpus Composition

• 1,005,380 policies from 995,475 domains make up the corpus, with policies averaging about 1,871 words and ranging from 143 to 16,980 words.The corpus spans over 800 top-level domains; the authors note that popular TLD shares and Common Crawl representativeness statistics support treating it as a representative sample of the web.Source: §5 · p. 5

Readability

• Using the widely used Flesh-Kincaid Grade Level (FKG) metric to assess corpus policy readability and enable comparison with prior literature, the policies had a mean score of 14.87 on the U.S. grade-level scale.Source: §5 · p. 5

Topic Modelling

• The topic analysis used LDA on policy paragraphs, lemmatized after regex tokenization, and selected nine topics because larger topic counts yielded less coherent clusters.The authors mapped topics to OPP-115 categories, applying a top-down approach rather than the category scheme’s bottom-up expert analysis.Source: §5 · pp. 5–6
• LDA topics split first-party collection into purpose and information type versus collection method, and third-party sharing into collection action versus purpose and effects.Cookies formed a separate topic; privacy contact information, data security, and policy change also appeared as separate topics, while the international-audiences topic primarily concerned Europe and GDPR. The authors suggest divergence from OPP-115 categories reflects differences between expert expectations and themes found in the corpus.Source: §5 · p. 6
• About 77% of policies discuss third parties, 55% mention GDPR or European audiences, and 38% contain language about advertising and analytics.Data security, policy change, and contact information also occur in a majority of policies.Source: §5 · p. 6, Figure 2 · p. 6
• More popular domains tend to have policies addressing more topics, and their policies are also longer and slightly more difficult to read.The authors suggest this may reflect larger, more diverse audiences and greater regulatory scrutiny.Source: §5 · pp. 6–7, Figure 3 · p. 7

• PrivBERT starts from RoBERTaBASE because privacy policies’ legalese differs from general-purpose pretraining corpora, and it is pretrained on privacy policies.Training used dynamic masked language modeling for 50k steps with batch size 512 and peak learning rate 8e-5.Source: §6 · p. 7
• For data-practice classification, PrivBERT was fine-tuned on OPP-115’s expert-annotated policy segments as a multi-label task across 12 coarse-grained categories.The corpus was split 3:1:1 for training, validation, and testing; the tuned model used dropout 0.15 and learning rate 2.5e-5.Source: §6 · p. 7
• About 4% improvement in macro-average F1 over RoBERTa, with PrivBERT also improving F1 for every data-practice category.The comparison is on the data-practice classification task.Source: §6 · p. 7, Table 2 · p. 8
• For answer-sentence selection, PrivBERT was fine-tuned on PrivacyQA question-sentence pairs to classify each sentence as evidence or non-evidence.PrivacyQA has 1,750 questions from 35 privacy documents; training used dropout 0.2, learning rate 3e-6, and an 8:1 positive-to-negative class weighting.Source: §6 · pp. 7–8
• About 6% improvement over BERT on answer-sentence selection, measured by sentence-level F1.BERT results were reported by Ravichander et al. (2019).Source: Table 3 · p. 8, §6 · p. 8
• The results across these two disparate privacy-domain tasks suggest PrivBERT could improve performance on real-world privacy applications.Source: §6 · p. 8

7 Conclusion

• PrivaSeer is the first large-scale corpus of contemporary website privacy policies, containing just over 1 million documents, built with a pipeline spanning crawling through near-duplicate removal.The pipeline also included language detection, document classification, duplicate removal, document cross verification, and content extraction.Source: §7 · p. 8
• Topic modelling found privacy-practice themes that matched legal experts’ expectations in some ways but differed in others.The authors also found a positive relationship between domain PageRank and the number of policy topics; they hypothesize that popular domains’ larger, more diverse user bases prompt coverage of laws across jurisdictions.Source: §7 · pp. 8–9
• 14.87 years on average, measured with the Flesch-Kincaid readability metric, was required to understand a privacy policy—roughly two years of U.S. college education.This large-scale finding was consistent with prior research that privacy policies are generally difficult for the average internet user to understand.Source: §7 · p. 9
• PrivBERT, a privacy-domain language model pretrained from RoBERTa, achieved state-of-the-art results on data practice classification and question answering.The authors expect PrivBERT to improve privacy-domain tasks and support stable, reliable privacy-preserving technology.Source: §7 · p. 9
• The authors expect PrivaSeer to advance automated extraction of salient privacy-policy details, benefiting internet users, regulators, and researchers.They also expect PrivBERT to help improve privacy-domain tasks and support stable and reliable privacy-preserving technology.Source: §7 · p. 9