Source-linked AI summary
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, Norman Sadeh
TL;DR
Privacy policies are difficult to read and interpret, while users need to identify privacy issues relevant to them. The paper introduces PrivacyQA, evaluates neural baselines, and finds substantial room for improvement in answering privacy-policy questions.
Problem
Privacy policies are difficult to read and involve nuanced, individual data-practice trade-offs, motivating tools that help users identify issues relevant to them.
Method
The paper introduces PrivacyQA, a corpus of 1750 privacy-policy questions paired with over 3500 expert annotations, and evaluates question-answering baselines.
Results
39.8 F1 is achieved by the best-performing BERT + UNANSWERABLE baseline on answer sentence selection, leaving considerable room to reach human performance.
Takeaways & Limitations
PrivacyQA supports research on privacy-domain question answering and exposes answerability challenges involving user intent, policy content, and evidence selection.
Takeaways & Limitations
Privacy policies require careful expert understanding because they are legal documents.
Abstract
from arXiv · showhide
Privacy policies are long and complex documents that are difficult for users to read and understand, and yet, they have legal effects on how user data is collected, managed and used. Ideally, we would like to empower users to inform themselves about issues that matter to them, and enable them to selectively explore those issues. We present PrivacyQA, a corpus consisting of 1750 questions about the privacy policies of mobile applications, and over 3500 expert annotations of relevant answers. We observe that a strong neural baseline underperforms human performance by almost 0.3 F1 on PrivacyQA, suggesting considerable room for improvement for future systems. Further, we use this dataset to shed light on challenges to question answerability, with domain-general implications for any question answering system. The PrivacyQA corpus offers a challenging corpus for question answering, with genuine real-world utility.
1 Introduction
Privacy policies are legally consequential but rarely read in practice, while users’ data compromises are highly individual. PrivacyQA addresses this gap with a corpus designed to support personalized question answering about privacy policies.
- 1 Introduction: Privacy policies disclose how companies gather, use, share, and manage user data under a notice-and-choice framework.The framework is accepted in many legal jurisdictions, including the United States and European Union.
- 1 Introduction: Figure 1 illustrates privacy questions from Keep and Viber alongside policy sentences relevant to determining their answers.These examples connect user queries with supporting policy evidence.
- 1 Introduction: Users seldom read privacy policies, and their nuanced data compromises discourage one-size-fits-all privacy notices.This mismatch makes selective exploration of personally salient privacy issues important.
- 1 Introduction: Increasingly visible data misuse scandals have heightened public privacy concerns and motivated NLP research in this domain.The authors connect this motivation to helping users quickly identify privacy issues most salient to them.
- 1 Introduction: PRIVACYQA contains 1750 questions about privacy policies paired with over 3500 expert annotations of relevant answers.The corpus is intended to advance question-answering methods and expose understudied answerability challenges.
2 Related Work
Prior work has sought to make privacy policies easier to understand through prescriptive communication and NLP-based extraction of data practices. PrivacyQA instead supports personalized querying, with a larger, expert-annotated corpus covering diverse question types.
- 2 Related Work: Earlier approaches aimed to make privacy information easier to understand, but prescriptive communication methods were not widely adopted by industry.Recent NLP work instead focuses especially on identifying specific data practices in privacy policies.
- 2 Related Work: PrivacyQA adopts a personalized approach that lets users query a policy and selectively explore content salient to them.This differs from approaches centered only on general policy presentation or practice extraction.
- 2 Related Work: PrivacyQA policies are in English and are freely available as a research resource.The corpus is distributed through the project’s public repository.
- 2 Related Work: Compared with PolisisQA, PRIVACYQA contains 10x as many questions and answers, uses legally trained domain experts, and includes unanswerable and subjective questions.These design choices target privacy-policy question answering with broader coverage and expert interpretation.
- 2 Related Work: Privacy-policy question answering lies between open-domain reading comprehension and legal-document NLP because policies have legal implications but address the general public.The authors note that policy writers and audiences may have different goals.
3 Data Collection
PRIVACYQA was constructed from diverse mobile-app privacy policies by eliciting questions without showing workers the policies, then obtaining legally trained expert annotations. The corpus contains 1,750 questions across 35 applications, with questions spanning privacy categories and a held-out test split.
- Data collection: The corpus collects privacy policies from 35 mobile applications across multiple Google Play Store categories.Applications were selected to include both well-known and lesser-known apps, using a 5-million-install threshold within categories.
- Crowdsourced Question Elicitation: Crowdworkers asked questions using app names, descriptions, and navigable screenshots rather than seeing the privacy policies.This design aimed to avoid inadvertent biases and encouraged questions beyond answers suggested by policy text.
- Answer validation: Seven legally trained experts identified relevant policy evidence and annotated question relevance, subjectivity, category, and answer likelihood.Expert annotations were intended to provide legally sound answers to questions posed by crowdworkers.
- Dataset statistics: The dataset uses 1,350 training questions from 27 applications and 400 test questions from 8 policy documents.Privacy policies average about 3,000 words, while answers typically identify about 100 words of evidence.
- Answer validation: Experts agreed on answers for 74% of questions, while disagreement on 26% reflected differing intent, evidence selection, policy interpretation, partial answers, and other legitimate differences.The largest identified source was differing interpretations of question intent at 11%.
4 Experimental Setup
The experiments frame privacy-policy QA as answerability prediction and extractive evidence-sentence selection. Baselines include feature-based, neural, lexical, no-answer, and human systems, with answerability information incorporated into one BERT evidence model.
- 4.1 Answerability Identification Baselines: Answerability identification predicts whether a question can be answered from the question alone, providing a prior for downstream QA.Three baselines substantially outperform a majority-class baseline.
- Results: BERT performs best on binary answerability identification, and most baselines considerably exceed the majority-class baseline.The results indicate that question text contains information about answerability in this domain.
- Baselines: Baselines include SVM feature sets, a CNN encoder, BERT answerability classification, lexical word-count retrieval, no-answer prediction, and human performance.The SVM variants use question bag-of-words, length, and part-of-speech features; BERT also has a two-stage answerability-plus-evidence variant.
- 4.2 Evidence Identification: The answer-sentence selection task identifies evidence sentences from all candidate sentences in each privacy policy.Performance is evaluated with sentence-level F1 based on overlap between predicted and gold-reference sentences.
- Evidence Identification: The BERT + UNANSWERABLE system first predicts answerability and then selects evidence sentences only when the question is predicted answerable.This tests whether answerability information improves evidence identification.
- Evaluation: The evaluation includes a human baseline computed by comparing each annotator reference answer with the remaining references.The reported human score averages the maximum F1 across reference answers.
5 Results and Discussion
PrivacyQA results show that answerability and answer selection remain difficult, with errors concentrated in answerable questions and several distinct forms of unanswerability. These findings indicate that effective systems must distinguish among question interpretation, policy scope, and policy silence.
- Answer Selection: 39.8 F1 was achieved by BERT + UNANSWERABLE on answer sentence selection, leaving considerable headroom toward human performance.The No-answer baseline reached 28 F1, and incorporating answerability information improved performance in this difficult domain.
- Answerability Errors: Most BERT answerability mistakes were answerable questions incorrectly identified as unanswerable.Experts found that most such questions were relevant, although many were subjective; only 6 were unexpected or usually unanswered in privacy policies.
- Unanswerability Factors: 573 questions were analyzed for unanswerability, revealing incomprehensibility, irrelevance, ill-formedness, silence, and atypicality as distinct factors.The analysis treats unanswerable questions as heterogeneous rather than as one uniform class.
- Unanswerability Factors: 50% of comprehensible questions were likely answerable from the policy, while 33.1% were privacy-related but outside policy scope and 16.9% were completely out-of-scope.Among relevant, well-formed questions, 95.7% were not answered by the policy despite it being reasonable to expect an answer.
- Unanswerability Factors: 95.7% of relevant, well-formed questions were not answered by the policy, although an answer was reasonably expected.The remaining 4.3% were reasonable questions of a kind generally not discussed in privacy policies.
- Broader Implications: Unanticipated and unanswerable questions should be expected in question-answering systems interacting with real users.Related analysis of natural search queries also found varied forms of unanswerability, including atypical and incomplete questions.
6 Conclusion
PRIVACYQA introduces a substantial expert-annotated corpus for privacy-policy question answering and exposes both low baseline performance and complex answerability challenges. The authors position the resource as a basis for multidisciplinary work on matching user intent with policy information.
- Results: Strong neural baselines achieved only 39.8 F1 on PRIVACYQA, indicating considerable room for future research.The paper also identifies several considerations that affect question answerability.
- Implications: The paper calls for multidisciplinary efforts to understand user intent and reconcile it with information in policy documents.This proposed direction spans both privacy and NLP communities.