Source-linked AI summary
Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi
TL;DR
Human-aligned privacy evaluation with frontier LLMs is costly and difficult to deploy safely on sensitive text. The paper distills Mistral Large 3 judgments into lightweight encoder classifiers and finds strong agreement with human annotations, with practical utility for de-identification evaluation.
Problem
Privacy evaluation needs to reflect human-perceived sensitivity, but frontier LLM evaluators are costly and sensitive texts may not be shareable with third-party APIs.
Method
The paper distills Mistral Large 3 privacy judgments into lightweight encoder classifiers trained on a large privacy-annotated text corpus.
Results
α = 0.737 for the distilled model’s human alignment surpasses the teacher model’s α = 0.716, while the models support efficient local inference.
Takeaways & Limitations
Distilled privacy evaluators can serve as scalable automatic metrics for quantifying privacy reduction in text de-identification systems.
Takeaways & Limitations
The models inherit the teacher LLM’s privacy notion and biases, use an English-only dataset, and evaluate texts largely without audience, purpose, or setting.
Abstract
from arXiv · showhide
Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.
1. Introduction
Human-aligned privacy evaluation is promising but frontier LLMs are costly and can create privacy risks when sensitive text is sent to external APIs. This work distills LLM privacy judgments into lightweight local models and applies them to de-identification assessment.
- LLMs can approximate aggregated human privacy judgments, motivating their use as human-aligned evaluators.
- Frontier LLM deployment is limited by computational and financial costs, while third-party processing can expose the sensitive text being evaluated.
- The paper distills Mistral Large 3 judgments on 200,000 user-written texts into lightweight encoder classifiers for fast, local, privacy-preserving inference.
- The central question is whether LLM privacy reasoning can transfer to smaller models without sacrificing alignment with human judgments.
- The distilled evaluators are validated against human annotations and demonstrated as scalable metrics for quantifying privacy reduction in de-identification systems.
- α = 0.737 for the distilled model’s human alignment exceeds the teacher model’s α = 0.716 while enabling efficient, fully local inference.
2. Related Work
Prior privacy-preserving NLP evaluation commonly uses threat-model-specific proxies, while knowledge distillation offers a route to transfer large-model capabilities into efficient smaller models.
- Privacy-preserving NLP commonly evaluates systems with re-identification rates, simulated attacks, plausible deniability, or semantic similarity measures.
- These proxy metrics capture specific threat models but do not directly represent humans’ perceived sensitivity of text.
- Knowledge distillation transfers capabilities from large teacher models to smaller student models, including lightweight transformer classifiers.
- Label-only distillation supports black-box knowledge transfer when teacher logits are unavailable.
3. Methodology
The methodology constructs a large English corpus with five-level privacy labels generated by Mistral Large 3, then distills those judgments into encoder-based classifiers across diverse domains.
- Privacy Annotation Framework: The five-point Likert scale ranges from 1 (Harmless) to 5 (Extremely private) and incorporates identifiers, contextual signals, self-disclosure, and re-identification cues.
- Data: The corpus contains approximately 200,000 English texts sampled evenly from 10 publicly available datasets spanning diverse domains.
- Teacher Model Annotation: Mistral Large 3 assigns discrete privacy sensitivity scores using structured prompts with explicit scale definitions.
- Dataset Distribution: 46% of texts are rated harmless, whereas the most sensitive category comprises about 6% of samples.
- Dataset Distribution: Health and confession-oriented domains contain the highest proportions of private content, while review and microblog domains have less than 6% rated somewhat private or above.
- Student Models: Four encoder classifiers are fine-tuned for five-class classification using a shared training recipe and selected by validation macro F1.
4. Experiments
The experiments test whether distilled encoders learn the privacy classification task and preserve alignment with human judgments, then assess their use for de-identification evaluation. The results show strong held-out classification, human-level agreement, and sensitivity to identifier masking.
- Learning the Distilled Task: 74.9% accuracy and 68.1 macro F1 substantially outperform majority and random baselines on the held-out 5-class privacy test set.Ettin-150M also reaches 68.6 F1 on the most sensitive class, C5.
- Alignment with Human Privacy Judgments: Krippendorff’s α reaches 0.737 against average human ratings, exceeding Mistral Large 3’s α = 0.716 on the 250-text benchmark.The benchmark contains 677 human annotations across 250 texts.
- Alignment with Human Privacy Judgments: α = 0.514 (±0.265) against individual annotators closely matches the inter-human pairwise average of α = 0.54.This indicates model–individual disagreements are comparable in magnitude to disagreements among humans.
- De-Identification Evaluation: Masking direct identifiers produces a larger per-entity privacy reduction than masking quasi-identifiers, with ∆ = 0.34 versus ∆ = 0.23.The direct-identifier effect occurs despite 1,612 direct identifiers versus 19,197 quasi-identifiers.
5. Discussion & Future Work
The discussion presents distilled privacy evaluators as practical tools for scalable, on-device assessment and identifies future research directions. It also emphasizes unresolved requirements around contextualization, robustness, calibration, and deployment safety.
- Use Cases: On-device privacy classifiers can support corpus curation, privacy-aware comparison of de-identification systems, and real-time user-facing privacy warnings.These workflows include routing high-risk examples for review, filtering, or access control before training.
- Future Work: Compact evaluators enable large-scale attribution and counterfactual studies separating identifiability cues from topical sensitivity.The examples include names, locations, unique events, health, legal issues, and mental health.
- Future Work: Combining privacy scores with utility measures could support privacy–utility optimization for finding minimal changes that reduce sensitivity.The passage gives semantic similarity as an example utility measure.
- Future Work: Future work should incorporate audience, purpose, and setting, and explore personalization using small amounts of user preference data.This extends beyond a single global notion of privacy.
- Future Work: Deployment requires calibrating scores, handling out-of-domain inputs, and auditing domain- and demographic-dependent failure modes.The passage identifies robustness as open before integration into automated pipelines.
6. Conclusion
The paper presents knowledge distillation as a way to create efficient privacy sensitivity classifiers from LLM judgments, preserving agreement with human annotations while enabling faster, private inference. Evaluation on the Text Anonymization Benchmark supports their utility for assessing de-identification.
- Knowledge distillation converts LLM privacy assessments into efficient privacy sensitivity classifiers.
- A 150M-parameter Ettin encoder achieves strong agreement with human annotations while enabling private and faster inference.
- Evaluation on the Text Anonymization Benchmark shows meaningful differences between direct and quasi-identifiers in expert-annotated documents.
- The authors release code, models, and dataset to support reproducible privacy evaluation in NLP.
Limitations
The classifiers inherit limitations from the teacher model and privacy-sensitivity formulation. Their scope is also constrained by English-only training and isolated text evaluation without explicit context.
- The models inherit the teacher LLM’s privacy notion and potential biases.
- A single 1–5 sensitivity score may conflate identifiability with topic sensitivity.
- Training data is English-only, leaving multilingual transfer untested.
- The classifier evaluates texts largely in isolation without explicit audience, purpose, or setting information.
- Teacher labeling can be stochastic, and alternative teacher models or distillation strategies were not systematically studied.Multiple teachers or a small amount of human-labeled calibration data could reduce noise and improve robustness.
- The score captures perceived sensitivity under the adopted scale rather than a formal privacy guarantee or adversarial re-identification risk.
Ethical Considerations
The work processes potentially sensitive user-generated content and positions the classifier as a research evaluation tool rather than a system for individual decisions or surveillance. Human oversight remains advised.
- The source datasets are publicly available and have previously been used in research.
- The classifier is intended to evaluate privacy-preserving methods and support privacy research, not make decisions about individuals or enable surveillance.
- The authors caution against using model scores as a hard gate without human oversight.The privacy scale is subjective, so scores should inform rather than replace human judgment.
A. Dataset Sources Description
The privacy corpus aggregates user-written texts from ten publicly available datasets spanning varied genres, platforms, and disclosure styles. This diversity covers explicit identifiers, contextual clues, health and legal sensitivity, and stylometric linkability.
- The corpus combines ten publicly available datasets covering blogs, emails, health forums, legal advice, reviews, microblogs, and online discussions.
- Blog Authorship Corpus contributes long-form personal storytelling and stylistic signatures relevant to linkability-based privacy risks.
- Enron Emails add semi-private organizational communication with operational details, social relations, contact information, and workplace context.
- Medical Questions contribute health-related texts containing sensitive attributes such as symptoms, medications, ages, and conditions.
- Mental Health Blog posts represent long-form self-disclosure and peer support, with privacy signals driven by sensitive topics and personal experiences.
- Reddit Confessions broaden coverage to anonymous, confession-oriented self-disclosure rooted in intimate content.
- Reddit Legal Advice contributes sensitive situational details and contextual clues involving disputes, employment, alleged wrongdoing, locations, and timelines.
- Reddit Mental Health Posts add shorter, conversational, fragmented expressions of privacy-sensitive themes.
B. Teacher Prompt
The teacher prompt enforces a discrete 1–5 privacy scale for rating how sensitive or private user text is. It distinguishes harmless text from text containing highly sensitive or private information.
- The prompt requires privacy ratings on a discrete 1–5 scale.
- Ratings assess whether user text contains personal, sensitive, or indirectly identifiable information.
- Score 1 denotes text completely free of private or sensitive information, including direct and indirect identifiers.
- Scores 2–4 represent progressively greater privacy, from mostly non-private text to text containing several direct or indirect identifiers.
- Score 5 denotes extremely private text containing highly sensitive information, such as direct personal identifiers, that should not be made public.