Source-linked AI summary
PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization
Stephen Meisenbacher, Andreea-Elena Bodea, Ahmet Bilal Akın, Alexandra Klymenko, Jana Diesner, Florian Matthes
TL;DR
Text-to-text privatization lacks standardized, accessible evaluation that balances privacy and utility. PrivBench addresses this gap with a holistic, modular, real-time benchmarking platform grounded in a systematic review. Its case study provides a nuanced comparison of methods, while the platform remains adaptable to evolving evaluation strategies and acknowledges that larger datasets would yield more robust scores.
Problem
Text-to-text privatization evaluation lacks uniform approaches and accessible comparative mechanisms while needing to balance privacy protection with utility preservation.
Method
PrivBench combines a systematic review of 271 papers with 10 modular evaluation approaches spanning privacy and utility metrics.
Results
The case study shows that DP-based methods are stronger on privacy, entity-based methods on utility, and utility-preserving capabilities dominate the overall privacy-utility balance.
Takeaways & Limitations
PrivBench provides a free, public platform intended to improve comparability and transparency of text privatization effectiveness through holistic evaluation.
Takeaways & Limitations
PrivBench’s evaluation datasets are capped at 1000 texts per module, easing turnaround but producing less robust scores than larger datasets would.
Abstract
from arXiv · showhide
Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of text-to-text privatization, where the goal is to transform a sensitive input text into a privatized output by ideally masking (in)directly identifiable or otherwise private information. The evaluation of text-to-text privatization, however, is not straightforward, and the extant literature has utilized a myriad of techniques and metrics to quantify the privacy-preserving capabilities of privatization methods. Seeking to unify the evaluation of text-to-text privatization, we introduce PrivBench, a holistic and modular benchmarking platform for researchers and practitioners working on text privatization. PrivBench is holistic in that it evaluates privatization on a series of defined desiderata, which are structured into modules. PrivBench is not only modular but also extensible, allowing for future updates and benchmark versions. PrivBench is user-centered and promotes competition via real-time evaluation and a live public leaderboard. The platform is free to use and openly accessible at https://privbench.com/.
1 Introduction
Text-to-text privatization must balance privacy protection with utility, but its evaluation lacks uniformity and accessible comparative mechanisms. PrivBench addresses these shortcomings through a holistic, modular benchmarking platform.
- Text-to-text privatization evaluation must measure privacy protection while preserving utility, creating a privacy-utility trade-off.
- Existing evaluation approaches use diverse techniques and metrics with little standardization or accessible comparative analysis.
- PrivBench evaluates text-to-text privatization holistically across multiple privacy and utility modules.
- The platform runs benchmarks in real time, supports accounts and public leaderboard submissions, and enables competition and comparability.
- PrivBench is openly accessible, free to use, and developed from a survey of evaluation techniques and a comparative case study of 14 privatization methods.
2 Related Work
Prior work has surveyed privacy evaluation and introduced specialized benchmarks, but existing resources do not comprehensively address text-to-text privatization. PrivBench builds on these efforts with a modular, public, interactive, and extensible benchmark.
- Ren et al. surveyed 47 publications and categorized text privacy metrics into six categories, highlighting evaluation’s multi-faceted nature.
- Existing benchmarks target language-model privacy or specific frameworks such as Differential Privacy rather than the wider spectrum of text-to-text privatization.
- TauEval evaluates utility across eight tasks but emphasizes utility over privacy, lacks literature-grounded metric selection, and is distributed as software without a public-facing platform.
- PrivBench is modular, publicly accessible, no-code, interactive, and extensible for future improvements and version updates.
3 Evaluating Text Privatization
PrivBench grounds its evaluation design in a systematic review of text-privatization research, selecting widely adopted privacy and utility approaches for modular benchmarking. The review identified 35 recurring approaches across six categories and operationalized 10 of them using six datasets.
- The authors systematically reviewed 271 papers and aggregated their evaluation strategies to inform PrivBench’s modular design.
- The review used a collaboratively designed search string across four academic databases and screened 2179 deduplicated papers through researcher voting.
- One researcher qualitatively coded all 271 papers, with 40 papers double-coded to assess agreement.
- The review identified 35 evaluation approaches across six categories, retaining approaches appearing in more than two papers.
- Ten evaluation approaches were selected for the first PrivBench version based on adoption, privacy-utility balance, and suitability for text-to-text privatization.
- Six datasets were mapped to the selected approaches, forming the foundation of PrivBench’s evaluation modules.
4 System Design
PrivBench is a web-based benchmarking system designed to standardize evaluation and enable comparison of text privatization methods. Its researcher and administrator workflows support submissions, public rankings, modular maintenance, versioning, and future evolution.
- System architecture: PrivBench uses separate containers, Celery, and Redis to queue and parallelize module evaluations.
- Researcher: PrivBench targets privacy researchers and practitioners by enabling standardized evaluation and comparability with existing text privatization methods.
- Researcher: Researchers submit privatized datasets through a five-step workflow covering information, dataset download, metadata, upload, and benchmarking.
- Researcher: The platform presents benchmark results in user accounts and allows researchers to publish submissions to a searchable, filterable leaderboard.
- Admin: Administrators can add, edit, or remove modules and publish changes through a versioning process.
- Versioning: The versioning system preserves older versions and scores, supports rollbacks, and lets users update or rerun submissions after module changes.
5 A Comparative Case Study on Holistic Text Privatization Evaluation
PrivBench evaluates 14 text-to-text privatization methods across privacy and utility modules, revealing persistent trade-offs and motivating benchmark refinement through redundancy analysis.
- Methods: The case study evaluates 14 text-to-text privatization methods across diverse sub-disciplines using 29 configurations.
- Datasets: The benchmark uses six datasets, with each module mapped to one or more datasets and scores averaged when multiple datasets apply.Each dataset contributes a 1000-row random sample.
- Results: Differential Privacy methods generally protect privacy more strongly, whereas entity-based methods perform better on utility metrics.Entity-based methods include Presidio, Philter, Textwash, and GLiNER; the study also tests LLM-based and other privatizers.
- Results: Few methods consistently preserve utility alongside strong privacy, exposing a central challenge for text-to-text privatization.The case study identifies the need for methods that protect privacy without extreme utility degradation.
- Validation and revision: Five module pairs show high rank correlation, including SIM ↔ TUT at 0.96 and MAU ↔ TUT at 0.94.The correlations are statistically significant with rs > 0.8 and p < 0.05.
- Validation and revision: PrivBench v1.1.0 removes SIM, MAU, and PEM to reduce redundant evaluation signals, while retaining PEM as the basis for LRO.The revised benchmark leaves three utility modules and four privacy modules.
- Validation and revision: After module validation, DP-based methods achieve more favorable balances, while non-DP methods’ empirical privacy difficulties become more prominent.The updated score distributions still show a significant challenge in optimizing privacy and utility jointly.
6 Conclusion
PrivBench is a deployed, public, and free platform for holistic, modular evaluation of text-to-text privatization. Its case study supports improved comparability and transparency while the platform is designed to evolve with the field.
- PrivBench is a fully deployed, public, and free-to-use benchmarking platform for holistic and modular text-to-text privatization evaluation.
- The platform provides a collaborative and competitive environment for researchers and practitioners working in text privacy.
- Its case study demonstrates opportunities for better comparability and higher transparency of text privatization effectiveness.
- PrivBench can continue evolving as understanding of text-privacy evaluation advances.
Limitations
The benchmark is constrained by current knowledge of privacy evaluation, capped evaluation datasets, equal module weighting, and unmeasured computational efficiency and user experience.
- PrivBench reflects the current understanding of how to evaluate technically good privacy protection in textual data.The authors identify modular updates as the mitigation for this scope boundary.
- Evaluation datasets are capped at 1000 texts per module, which improves turnaround time but may reduce scoring robustness compared with larger datasets.
- Overall scores use equal weighting across modules, while future work may model relative weights or human preferences.
- The platform cannot evaluate computational efficiency, despite its relevance to practical applicability.
- The authors did not conduct user studies validating user-friendliness or general user experience.They identify user studies as a next development step as adoption grows.
Ethical Considerations
PrivBench uses licensed public datasets and discourages private or proprietary uploads, while applying access controls and transparent policy practices to mitigate misuse.
- The benchmark’s six underlying datasets are used under their original open-source or stated licenses.
- PrivBench relies solely on public research datasets and does not require proprietary or private data submissions.Users are advised to submit only privatized versions of the provided public datasets.
- Users are explicitly advised against uploading personally identifiable information, proprietary content, or sensitive private data.
- The website applies access controls, bot prevention, and standard secure web-development practices to prevent malicious use.
- PrivBench maintains transparent Privacy Policy and Terms of Service documents intended to be updated as the platform evolves.
A Complete Literature Review Analysis Results
The literature review analysis is summarized by categories of text privatization evaluation metrics. The section also records a reported relative gain of 8.
- The analysis reports a relative gain of 8.
- Table 3 summarizes the literature review analysis by categories of text privatization evaluation metrics.
- The table organizes evaluation metrics into categories for the literature review analysis.
B Detailed System Architecture of PrivBench
PrivBench’s system architecture is presented at the component level, alongside a simplified UML class diagram of its web application.
- Figure 4 presents the component-level architecture of PrivBench.
- The architecture section uses both a component-level view and a UML class-diagram view.
- Figure 5 presents a simplified UML class diagram of the PrivBench web application.