PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

• Text-to-text privatization evaluation must measure privacy protection while preserving utility, creating a privacy-utility trade-off.Source: §1 · p. 1
• Existing evaluation approaches use diverse techniques and metrics with little standardization or accessible comparative analysis.Source: §1 · p. 1
• PrivBench evaluates text-to-text privatization holistically across multiple privacy and utility modules.Source: §1 · p. 1
• The platform runs benchmarks in real time, supports accounts and public leaderboard submissions, and enables competition and comparability.Source: §1 · p. 1
• PrivBench is openly accessible, free to use, and developed from a survey of evaluation techniques and a comparative case study of 14 privatization methods.Source: §1 · p. 2

• Ren et al. surveyed 47 publications and categorized text privacy metrics into six categories, highlighting evaluation’s multi-faceted nature.Source: §2 · p. 2
• Existing benchmarks target language-model privacy or specific frameworks such as Differential Privacy rather than the wider spectrum of text-to-text privatization.Source: §2 · p. 2
• TauEval evaluates utility across eight tasks but emphasizes utility over privacy, lacks literature-grounded metric selection, and is distributed as software without a public-facing platform.Source: §2 · p. 2
• PrivBench is modular, publicly accessible, no-code, interactive, and extensible for future improvements and version updates.Source: §2 · p. 2

• The authors systematically reviewed 271 papers and aggregated their evaluation strategies to inform PrivBench’s modular design.Source: §3 · p. 2
• The review used a collaboratively designed search string across four academic databases and screened 2179 deduplicated papers through researcher voting.Source: §3 · pp. 2–3
• One researcher qualitatively coded all 271 papers, with 40 papers double-coded to assess agreement.Source: §3 · p. 3
• The review identified 35 evaluation approaches across six categories, retaining approaches appearing in more than two papers.Source: §3 · p. 3
• Ten evaluation approaches were selected for the first PrivBench version based on adoption, privacy-utility balance, and suitability for text-to-text privatization.Source: §3 · p. 3
• Six datasets were mapped to the selected approaches, forming the foundation of PrivBench’s evaluation modules.Source: §3 · p. 3

System architecture

• PrivBench uses separate containers, Celery, and Redis to queue and parallelize module evaluations.Source: §4 · p. 4

Researcher

• PrivBench targets privacy researchers and practitioners by enabling standardized evaluation and comparability with existing text privatization methods.Source: §4 · p. 4
• Researchers submit privatized datasets through a five-step workflow covering information, dataset download, metadata, upload, and benchmarking.Source: §4 · p. 4
• The platform presents benchmark results in user accounts and allows researchers to publish submissions to a searchable, filterable leaderboard.Source: §4 · p. 4

Admin

• Administrators can add, edit, or remove modules and publish changes through a versioning process.Source: §4 · p. 4

Versioning

• The versioning system preserves older versions and scores, supports rollbacks, and lets users update or rerun submissions after module changes.Source: §4 · pp. 5–6

Methods

• The case study evaluates 14 text-to-text privatization methods across diverse sub-disciplines using 29 configurations.Source: §5 · p. 6

Datasets

• The benchmark uses six datasets, with each module mapped to one or more datasets and scores averaged when multiple datasets apply.Each dataset contributes a 1000-row random sample.Source: §5 · p. 6

Results

• Differential Privacy methods generally protect privacy more strongly, whereas entity-based methods perform better on utility metrics.Entity-based methods include Presidio, Philter, Textwash, and GLiNER; the study also tests LLM-based and other privatizers.Source: §5 · p. 6
• Few methods consistently preserve utility alongside strong privacy, exposing a central challenge for text-to-text privatization.The case study identifies the need for methods that protect privacy without extreme utility degradation.Source: §5 · p. 6

Validation and revision

• Five module pairs show high rank correlation, including SIM ↔ TUT at 0.96 and MAU ↔ TUT at 0.94.The correlations are statistically significant with rs > 0.8 and p < 0.05.Source: §5 · p. 7
• PrivBench v1.1.0 removes SIM, MAU, and PEM to reduce redundant evaluation signals, while retaining PEM as the basis for LRO.The revised benchmark leaves three utility modules and four privacy modules.Source: §5 · p. 7
• After module validation, DP-based methods achieve more favorable balances, while non-DP methods’ empirical privacy difficulties become more prominent.The updated score distributions still show a significant challenge in optimizing privacy and utility jointly.Source: §5 · p. 7

• PrivBench is a fully deployed, public, and free-to-use benchmarking platform for holistic and modular text-to-text privatization evaluation.Source: §6 · p. 7
• The platform provides a collaborative and competitive environment for researchers and practitioners working in text privacy.Source: §6 · p. 7
• Its case study demonstrates opportunities for better comparability and higher transparency of text privatization effectiveness.Source: §6 · p. 7
• PrivBench can continue evolving as understanding of text-privacy evaluation advances.Source: §6 · p. 7

• PrivBench reflects the current understanding of how to evaluate technically good privacy protection in textual data.The authors identify modular updates as the mitigation for this scope boundary.Source: Limitations · p. 8
• Evaluation datasets are capped at 1000 texts per module, which improves turnaround time but may reduce scoring robustness compared with larger datasets.Source: Limitations · p. 8
• Overall scores use equal weighting across modules, while future work may model relative weights or human preferences.Source: Limitations · p. 8
• The platform cannot evaluate computational efficiency, despite its relevance to practical applicability.Source: Limitations · p. 8
• The authors did not conduct user studies validating user-friendliness or general user experience.They identify user studies as a next development step as adoption grows.Source: Limitations · p. 8

• The benchmark’s six underlying datasets are used under their original open-source or stated licenses.Source: Ethical Considerations · p. 8
• PrivBench relies solely on public research datasets and does not require proprietary or private data submissions.Users are advised to submit only privatized versions of the provided public datasets.Source: Ethical Considerations · p. 8
• Users are explicitly advised against uploading personally identifiable information, proprietary content, or sensitive private data.Source: Ethical Considerations · p. 8
• The website applies access controls, bot prevention, and standard secure web-development practices to prevent malicious use.Source: Ethical Considerations · p. 8
• PrivBench maintains transparent Privacy Policy and Terms of Service documents intended to be updated as the platform evolves.Source: Ethical Considerations · p. 8

• The analysis reports a relative gain of 8.Source: A Complete Literature Review Analysis… · p. 22
• Table 3 summarizes the literature review analysis by categories of text privatization evaluation metrics.Source: Table 3 · p. 22
• The table organizes evaluation metrics into categories for the literature review analysis.Source: Table 3 · p. 22

• Figure 4 presents the component-level architecture of PrivBench.Source: Figure 4 · p. 23
• The architecture section uses both a component-level view and a UML class-diagram view.Source: Figure 4 · p. 23, Figure 5 · p. 23
• Figure 5 presents a simplified UML class diagram of the PrivBench web application.Source: Figure 5 · p. 23