PaperlayerView exact revision on arXiv

Source-linked AI summary

The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization

Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, Montserrat Batet

arXiv:2202.00443v2cs.CLcs.AI

TL;DR

Text anonymization lacks privacy-oriented resources and evaluation that capture disclosure risk beyond predefined entity categories. TAB provides a broad annotated corpus and risk-sensitive metrics, showing that NER-based methods offer weak protection while a fine-tuned anonymization model achieves stronger protection and better privacy–utility balance.

  • Problem

    Existing text-privacy resources focus largely on clinical documents and NER-based de-identification, while anonymization must account for how any textual element can enable disclosure.

  • Method

    The paper introduces the TAB corpus with explicit co-reference annotations and evaluation metrics that weight terms by disclosure risk and conveyed information.

  • Results

    NER-based methods offer weak protection, whereas a pre-trained language model fine-tuned for anonymization provides stronger disclosure protection and a better privacy–utility balance.

  • Takeaways & Limitations

    TAB supports more accurate evaluation and comparison of anonymization algorithms by measuring privacy protection alongside data utility.

  • Takeaways & Limitations

    The benchmark focuses on detecting terms to suppress or mask, while utility-preserving generalization remains future work; manual annotations also provide no formal privacy guarantees.

Abstract

from arXiv · show

We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared to previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored towards measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymisation-benchmark

1. Introduction

Text anonymization must prevent direct and indirect re-identification while preserving useful document content, but existing benchmarks and methods often focus narrowly on predefined entity categories. TAB addresses this gap with a privacy-oriented corpus, metrics, and baseline evaluation.

  • Motivation: Existing de-identification methods can miss less conspicuous re-identifying information and may overestimate privacy protection.Such information includes physical appearance, profession, political opinions, and other elements beyond predefined entity categories.
  • The anonymization task: Text anonymization seeks to mask direct and indirect re-identifying information without altering non-identifying content or its useful semantics.The task therefore involves balancing privacy protection with utility preservation.
  • Contributions: TAB contains 1,268 ECHR court cases annotated with personal-information categories, identifier types, confidential attributes, co-reference relations, and masking decisions.Its annotation process explicitly targets spans that may disclose the identity of a protected person rather than only predefined semantic categories.
  • Contributions: The proposed metrics assess privacy protection and utility preservation more accurately by weighting disclosure risk and evaluating entities rather than isolated occurrences.An entity is considered protected only when all of its mentions are masked.
  • Contributions: Baseline experiments show that NER-oriented de-identification methods provide weak identity-disclosure protection, whereas a BERT model fine-tuned for anonymization provides stronger protection and a better privacy–utility balance.The baselines include generic NER, privacy-oriented NER de-identification, and BERT sequence-labeling models.

3. Related benchmarks

Existing privacy-oriented text benchmarks are concentrated in clinical data or use restricted, shallow, pre-anonymized, or narrow-domain resources. These limitations reduce their realism and suitability for evaluating general text anonymization.

  • Clinical benchmarks: Most existing clinical benchmarks annotate predefined PHI categories, with datasets such as i2b2/UTHealth and CEGS N-GRID centered on medical records.Other efforts cover Swedish clinical notes and synthetic Spanish clinical data.
  • Non-medical datasets: Non-medical datasets are often small, shallowly annotated, non-public, restricted, semi-structured, or limited to NER-oriented categories.Examples include legal documents, pseudonymized learner data, and StackOverflow job postings.
  • Non-medical datasets: Public email corpora such as Enron are less realistic for evaluation because sensitive original content was already redacted or pseudonymized.The Enron corpus contains about 0.5 million messages from 150 employees, but its most disclosive information is not preserved.
  • Alternative sources: Wikipedia biographies avoid privacy issues and offer dense personal information, but they are substitutes for real-world personal documents rather than direct evaluations on such documents.Their use is motivated by public availability and information density.
  • DLP datasets: DLP corpora label documents by overall sensitivity rather than annotating the personal information needed to evaluate anonymization methods.Some originate from data leakages, raising ethical concerns about reuse for anonymization research.

4. The Text Anonymization Benchmark (TAB)

TAB was designed to overcome the clinical, NER-focused, and single-person limitations of earlier privacy datasets. It uses public ECHR court judgments and annotates realistic anonymization decisions for protected individuals.

  • Corpus design: Earlier datasets often simplify anonymization by describing one individual per document, whereas many real documents refer to multiple individuals.TAB was designed to address this more realistic setting.
  • Corpus design: TAB selects rich plain-text documents containing varied direct and quasi-identifiers, biographical details, demographic traits, and event information.The selection criteria also require public text sources and real-world personal information.
  • Corpus construction: The corpus consists of English-language ECHR judgments selected to exclude previously anonymized cases and judgments released after 2018.Annotation focuses on the Introduction and Statement of Facts, where most personal identifiers occur.
  • Annotation process: Annotators receive the document and a specified person to protect, then identify personal information, decide what to mask, classify identifier type and confidentiality, and link entity mentions.This process treats the corpus annotation as an actual anonymization task rather than ordinary entity labeling.
  • Annotation process: Twelve university students spent about 800 hours annotating and quality-checking the court cases, averaging 22 minutes per case.They used detailed guidelines, training, and a web-based annotation interface.

Step 1: Entity Detection

TAB’s first annotation stage identifies personal-information spans using a broad inventory that extends beyond conventional named entities. Subsequent labels distinguish masking needs, identifier type, confidentiality, and links between mentions.

  • Step 1: Entity Detection: Entity detection marks all spans containing personal information and assigns semantic classes without limiting annotation to proper nouns.Demographic information may appear as common nouns or adjectives.
  • Step 1: Entity Detection: The inventory covers people, codes, locations, organizations, demographic attributes, dates, quantities, and miscellaneous personal information.Examples include names, phone numbers, addresses, professions, physical descriptions, diagnoses, and monetary values.
  • Step 1: Entity Detection: Pre-annotations combine spaCy named-entity recognition with handcrafted heuristics, which annotators validate, revise, or remove.The heuristics target entities such as dates, codes, and quantities.
  • Step 2: Masking: Later annotation distinguishes spans that should be masked from those that can remain clear, and separates direct identifiers from quasi-identifiers.Quasi-identifiers can enable re-identification when combined with other information and background knowledge.
  • Confidential attributes: Annotators separately mark confidential attributes such as beliefs, political opinions, sex-related information, ethnicity, and health data.These attributes matter for preventing attribute disclosure even though they are typically not quasi-identifiers.

Step 4: Entity linking

Entity linking connects mentions that refer to the same underlying entity, enabling complete masking despite differing surface forms. The corpus also records annotation decisions, quality review, splits, and agreement characteristics relevant to anonymization.

  • Step 4: Entity linking: Co-reference links connect differently worded mentions of the same entity so all mentions can be masked for successful protection.Identical strings are linked by default, but annotators can override that choice; pronouns and possessives are generally excluded.
  • Corpus release: The released annotations associate each text span with its offset, semantic category, identifier type, confidential attributes, and entity identity.
  • Corpus release: The corpus uses an 80% training, 10% development, and 10% test split, with development and test cases selected for annotation coverage and quality review.
  • General statistics: Annotators masked 67.9% of entities on average, while 4.4% were direct identifiers, 63% quasi-identifiers, and 32% required no masking.
  • Inter-annotator agreement: Entity-type agreement was high for partial span-level κ and α, although exact spans were lower and confidential-status agreement was affected by label imbalance.

6. Evaluation metrics

The paper proposes evaluation metrics that measure text-anonymization privacy and utility more directly than standard IR metrics, accounting for entity completeness, identifier importance, and multiple valid annotations.

  • 6.2 Metrics for privacy protection: Micro-averaging over annotators accommodates multiple valid anonymization solutions and gives greater weight to entities marked by more annotators.A perfect score requires masking all entities marked as direct identifiers by at least one annotator.
  • 6. Evaluation metrics: Three metrics jointly assess anonymization: entity-level recall for direct identifiers, entity-level recall for quasi-identifiers, and weighted token-level precision.The first two target privacy protection, while the precision metric evaluates utility preservation.
  • 6.2 Metrics for privacy protection: An entity counts as protected only when an anonymization model masks every mention of it in a document.This prevents a remaining occurrence from disclosing the identifier despite masking other mentions.
  • 6.2 Metrics for privacy protection: Direct and quasi-identifiers receive separate entity-level recall measures because they differ in their contribution to re-identification risk.The corpus must therefore categorize spans by identifier type and group referring mentions into entities.
  • 6.3 Metrics for utility preservation: Weighted precision penalizes unnecessary masking according to token informativeness, assigning higher penalties to rarer or less predictable content.Token information content is estimated using a masked language model such as BERT, with low predicted probability indicating higher information content.

7. Empirical results

The experiments show that TAB-oriented fine-tuning improves anonymization over zero-shot NER baselines, while entity-level privacy metrics reveal weaknesses hidden by mention-level scores.

  • 7.1 Baseline Performance: Presidio’s mention-level recall is around 0.7, but entity-level recall for direct identifiers ERdi is only around 0.45.Because case numbers were not among Presidio’s trained categories, missing them can leave the applicant identifiable despite detecting names.
  • 7.1 Baseline Performance: Both zero-shot baselines provide a poor balance between privacy protection and utility preservation.The generic NER model over-masks, while Presidio’s broader ORG setting raises recall at the cost of substantially lower precision.
  • 7.1 Baseline Performance: The weighted utility metric WPdi+qi exceeds standard precision because it weights unnecessarily masked terms by information content.This indicates that the unnecessary masks are generally less informative and that the weighted metric better estimates preserved utility.
  • 7.2 Performance of fine-tuned models: Fine-tuned Longformer outperforms the generic NER and Presidio baselines, with the selected model using a 4096-token window and (10,1) label weight.Higher MASK-label weights improve recall, while larger windows improve precision; the final configuration is evaluated in Table 9.
  • 7.2 Performance of fine-tuned models: False negatives are concentrated in MISC, DEM, and ORG entities, whose forms or masking decisions are especially difficult.Regional-court mentions are particularly challenging because they can provide indirect cues about an applicant’s residence.
  • 7.2 Performance of fine-tuned models: The TAB-fine-tuned Longformer detects many personal-information spans in Wikipedia biographies, whereas generic RoBERTa misses many direct and indirect identifiers.This evaluation tests transfer across substantially different linguistic forms.

8. Conclusions and future work

TAB combines a privacy-oriented benchmark with evaluation metrics intended to assess both disclosure protection and utility preservation. The authors identify annotation limitations, rely on public-knowledge assumptions, and plan to extend the benchmark toward privacy-preserving replacements.

  • 8. Conclusions and future work: TAB combines a large, general, freely available corpus with evaluation metrics designed specifically for text anonymization.The corpus and associated metrics are presented as a benchmark for evaluating privacy protection rather than de-identification alone.
  • 8. Conclusions and future work: Manual annotations may contain residual errors and cannot provide formal privacy guarantees like explicit privacy models.
  • 8. Conclusions and future work: The annotations depend on the assumption that potential attackers access only publicly available background knowledge about cases and individuals.
  • 8. Conclusions and future work: NER-based methods offer weak protection, whereas a pretrained language model fine-tuned for anonymization provides stronger disclosure protection and a better privacy–utility balance.
  • 8. Conclusions and future work: The proposed metrics weight terms by disclosure risk and conveyed information, providing a more comprehensive assessment than standard IR-based metrics.
  • 8. Conclusions and future work: Future work will add privacy-preserving replacements for masked terms to evaluate utility-preserving generalization.

Disagreements between annotators

Annotators most often disagreed when entity types overlapped, especially for organizations carrying demographic information. Identifier-type disagreements were concentrated in NO_MASK versus QUASI decisions, with fewer DIRECT versus QUASI disagreements.

  • Disagreements between annotators: DEM–ORG was the most frequent entity-type disagreement, often when organization names also revealed demographic information.
  • Disagreements between annotators: The Ombudsman entity was especially difficult because its label could be DEM, ORG, or PERSON depending on context and interpretation.
  • Disagreements between annotators: Figure 4 visualizes entity-type disagreements as a heatmap of label-mismatch counts.
  • Disagreements between annotators: Table 12 records 4,299 unique masking disagreements after counting identical annotator-pair disagreements only once.
  • Disagreements between annotators: Most masking disagreements involved NO_MASK versus QUASI labels for ORG, DATETIME, DEM, and LOC entities.
  • Disagreements between annotators: Approximately 5% of masking disagreements were DIRECT versus QUASI, mainly for PERSON and CODE entities.

Example of document

The example document concerns an application against the United Kingdom before the European Court of Human Rights. It identifies the applicant, representatives, government agent, legal basis, and procedural history.

  • Example of document: The case originated in application no. 19840/09 against the United Kingdom and was lodged under the Convention.
  • Example of document: Harry Shindler was the applicant, represented by C. Oliver, while the United Kingdom Government was represented by D. Walton.
  • Example of document: The applicant alleged that disenfranchisement caused by residence outside the United Kingdom violated Article 3 of Protocol No. 1.
  • Example of document: The application was communicated to the Government on 14 December 2010 and admissibility and merits were considered together.

I. THE CIRCUMSTANCES OF THE CASE

The applicant was a British retiree living in Italy who lost voting eligibility after exceeding the permitted period of residence abroad. He therefore could not vote in the 5 May 2010 general election.

  • I. THE CIRCUMSTANCES OF THE CASE: The applicant was born in 1921, lived in Ascoli Piceno, Italy, and moved there with his Italian wife after retiring.
  • I. THE CIRCUMSTANCES OF THE CASE: British citizens living overseas may vote for fewer than fifteen years, but the applicant exceeded that criterion and was ineligible.
  • I. THE CIRCUMSTANCES OF THE CASE: The applicant was unable to vote in the United Kingdom general election held on 5 May 2010.

PROCEDURE

The application was lodged against the United Kingdom under the Convention, and the applicant alleged that overseas residence unlawfully disenfranchised him. The Court communicated the application to the Government and joined admissibility with the merits.

  • The application was lodged against the United Kingdom under Article 34 of the Convention by a national whose identity and identifying details are anonymized.
  • The applicant was represented by a lawyer, while the United Kingdom Government was represented by its Agent from the Foreign and Commonwealth Office.
  • The applicant alleged that disenfranchisement resulting from residence outside the United Kingdom violated Article 3 of Protocol No. 1.
  • The application was communicated to the Government, and admissibility and merits were examined together under Article 29 § 1.

I. THE CIRCUMSTANCES OF THE CASE

The guidelines describe a four-stage annotation process for court cases: identify entities, assess masking needs, record confidentiality, and link coreferential mentions. They define detailed semantic categories and span-selection rules for consistent annotation.

  • Annotators identify and correct entities, assign semantic types, determine whether spans are direct or quasi-identifiers, and record confidential attributes and relations.
  • The semantic categories include people, codes, locations, organizations, demographics, dates, quantities, and miscellaneous identifying information.
  • Demographic, temporal, quantitative, organizational, and product-related information receives category-specific treatment, including units and relevant named entities.
  • Annotations should use minimal spans, include titles with person names, and treat connected entities separately unless they partially overlap.
  • The guidelines illustrate annotation decisions with examples covering names, identifiers, locations, organizations, dates, quantities, and products.
  • Ambiguous names are classified according to their referent, with institutions labeled ORG and geographic locations labeled LOC.

Relations

The relations and masking stages connect different mentions of the same entity and classify each entity by its re-identification risk and confidentiality. Reviewers then inspect whether masking sufficiently conceals the protected person.

  • Coreferential mentions such as “John Smith” and “Mr Smith” are linked so that references to the same underlying entity are recognized.
  • Entities are labeled DIRECT_ID, QUASI_ID, or NO_MASK according to whether they directly identify, jointly enable re-identification, or need not be masked.
  • Quasi-identifiers are assessed using publicly available knowledge and combinations of attributes, while immutable personal attributes are generally treated as quasi-identifiers.
  • Confidential information is categorized by sensitive domain, including beliefs, politics, sex, ethnicity, and health.
  • A masked review document replaces direct and quasi-identifiers with asterisks while leaving confidential attributes visible but marked for inspection.
  • Paired reviewers check whether masking conceals the person and inspect the original annotations for omissions, boundary errors, and incorrect labels.
Loading 2202.00443v2…