PaperlayerView exact revision on arXiv

Source-linked AI summary

Meddies-PII: A Multilingual Framework for Personally Identifiable Information Extraction in Clinical De-identification

Linh Uyen Le, Ho Dinh Tri, Christian Hoang, Huy Hoang Ha

arXiv:2609.12544v2cs.CLcs.AI

TL;DR

Clinical PII datasets are costly to annotate, while synthetic alternatives often underdescribe generation or rely on simple prompting. Meddies-PII-Dataset pairs attribute-conditioned multilingual synthesis with deterministic validation. Meddies-PII-Model ranks highest across the reported external and in-domain benchmarks.

  • Problem

    Manually annotated PII datasets are costly to construct, while synthetic alternatives often provide limited generation detail or use relatively simple prompting strategies.

    Source: §1 · p. 1

  • Method

    The paper constructs a multilingual synthetic clinical PII dataset using attribute-conditioned prompts and deterministic validation, then trains a BIOES token classifier on it.

    Source: §1 · pp. 1–2

  • Results

    Meddies-PII-Model achieves the highest performance among evaluated systems across all reported external benchmarks and the Meddies-PII Benchmark.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors report that the model’s benchmark performance demonstrates the downstream utility of the generated data.

    Source: §6 · p. 9

    The dataset is entirely synthetic, and evaluation does not include real clinical records; deterministic gates do not directly assess every document’s semantic naturalness.

    Source: §5 · p. 9

Abstract

from arXiv · show

Clinical de-identification relies on accurately identifying personally identifiable information (PII). However, manually annotated datasets are costly to construct, while existing synthetic alternatives often provide limited details about their generation process or rely on relatively simple synthesis strategies. We introduce Meddies-PII-Dataset, a corpus of one million synthetic clinical documents spanning seventeen languages and nine PII labels. The documents are generated using attribute-conditioned prompts and validated through thirteen deterministic gates that enforce structural and annotation consistency. To evaluate the dataset's utility, we train Meddies-PII-Model, a BIOES token classifier, and compare it with existing PII extraction systems using exact-match entity-level F1. Meddies-PII-Model achieves the highest performance among the evaluated systems on all reported benchmarks, with a mean F1 of 0.827 across fifteen external benchmarks, compared with 0.658 for the strongest baseline. Upon acceptance, we will publicly release the dataset, benchmark suite, model, generation framework, and evaluation code to support research on multilingual clinical de-identification.

1. Introduction

The introduction presents a multilingual synthetic clinical PII dataset and a controlled generation framework, then reports that its trained model leads on the evaluated benchmarks. It motivates this work by the cost of manual annotation and limited transparency in synthetic alternatives.

  • Manual PII datasets are costly and limited in scale, language coverage, and domain diversity, while synthetic alternatives often give little generation detail or use simple prompting.Source: §1 · p. 1
  • The framework combines attribute-conditioned prompting with deterministic validation to generate diverse documents and repair or reject samples that violate requirements.Source: §1 · p. 1
  • The dataset contains one million synthetic clinical documents across seventeen languages, annotated with a nine-label PII ontology.Source: §1 · p. 2
  • Meddies-PII-Model achieves the highest performance on all evaluated external benchmarks and the authors’ benchmark.Source: §1 · p. 2

2. Related Works

Related work covers the scale constraints of manually annotated PII resources, reproducibility concerns in synthetic datasets, and approaches to NER and PII extraction.

2.1. PII Datasets

  • Manually annotated PII datasets support supervised training and evaluation but remain costly to scale across languages and domains.Clinical resources have focused primarily on English, while other corpora extend coverage to French, German, Spanish legal text, and additional multilingual settings [12].Source: §2.1 · p. 2
  • Synthetic datasets offer a scalable alternative, but incomplete descriptions of generation controls can hinder reproducibility.Meddies-PII-Dataset addresses this gap with explicit generation attributes and deterministic validation; SPY is noted as a prior workflow described in greater detail.Source: §2.1 · p. 2

2.2. PII Models

  • NER methods include token-level sequence labeling with optional CRFs [13], contextual transformer encoders, span scoring [15], and schema-driven generalist models [16].Source: §2.2 · p. 2
  • PII extraction maps detected spans to privacy-related labels, and Meddies-PII-Model follows a token-classification approach using BIOES tags and a nine-label ontology.OpenAI Privacy Filter uses bidirectional token classification and span decoding; GLiNER2 offers schema-driven extraction.Source: §2.2 · p. 3

3. Meddies-PII-Dataset

Meddies-PII-Dataset combines multilingual synthetic clinical records and a defined PII ontology with a controlled, staged generation pipeline. The section also describes its benchmark’s language coverage and adversarial examples.

3.1. Overview

  • One million synthetic documents span seventeen languages, with selected samples from two collections mixed to preserve the target language distribution and broader coverage.The language selection emphasizes Southeast Asian languages while also covering other scripts, language families, and resource levels.Source: §3.1 · pp. 3–4
  • The nine-label PII ontology is informed by ISO/IEC 29100:2024 and covers identifiers including names, addresses, dates, contact details, account identifiers, private URLs, and secrets.Source: §3.1 · p. 4
  • The benchmark contains 5,100 documents evenly distributed across seventeen languages, with 100 standard and 200 adversarial examples per language.Each adversarial document combines four to five of thirteen perturbation techniques while preserving annotated PII spans.Source: §3.1 · pp. 4–5

3.2. Dataset Creation

  • Dataset creation proceeds through Preparation, Generation, and Verification, beginning with prompts that combine six attribute groups to control sample content and structure.The groups cover language, PII labels, medical documents, text formats, general rules, and scenarios.Source: §3.2 · p. 5
  • Each sample is generated by one selected model from a pool of nine large language models spanning different model families.Source: §3.2 · p. 6
  • Thirteen deterministic verification gates check annotation and structure, including label coverage, span counts, offsets, phone-number structure, native-script presence, and duplication.Unambiguous formatting errors may be repaired and revalidated; drafts requiring invented content or labels are rejected.Source: §3.2 · pp. 6–7

4. Evaluation

Evaluation compares four public PII systems on fifteen external benchmarks and an in-domain benchmark using exact-match entity-level micro-F1. Meddies-PII-Model ranks first across the reported benchmarks, though absolute scores are lower for Korean and Japanese.

4.1. Evaluation Settings

  • Exact-match entity-level micro-F1 pools entity counts after mapping system labels to a shared ontology, requiring exact PII labels and character boundaries.Source: §4.1 · p. 7

4.2. Results

  • 0.739–0.880 scores on twelve OpenPII 1.5M language benchmarks, with first place in every language; Korean and Japanese absolute scores remain lower.Source: §4.2 · pp. 7–8
  • 0.827 unweighted mean F1 across fifteen external benchmarks, 0.169 above OpenAI Privacy Filter, the strongest baseline by external mean.Source: §4.2 · p. 7, Table 1 · p. 8
  • 0.590 on CredData, 0.872 on Gretel, and 0.956 on Nemotron, versus strongest-baseline scores of 0.370, 0.729, and 0.878, respectively.Source: §4.2 · p. 8
  • 0.878 F1 on the Meddies-PII Benchmark versus 0.540 for the strongest baseline; this in-domain result is reported separately from the external mean.Source: §4.2 · p. 8
  • Upon acceptance, the authors will publicly release the dataset, benchmark, model, generation framework, and evaluation code.Source: §4.2 · p. 8

5. Limitations

The dataset's synthetic construction and evaluation scope constrain how directly its findings apply to real clinical records.

  • The entirely synthetic dataset may not capture all linguistic variation, documentation practices, or annotation ambiguity in real clinical records.Source: §5 · p. 9
  • Deterministic gates enforce declared structural and annotation requirements but do not directly measure the semantic naturalness of every generated document.Source: §5 · p. 9
  • Evaluation focuses on synthetic PII benchmarks and Meddies-PII Benchmark data; appropriately governed real clinical records remain a future evaluation direction.Source: §5 · p. 9

6. Conclusion

The conclusion presents a million-document, seventeen-language synthetic clinical PII dataset and a controlled generation process validated by thirteen deterministic gates. The model leads on reported benchmarks, and the authors plan to release the research artifacts publicly upon acceptance.

  • Meddies-PII-Dataset contains one million synthetic clinical documents across seventeen languages and nine PII labels.Source: §6 · p. 9
  • The refined generation process controls document and scenario attributes, selects one generator per sample from nine models, and applies thirteen deterministic gates.Source: §6 · p. 9
  • Meddies-PII-Model achieves the highest performance among evaluated systems across all reported external benchmarks and the Meddies-PII Benchmark.Source: §6 · p. 9
Loading 2609.12544v2…