PaperlayerView exact revision on arXiv

Source-linked AI summary

MedDeID enables locally governed clinical-text de-identification from real or synthetic training data

Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe G. Jorens, Kris Laukens

arXiv:2609.10049v1cs.CLcs.LG

TL;DR

Clinical notes are valuable for research but contain identifying information that limits reuse, especially when data cannot leave an institution. MedDeID provides a locally governed workflow using real or synthetic training data, with synthetic training achieving higher primary-care recall than hospital training and English transfer shown on synthetic benchmarks.

  • Problem

    Limited access to shareable clinical text and weak cross-setting transfer constrain locally governed de-identification while preserving clinically useful information.

  • Method

    MedDeID combines local annotation or synthetic-note generation with compact-model training, inference, pseudonymisation, and evaluation across Dutch and English workflows.

  • Results

    90.3% versus 87.0% recall on primary-care notes favored synthetic-trained over hospital-trained models, while the English model transferred to two synthetic benchmarks.

  • Takeaways & Limitations

    MedDeID offers a reproducible route to locally deployable de-identification when real training data cannot be shared, with synthetic training providing complementary variation.

  • Takeaways & Limitations

    Claims are bounded by small benchmarks, including a single-hospital 300-note set and single-physician annotation of the primary-care and synthetic sets.

Abstract

from arXiv · show

Clinical notes contain personally identifiable information (PII), restricting reuse for research and medical AI, especially when data cannot leave an institution. We developed MedDeID, an on-premises framework combining in-house annotation and synthetic-note generation with model training, inference, pseudonymisation and evaluation. On an independently annotated, adjudicated 300-note Dutch hospital benchmark, a hospital-trained compact transformer detected 98.9% of identifying text while redacting 0.24% of text outside annotated identifiers; a synthetic-only counterpart detected 96.1%. On 100 primary-care notes, the synthetic-trained model achieved higher recall than the hospital-trained model (90.3% versus 87.0%) and greater robustness to identifier-format perturbations. An English instantiation trained without real text detected 99.7% and 98.9% of annotated identifier characters on two external synthetic benchmarks. These results demonstrate transfer of the workflow to another language, but not clinical English performance. MedDeID provides a route to locally governed de-identification using real or synthetic training data.

Introduction

Clinical free text is highly valuable for research and medical AI but saturated with personally identifiable information, while Dutch/Flemish de-identification faces language-specific transfer barriers and lacks public real-data benchmarks. MedDeID addresses these constraints through a locally deployable workflow comparing real and synthetic Dutch training and adapting the workflow to synthetic English without establishing real-English clinical performance.

  • Motivation: Clinical free text captures reasoning, history and nuance unavailable in structured fields, making it valuable for clinical research and medical AI but saturated with PII.The relevant records include admission documentation, discharge letters, progress notes and consultation reports.
  • Problem: Dutch and Flemish de-identification is language-specific because identifiers depend on local lexicons, name morphology, date and address conventions, and clinical shorthand.These language-dependent patterns limit transfer from methods developed for other languages.
  • Problem: Public real-data benchmarks for Dutch clinical de-identification remain unavailable, hindering independent reproduction; synthetic pre-training and related methods depend on training regime and evaluation domain.Existing systems also often stop at detection, motivating workflows that address broader de-identification needs.
  • Contribution: MedDeID addresses poor transfer, restricted clinical-text sharing, and information loss during pseudonymisation with a locally deployable workflow supporting training on local annotations and synthetic data.The study compares real-data and synthetic-only training on authentic Dutch clinical text using the same compact architecture.
  • Study design: The study evaluates independent Dutch models on held-out hospital and primary-care notes, then tests a separate synthetic-English model for workflow adaptation without transfer from Dutch models or real-English clinical evaluation.This design tests adaptation to another language rather than rebuilding the entire workflow.

Evaluation datasets and clinical benchmarks

The evaluation used six data resources divided into nine non-overlapping training and evaluation partitions. It included two real Dutch clinical benchmarks and synthetic benchmarks for testing workflow shareability and cross-language reproducibility.

  • Six data resources were divided into nine non-overlapping training and evaluation partitions.
  • The real Dutch benchmarks comprised a held-out, independently annotated and adjudicated hospital set (n=300) and a separately annotated primary-care set (n=100).
  • The Dutch synthetic benchmark tested the shareable complete workflow, while three entirely synthetic English benchmarks tested reproduction in another language.

Performance on the Dutch hospital benchmark

On the 300-note Dutch hospital benchmark, the hospital-trained MedDeID model achieved 98.9% core PII recall with 0.24% non-PII redaction. Its recall was within annotator performance and exceeded all reported external comparators while maintaining the lowest evaluated over-redaction rate.

  • Benchmark performance: 98.9% core PII recall was achieved on the 300-note hospital benchmark, with 0.24% non-PII redaction.The model used a RobBERT-2023 Dutch encoder with dual heads and MedDeID post-processing.
  • Benchmark performance: 98.8% and 98.5% recall were achieved by the two annotators, placing the model’s recall within 0.1–0.4 percentage points of their results.The model’s non-PII redaction rate lay between the annotators’ 0.17% and 0.36%.
  • External comparators: 88.0% recall was achieved by upgraded Belgian DEDUCE, compared with 86.4% for deidentify and 84.2% for Qwen3-8B.Other reported comparators included GLiNER-PII at 76.6%, original Dutch DEDUCE at 75.9%, OpenAI at 72.1%, and OpenMed at 49.1%.
  • Redaction specificity: 0.236% of characters outside gold spans were redacted by the complete MedDeID pipeline, below every other evaluated model and between the annotators’ 0.174–0.359%.Deidentify redacted 0.558%.
  • Metadata augmentation: 98.9% recall followed metadata inclusion, up marginally from 98.8% for the hospital-trained model, while Belgian DEDUCE increased from 78.5% to 88.0%.The pipeline can use patient and caregiver names stored in note metadata.

Synthetic training and cross-setting performance

Synthetic-only training achieved strong de-identification across hospital, primary-care, and public synthetic benchmarks, while outperforming hospital-trained models in primary-care recall but assigning identifier categories less reliably. The workflow therefore supports local deployment without real patient data, with performance varying by setting and evaluation metric.

  • Hospital benchmark: The synthetic-trained model used no real patient data and reached 96.1% core PII recall on the real hospital benchmark.This was 2.8 percentage points below the hospital-trained model but above every tested external comparator.
  • Primary-care benchmark: 90.3% versus 87.0%: synthetic-trained recall exceeded hospital-trained recall on 100 Belgian primary-care notes.The difference was 3.3 percentage points (1.7–5.2), although every system recalled less core PII in primary care than in hospital notes.
  • Category assignment: 98.9% versus 92.6% on hospital notes and 89.8% versus 79.3% on primary-care notes: hospital-trained models assigned identifier categories more accurately.The synthetic-trained model was less reliable after span detection in both care settings.
  • Non-PII redaction: 1.22% versus 0.29%: synthetic-trained non-PII redaction exceeded the hospital-trained model’s rate on primary-care notes.External detectors redacted 11.46% and 8.25% of characters outside gold spans, indicating greater loss of research-relevant text.
  • Public synthetic benchmark: 99.8% core PII recall with 0.28% non-PII redaction: the synthetic-trained model performed strongly on the openly released synthetic benchmark.The hospital-trained model reached 96.6% recall, while deidentify reached 76.7% recall with 0.37% non-PII redaction and Qwen3-8B reached 90.9% recall with 1.28% non-PII redaction.

Performance on English synthetic benchmarks

MedDeID’s English model, trained exclusively on synthetic English text, achieved high annotation-character recall on two external synthetic benchmarks. These results support synthetic-data training feasibility but do not establish performance on authentic English clinical notes or English-language clinical safety.

  • Model and benchmarks: The English instantiation used separate British- and American-English profiles and was trained exclusively on synthetic English text.It was evaluated on two independently sourced external synthetic benchmarks rather than by transferring the Dutch model.
  • Technetium-I: 99.730% annotation-character recall was achieved on Technetium-I, the highest among seven systems.GLiNER achieved 97.773%, a difference of 1.957 percentage points; MedDeID’s non-PII redaction rate was 1.606%.
  • ASQ-PHI: 98.9% annotation-character recall was achieved on ASQ-PHI, the highest among seven evaluated systems.GLiNER achieved 96.3%, a difference of 2.58 percentage points; raw non-PII redaction was 6.21%.
  • Interpretation and limitation: High recall across both benchmarks supports training a competitive English de-identifier entirely from synthetic data.Technetium-I tests structured clinical-note formats at scale, whereas ASQ-PHI tests adversarial queries and hard negatives; neither establishes authentic English clinical-note performance or clinical safety.

Robustness to formatting and value perturbations

The models were tested under changes to identifier values and formatting across three test sets. The synthetic-trained model was generally more stable, with smaller average recall losses and fewer statistically significant declines, although it was not superior for every perturbation.

  • Robustness to formatting and value perturbations: Recall was measured after independently changing names, capitalization, name format, date format, date values, or age format in 300 notes across three test sets.The evaluation used 100 notes from each test set and compared each perturbation with unmodified notes.
  • Robustness to formatting and value perturbations: 19.2 points was the hospital-trained model’s largest recall loss, versus 4.6 points for the synthetic-trained model across all settings.The hospital-trained loss occurred after altering patient-name formatting on the synthetic benchmark; the synthetic-trained loss occurred under primary-care caregiver-name perturbation.
  • Robustness to formatting and value perturbations: 1.25 points was the synthetic-trained model’s mean positive recall loss, compared with 2.96 points for the hospital-trained model.Improvements were counted as zero when calculating mean positive recall loss.
  • Robustness to formatting and value perturbations: 5 versus 7 perturbations produced statistically significant recall losses for the synthetic-trained and hospital-trained models, respectively.The synthetic-trained model was more stable in each test set, but not under every individual perturbation.

Pseudonymisation validity and retained clinical detail

MedDeID’s pseudonymisation layer is designed to preserve clinical meaning while disrupting absolute identifiers through complete date handling, patient-specific shifting, and age-dependent detail reduction. Separate evaluation showed low transformation-layer failure rates, with higher end-to-end failures when model-predicted spans were used.

  • Date handling: Dates are fully healed before substitution, then shifted by one patient-specific offset to preserve intervals while breaking the absolute timeline.Coarse date expressions are represented as compatible calendar intervals, with both bounds shifted and output widened only when needed at the source resolution.
  • Age handling: Age and birth-date detail is reduced according to age, retaining finer units for younger children and neonates to preserve clinically relevant paediatric information.Older patients are represented using whole years; the retained level of detail is reported by age group in Supplementary Table S10b.
  • Pseudonymisation evaluation: 0.0% (0/1,693) synthetic, 0.8% (14/1,798) hospital, and 3.0% (36/1,219) primary-care gold-span transformation failures were observed with a fixed 371-day shift.The pseudonymisation layer was evaluated separately by transforming every gold date and age/birthdate span and checking each output against the intended transformation.
  • Pseudonymisation evaluation: 1.5%, 3.4%, and 6.0% end-to-end failure rates occurred on synthetic, hospital, and primary-care text when model-predicted spans were used.Gold spans with unredacted characters occurred at rates of 0.7%, 2.2%, and 2.4%, respectively.

Inference time on GPU and CPU hardware

Inference on the 300-note hospital set was dramatically faster with the compact hospital-trained model than with the local large language model on the same NVIDIA T4. The hospital-trained model also completed the set on CPU alone in 320.3 seconds.

  • GPU inference: 18.4 s was the hospital-trained model’s GPU time for de-identifying the 300-note hospital set on an NVIDIA T4.The local large language model took 11,411.5 s (3 h 10 min) on the same GPU, a 621-fold wall-time difference.
  • GPU inference: 11,411.5 s (3 h 10 min) was the local large language model’s time for the same 300-note hospital set on an NVIDIA T4.This contrasted with 18.4 s for the hospital-trained model, corresponding to a 621-fold wall-time difference.
  • CPU inference: 320.3 s (5.3 min) was the hospital-trained model’s CPU-only time for the 300-note hospital set.The passage describes this as roughly 36 times faster than the GPU-base.

Deployment and local governance framework

MedDeID is a reproducible, locally governed framework linking synthetic-data generation, annotation, model training, inference, pseudonymisation and evaluation through a shared data contract. It supports institution-specific model development while allowing text de-identification with a local package and compatible model, without requiring clinical training data to leave the institution.

  • Framework architecture: MedDeID connects synthetic generation, human annotation, model training, inference, pseudonymisation and evaluation through a shared contract for taxonomies, JSONL schemas, character offsets and post-processing.The same contract also links predictions to curated, sub-annotated benchmarks that guide annotation, data generation and model refinement.
  • Local governance: MedDeID offers a synthetic-only route when real clinical training data cannot be shared, while hospital-trained models define attainable in-domain performance.The framework connects these complementary routes so institutions can choose locally governed training and deployment strategies.
  • Local deployment: Users who only need de-identification can run the local meddeid package with a compatible model, while institutions can build, evaluate and refine their own models reproducibly.Belgian DEDUCE supplies a directly usable CPU baseline and Belgian lookup foundations.
  • Deployment scope: The workflow remains subject to deployment constraints because prospective adoption across multiple independent healthcare institutions has not yet been evaluated.Performance on authentic English clinical notes also remains unknown because the English model was trained and tested only on synthetic English data.
  • Clinical utility: Post-processing preserves event order and intervals by applying one date shift to all dates for a patient or hospital stay.Age transformations retain clinically meaningful information, with coarser adult handling and graded precision for children.

Competing interests

The authors declare no financial or non-financial competing interests.

  • The authors declare no financial competing interests.
  • The authors declare no non-financial competing interests.
  • Overall, the authors report no competing interests of either financial or non-financial nature.

Supplementary Information · S1. Dataset summary · S2. Annotation scheme

The supplementary information documents dataset composition and defines a two-level annotation scheme for identifying text, including character-level rules for the core PII denominator. It also records benchmark scope and cautions that specialty counts describe the study corpus rather than hospital documentation overall.

  • Supplementary Information: The supplementary information provides extended dataset summaries, benchmark tables, additional analyses and figures, plus the complete English annotation guideline.Dutch- and English-language annotation guidelines are available in the Zenodo archive.
  • S1. Dataset summary: The six evaluation datasets are compared by annotation volume and mapped label composition, while density includes all documents or queries, including 219 ASQ-PHI hard-negative queries.The composition mapping uses nine broad categories and is descriptive rather than inferential.
  • S1. Dataset summary: 55,379 of 579,920 characters (9.55%) in the 300-note hospital benchmark fall within gold annotations, with 42,093 contributing to the core PII denominator.The remaining 524,541 characters form the denominator for the non-PII redaction rate.
  • S1. Dataset summary: The primary-care benchmark has the highest annotation density at 40.1 gold annotations per note despite containing 100 notes compared with 300 in the other listed benchmarks.Its longer notes explain the higher density.
  • S1. Dataset summary: Clinical specialty information was available for 258 of 300 hospital benchmark notes (86.0%), covering 40 mapped departments or specialties.Counts describe the study corpus, not the underlying distribution of hospital documentation.
  • S2. Annotation scheme: The annotation scheme uses span-level labels for identifier type and referent, followed by character-level sub-annotations distinguishing identifying characters from punctuation, titles and clinical context.The scheme draws on NIH/NLM Scrubber, HIPAA’s 18 identifiers and GraSCCo/GeMTeX, adapted to the Belgian and Dutch context.
  • S2. Annotation scheme: Core PII recall includes all sub-annotation categories except formatting, additional_info, medical_info, title and time.Excluded characters may remain in full annotated spans but are omitted from the core PII denominator.

S3. Full Dutch benchmark results … S7. Non-PII redaction analysis

The supplementary analyses detail Dutch benchmark performance, label-specific recall, span detection and label fidelity, perturbation stability, and non-PII redaction across hospital and primary-care text. Results show strong overall detection, category-specific gaps, sensitivity to some input changes, and low non-PII redaction for MedDeID relative to other detectors.

  • S4. Per-label performance: 100.0% recall was observed for patient names, caregiver identifiers, and patient and caregiver addresses, whereas Profession reached 72.7% and Organization:Other 21.4%.The lower-recall categories had 31 and 7 spans, respectively, so these estimates should be interpreted cautiously.
  • S5. Span detection and label fidelity: Core PII recall measures identifying characters redacted regardless of predicted label, while label fidelity measures correct identifier type and role among detected spans.Predictions and annotations were paired one-to-one when their core-PII character positions overlapped.
  • S6. Stability under input perturbations: 1.25 pp of average loss was caused by perturbations for meddeid-dutch-synth across the three test sets.Net loss is a target-span-weighted mean of baseline minus perturbed recall, averaged equally across test sets.
  • S6. Stability under input perturbations: Year-shifted dates produced the clearest consistent degradation, particularly for the hospital-trained model, while most name effects showed little or no change.The hospital-trained model’s recall fell for earlier date years, whereas the synthetic-trained model remained comparatively stable.
  • S7. Non-PII redaction analysis: 0.29% versus 11.46% and 8.25% non-PII redaction was observed for meddeid-dutch-uza versus the OpenAI neural PII and OpenMed detectors on primary-care text.Most excess redaction from the other detectors came from false-positive spans; boundary extensions contributed only a small amount.

S8. Model and timing specifications

The supplementary specifications describe MedDeID’s English synthetic-data training setup, model-selection procedure, transformation evaluations, and configurable age-dependent retained granularity. Benchmark reporting includes uncertainty intervals and cautions that synthetic benchmarks do not constitute clinical validation.

  • Training and model: 6,700 synthetic English clinical documents, split evenly between en-GB and en-US, were used without real patient text.The English model used FacebookAI/roberta-base.
  • Model selection: Selected epoch counts determined final-refit duration, while benchmarks were withheld during selection and evaluated once after refitting.The reported selection counts were 20, 21, and 7 epochs, with selected counts of 17, 17, and 4.
  • Transformation evaluation: Gold-span evaluation isolated deterministic transformation, whereas predicted-span evaluation assessed the full end-to-end pipeline using model predictions.Both used a fixed creation date of 15 January 2025, a +371-day shift, and birthdate-to-age replacement.
  • Age-dependent granularity: Age and birth-date values were reduced more aggressively with increasing age, while finer detail was retained for early childhood.MedDeID Suite allows local customisation of age groups and retained detail levels.
  • Benchmark reporting: 95% document-clustered bootstrap confidence intervals accompanied benchmark percentages, while synthetic benchmarks were identified as reproducibility or in-domain tests rather than clinical validation.Metadata injection did not affect MedDeID recall or non-PII redaction in the held-out English benchmark.

Appendix A. Annotation guidelines

The appendix defines English de-identification annotation rules for personal names, locations, organizations, and dates. Labels are subdivided by relationship to the patient and governed by contextual inclusion and exclusion criteria.

  • Names: Names are classified as patient, caregiver, or other, with patient labels covering first names, surnames, and initials.Caregiver names include healthcare workers; other names include family, friends, non-index patients, and external people, with a reproductive-medicine exception.
  • Names: Caregiver-name annotation includes immediately adjacent professional titles but excludes non-adjacent titles, specialty job descriptions, and organizational roles.The guideline illustrates these distinctions with examples involving doctors, radiologists, department heads, and care-home contacts.
  • Locations: Address_Location labels distinguish patient, caregiver, and other addresses, while country- and region-level locations follow population and address-context exceptions.Care-institution locations are instead labeled Organization:Healthcare when they identify an institution rather than an address.
  • Organizations: Organizations are split into healthcare and other because healthcare institutions may remain clinically useful, whereas unrelated organizations primarily add re-identification risk.Healthcare annotations include institution names and associated codes, route numbers, hospital-specific applications, and relevant filenames, but not medical-device manufacturers.
  • Dates: Dates include numeric and written day, month, or year expressions, with special rules for holidays, weekdays, seasons, times, and linked date ranges.Public holidays count as dates; seasons and clock times do not, and dates are generally annotated separately unless one is essential to another.
Loading 2609.10049v1…