Source-linked AI summary
MedDeID enables locally governed clinical-text de-identification from real or synthetic training data
Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe G. Jorens, Kris Laukens
TL;DR
PII in clinical notes and the lack of public real-data Dutch benchmarks limit independent evaluation and reproduction. MedDeID provides a locally deployable workflow using local annotations or synthetic notes; on primary-care notes, the synthetic-trained model achieved 90.3% recall versus 87.0% for the hospital-trained model.
Problem
Public real-data benchmarks for Dutch clinical de-identification remain unavailable, making independent reproduction difficult as groups evaluate on inaccessible local corpora.
Source: Introduction · p. 3Method
MedDeID is a locally deployable workflow supporting training on local annotations or synthetic notes, evaluation on shareable synthetic benchmarks, and clinically informed date and age pseudonymisation.
Source: Introduction · p. 3Results
On primary-care notes, the synthetic-trained model achieved 90.3% recall versus 87.0% for the hospital-trained model.
Source: Discussion · pp. 9–10Takeaways & Limitations
The synthetic-only route offers a reproducible starting point where real training data cannot be shared.
Source: Discussion · p. 12Evidence is bounded by a 300-note single-hospital benchmark and single-physician annotation of the primary-care and Dutch synthetic sets, which may miss identifiers and make dataset differences annotation-dependent.
Source: Discussion · p. 11
Abstract
from arXiv · showhide
Clinical notes contain personally identifiable information (PII), restricting reuse for research and medical AI, especially when data cannot leave an institution. We developed MedDeID, an on-premises framework combining in-house annotation and synthetic-note generation with model training, inference, pseudonymisation and evaluation. On an independently annotated, adjudicated 300-note Dutch hospital benchmark, a hospital-trained compact transformer detected 98.9% of identifying text while redacting 0.24% of text outside annotated identifiers; a synthetic-only counterpart detected 96.1%. On 100 primary-care notes, the synthetic-trained model achieved higher recall than the hospital-trained model (90.3% versus 87.0%) and greater robustness to identifier-format perturbations. An English instantiation trained without real text detected 99.7% and 98.9% of annotated identifier characters on two external synthetic benchmarks. These results demonstrate transfer of the workflow to another language, but not clinical English performance. MedDeID provides a route to locally governed de-identification using real or synthetic training data.
Introduction
Clinical free text is valuable for research and medical AI but rich in identifying information, so de-identification must protect privacy without discarding clinically useful detail. MedDeID addresses gaps in cross-language transfer, data sharing, and pseudonymisation through a locally deployable workflow.
Introduction
- De-identification must cover direct and quasi-identifiers in PII-rich clinical free text, a valuable substrate for research and medical AI.The introduction frames de-identification as a key safeguard for secondary use of clinical notes.Source: Introduction · p. 2
- English systems have mature benchmarks and high-F1 neural models; Philter’s audited UCSF deployment demonstrates production-scale de-identification [6,7,8–11,12].Philter was used to place more than 130 million notes from 2.75 million patients in the hands of over 600 researchers; its audit found no residual identifiers in 17 non-date HIPAA categories and estimated fewer than 0.025% of patients remained at risk through shifted dates.Source: Introduction · pp. 2–3
- Dutch de-identification faces language-specific transfer challenges: DEDUCE is institution-specific, while neural methods generalized across settings with less configuration [13,14,15].Dutch encoders offer practical compact-model backbones [16–18]; the evaluated LLM was slower than the local model and brittle to real-note formatting.Source: Introduction · p. 3
- Public real-data benchmarks for Dutch clinical de-identification remain unavailable; synthetic pre-training, diversity-aware data construction and teacher–student distillation are emerging routes, but benefits depend on training regime and evaluation domain; pseudonymisation requires preserving chronology and clinical meaning [25,26,27].Dates must be shifted so clinical chronology survives, and ages reduced to the coarsest granularity that retains clinical meaning; reducing a two-month-old infant to “0 years old” destroys information a paediatrician needs.Source: Introduction · p. 3
- MedDeID addresses transfer, data-sharing, and pseudonymisation challenges with a locally deployable workflow for local annotations or synthetic-note training, shareable synthetic evaluation, and clinically informed date and age handling.The study compares same-architecture Dutch models trained on annotated hospital notes versus synthetic Dutch notes, evaluated on held-out hospital and primary-care notes.Source: Introduction · p. 3
- A separate English model trained only on synthetic English data tests workflow adaptation, not transfer from Dutch models or performance on real English clinical text.Both instantiations use a common schema, character offsets, post-processing, pseudonymisation rules, and evaluation contract; core PII recall and non-PII redaction quantify detected identifying characters and removed text outside annotated spans, respectively.Source: Introduction · p. 4
Evaluation datasets and clinical benchmarks
Evaluation used six data resources split into nine non-overlapping partitions, including two real Dutch clinical benchmarks and synthetic benchmarks for sharing and cross-language workflow testing. No evaluation data were used for training, and synthetic English benchmarks do not establish performance on real English clinical text.
Clinical benchmarks
- The two real Dutch benchmarks were an independently annotated and adjudicated hospital set (n=300) and a separately annotated primary-care set (n=100).Their gold standards contained 4,269 and 4,010 PII annotations, respectively.Source: Evaluation datasets and clinical… · p. 4
Synthetic benchmarks
- The Dutch synthetic benchmark provides a shareable test of the complete workflow, while three entirely synthetic English benchmarks test reproduction in another language but not performance on real English clinical text [28,29].Source: Evaluation datasets and clinical… · p. 4
Dataset partitions
- Table 1 summarizes six data resources divided into nine non-overlapping training and evaluation partitions, with no evaluation data used for model training.Source: Evaluation datasets and clinical… · p. 4
Annotation composition
- Supplementary Fig. S1 compares annotation volume, density and mapped category composition across all six benchmarks.Source: Evaluation datasets and clinical… · p. 4
Performance on the Dutch hospital benchmark
On the 300-note Dutch hospital benchmark, meddeid-dutch-uza achieved 98.9% core PII recall with 0.24% non-PII redaction. Its recall exceeded the annotators’ reported recall by 0.1–0.4 percentage points, while non-PII redaction remained between their rates.
Performance on the Dutch hospital benchmark
- Core PII recall was 98.9% (98.5–99.3%) and Non-PII redaction was 0.24% (0.16–0.32%) for meddeid-dutch-uza on the 300-note hospital benchmark.The model was trained on 4,470 annotated UZA development notes.Source: Performance on the Dutch hospital… · p. 4
- The model’s recall was 0.1–0.4 percentage points higher than the annotators’ 98.8% and 98.5%, and its non-PII redaction rate fell between their 0.17% and 0.36%.Because the annotators helped create the adjudicated reference standard, this comparison does not show that the model is better than, or equivalent to, physicians.Source: Performance on the Dutch hospital… · p. 4
- Recall was 88.0% for upgraded Belgian DEDUCE, 86.4% for deidentify [14], 84.2% for Qwen3-8B, 76.6% for GLiNER-PII [31], 75.9% for original Dutch DEDUCE [13], 72.1% for OpenAI, and 49.1% for OpenMed.These differences varied by identifier category and by components within annotated identifier spans.Source: Performance on the Dutch hospital… · pp. 4–5
- With the complete MedDeID pipeline, the model redacted 0.236% of characters outside gold spans, below every other evaluated model and between annotators’ 0.174–0.359%; deidentify redacted 0.558%.Main-text results use the complete pipeline with metadata included.Source: Performance on the Dutch hospital… · p. 5
- Of 1,236 non-PII characters redacted, 353 came from entirely incorrect detections and 883 were extra characters at detected PII edges, so most over-redaction reflected imprecise boundaries.The counts are reported for meddeid-dutch-uza.Source: Performance on the Dutch hospital… · p. 5
- Adding patient and caregiver names from note metadata raised meddeid-dutch-uza recall from 98.8% to 98.9%, versus Belgian DEDUCE’s increase from 78.5% to 88.0%.Metadata availability may differ across institutions; results with and without metadata are reported in Supplementary Table S4a.Source: Performance on the Dutch hospital… · p. 5
Synthetic training and cross-setting performance
MedDeID’s synthetic-only Dutch model was evaluated on hospital, primary-care and public synthetic benchmarks. The findings show performance trade-offs across settings and support an open starting point for local deployment and fine-tuning where labelled corpora are unavailable.
Synthetic training and cross-setting performance
- 96.1% core PII recall on the real hospital benchmark was 2.8 percentage points below the hospital-trained model but above every tested external comparator.The synthetic-trained model used no real patient data.Source: Synthetic training and cross-setting… · p. 5
- An entirely synthetic training set provides an open starting point for local deployment and fine-tuning at sites unable to assemble a labelled corpus.The model was trained without real patient data.Source: Synthetic training and cross-setting… · p. 5
- 90.3% versus 87.0% core PII recall on primary-care notes favored the synthetic-trained over the hospital-trained model, a 3.3-point difference.Every system had lower core PII recall on primary-care than hospital notes; the local LLM scored 84.2% versus 60.8%, and DEDUCE 75.9% versus 57.2%.Source: Synthetic training and cross-setting… · p. 5
- 98.9% versus 92.6% on hospital notes and 89.8% versus 79.3% on primary-care notes were correct identifier-category assignments for hospital-trained versus synthetic-trained models.Thus, after detecting a span, the synthetic-trained model was less reliable at assigning its identifier category in both settings.Source: Synthetic training and cross-setting… · pp. 5–6
- 11.46% and 8.25% of non-gold-span characters were redacted by OpenAI and OpenMed on primary-care notes, versus 0.29% and 1.22% for the hospital-trained and synthetic-trained models.These rates indicate loss of research-relevant text that recall-only comparisons do not capture.Source: Synthetic training and cross-setting… · p. 6
- 99.8% core PII recall and 0.28% non-PII redaction were achieved by meddeid-dutch-synth on the openly released synthetic benchmark.The hospital-trained model reached 96.6% recall, while deidentify reached 76.7% recall with 0.37% non-PII redaction and Qwen3-8B reached 90.9% recall with 1.28% non-PII redaction; synthetic in-domain scores do not estimate real-clinical-text performance.Source: Synthetic training and cross-setting… · p. 6
Performance on English synthetic benchmarks
A separately trained English model using only synthetic text achieved the highest annotation-character recall among seven systems on both external synthetic benchmarks. These results support feasibility on synthetic English data but do not establish performance on authentic English clinical notes or cross-language transfer.
Performance on English synthetic benchmarks
- The English model was trained exclusively on synthetic English text using separate British- and American-English profiles, not transferred from the Dutch model.The evaluation used two independently sourced external synthetic benchmarks: Technetium-I and ASQ-PHI, which contains 1,051 queries and 2,973 annotations.Source: Performance on English synthetic… · p. 6
- 99.730% annotation-character recall on Technetium-I was highest among seven systems, versus GLiNER’s 97.773% (a 1.957-percentage-point difference); non-PII redaction was 1.606%.Technetium-I contains 74,700 multi-section clinical notes and 1,161,437 annotations.Source: Performance on English synthetic… · p. 6
- 98.9% annotation-character recall on ASQ-PHI was highest among seven systems, versus GLiNER’s 96.3% (a 2.58-percentage-point difference); age-adjusted non-PII redaction was 0.89%, second to OBI RoBERTa i2b2 at 0.55%.ASQ-PHI leaves explicit ages below 90 unannotated; after excluding those same age characters for every system, other comparator estimates ranged from 1.18% to 4.29%. MedDeID’s raw non-PII redaction rate was 6.21%.Source: Performance on English synthetic… · p. 6
- The benchmarks test structured clinical-note formats at scale and adversarial queries with hard negatives, but their synthetic nature leaves authentic English clinical performance and clinical safety unestablished.Because the English model was trained separately, these evaluations also do not test cross-language transfer.Source: Performance on English synthetic… · p. 7
Robustness to formatting and value perturbations
Across three test sets, the synthetic-trained model was more stable overall under identifier formatting and value perturbations, though it was not better under every individual perturbation. Date-value sensitivity remains important because date distributions vary across settings and over time.
Robustness to formatting and value perturbations
- The perturbation assessment changed one feature at a time across 100 notes from each of three test sets, covering names, capitalisation, name format, date format, date value, and age format.Recall changes were measured relative to the unmodified notes.Source: Robustness to formatting and value… · p. 7
- Changing which names appeared did not significantly reduce recall after correction, while the largest recall loss was 19.2 points for the hospital-trained model versus 4.6 points for the synthetic-trained model.The hospital-trained model’s largest loss followed altered patient-name formatting on the synthetic benchmark; the synthetic-trained model’s largest loss followed primary-care caregiver-name capitalisation.Source: Robustness to formatting and value… · p. 7
- Mean positive recall loss was 1.25 points for the synthetic-trained model versus 2.96 for the hospital-trained model, with significant losses in 5 versus 7 perturbations (Fig. 2).The pattern appeared in each test set, although the synthetic-trained model was not better under every perturbation; the authors note that date-value sensitivity matters as date distributions vary across settings and evolve over time.Source: Robustness to formatting and value… · p. 7
Pseudonymisation validity and retained clinical detail
MedDeID separately evaluates pseudonymisation and applies date and age transformations intended to retain clinically meaningful temporal and paediatric detail. Testing found gold-span transformation failures across synthetic, hospital and primary-care text, with unsupported or invalid source formats indicating a remaining portability gap.
Pseudonymisation validity and retained clinical detail
- MedDeID normalises partially detected dates, shifts exact dates by one patient-specific offset and shifts coarse date intervals without inventing a day.When a shifted interval crosses the source resolution’s boundary, the output is widened; this avoids false precision and directional bias while preserving intervals between events.Source: Pseudonymisation validity and retained… · pp. 7–8
- Age and birth-date reduction retains whole years for older patients but progressively finer units for younger children and neonates.The retained detail is age-dependent to avoid discarding clinically relevant paediatric age information; the levels are reported in Supplementary Table S10b.Source: Pseudonymisation validity and retained… · p. 8
- Gold-span transformation failures were 0.0% (0/1,693) on synthetic text, 0.8% (14/1,798) on hospital text and 3.0% (36/1,219) in primary care.With model-predicted spans, end-to-end failure rates were 1.5%, 3.4% and 6.0%, respectively; gold spans with unredacted characters occurred at rates of 0.7%, 2.2% and 2.4%. Most transformation failures reflected unsupported or invalid source formats.Source: Pseudonymisation validity and retained… · p. 8
Inference time on GPU and CPU hardware
On the 300-note hospital set, the hospital-trained compact model ran much faster than the local LLM on GPU and could also process the set on CPU alone. The passage recommends GPU acceleration for millions of archived notes and reports that larger general-purpose neural PII detectors were slower and had lower recall.
- 18.4 s versus 11,411.5 s: the hospital-trained model de-identified the 300-note set 621-fold faster than the local LLM on the same NVIDIA T4 (Fig. 3).Source: Inference time on GPU and CPU hardware · p. 8
- 320.3 s: the hospital-trained model completed the set on CPU alone, roughly 36 times faster than the GPU-based LLM run; CPU-only Belgian DEDUCE took 17.9 s.Source: Inference time on GPU and CPU hardware · p. 8
- CPU-only processing can serve smaller datasets or routine local batches without dedicated GPU infrastructure, while GPU acceleration is recommended for millions of archived notes.Source: Inference time on GPU and CPU hardware · p. 8
- Larger general-purpose neural PII detectors were slower than the compact transformer and had substantially lower recall across all three evaluation datasets.Source: Inference time on GPU and CPU hardware · p. 8
Deployment and local governance framework
This section describes MedDeID’s locally governed de-identification workflow, pseudonymisation evaluation, reproducibility procedures, and public access to data and software.
Deployment and local governance framework
- MedDeID’s shared contract links annotation, synthetic-data generation, training, inference and evaluation, allowing institutions to adapt models without exporting clinical text or locally trained weights.The framework also links prediction data to a curated, sub-annotated benchmark; Belgian DEDUCE provides a CPU baseline and Belgian lookup foundations. Synthetic notes used regional synthetic PII, exact character offsets and no real patient text.Source: Deployment and local governance… · p. 9, Methods · p. 14
Discussion
- 90.3% versus 87.0% recall: the synthetic-trained model outperformed the hospital-trained model on primary-care notes.On the hospital benchmark, the hospital-trained model reached the range observed for the two physician annotators, while its non-PII redaction rate lay between theirs.Source: Discussion · p. 9
- 84.2% and 60.8% recall: locally deployed Qwen3-8B scored these values on hospital and primary-care notes, respectively, while requiring more hardware and running slower than the compact transformer.Cloud use requires clinical text to leave local infrastructure, and larger models may pose particular practical barriers for smaller institutions.Source: Discussion · p. 10
- The primary-care recall advantage was consistent with synthetic data’s formatting diversity, but the synthetic-trained model was less accurate at assigning labels in both settings.Perturbation experiments found greater sensitivity in the hospital-trained model to name formatting and shifted date values; the utility of label errors remains uncertain and task-dependent.Source: Discussion · p. 10
- The English workflow achieved the highest annotation-character recall among seven systems on two synthetic benchmarks, demonstrating language portability rather than clinical validity.The model was trained and tested only on synthetic English data, so performance on real English clinical notes remains unknown.Source: Discussion · pp. 10–11
- The evidence is bounded by a 300-note, single-hospital benchmark and single-physician annotation of the 100-note primary-care set; multi-institution adoption of the full workflow remains unevaluated.Annotation differences may affect comparisons across datasets, and the annotator comparison uses a gold standard they helped construct.Source: Discussion · pp. 11–12
Methods
- In the study, identifiable records stayed in UZA’s secured environment, inference was local, and no identifiable data were exchanged with the participating general practice.The study received ethics approval, and individual informed consent was waived for retrospective use of routinely collected records.Source: Methods · pp. 12–13
Discussion
- MedDeID’s pseudonymisation preserves event order and intervals with per-patient or per-stay date shifts, while reducing age precision less for infants and young children.Preserving chronology does not guarantee anonymity: weekday and holiday patterns may narrow possible offsets, and date shifting remains pseudonymisation.Source: Discussion · p. 11, Methods · p. 16
Methods
- End-to-end pseudonymisation succeeded only when each gold date or age/birthdate span was fully covered, correctly labelled, and transformed according to protocol.The evaluation compared transformation-layer performance using gold spans with full-pipeline performance using meddeid-dutch-synth predictions; both used a fixed creation date of 15 January 2025, a +371-day shift, and birthdate-to-age replacement.Source: Methods · p. 18
- Confidence intervals used 10,000 document-level bootstrap samples, while stability degradation was tested by one-sided permutation tests with Benjamini–Hochberg correction across 27 cells per model.Rates were calculated from pooled character counts across sampled documents, with the same documents used for all systems; cells with fewer than five paired target spans or contributing notes were excluded.Source: Methods · pp. 18–19
Data availability
- Public synthetic resources include 6,793 Dutch notes and 6,700 English training notes plus a 300-note English benchmark, while real-EHR notes, annotations, and hospital-trained weights are restricted.The Dutch resources comprise 6,493 training notes and a 300-note benchmark; real-EHR materials cannot be shared publicly or on request because of patient privacy, GDPR, ethics, and institutional-governance restrictions.Source: Data availability · p. 19
Code availability
- MedDeID Suite 0.3.0 provides local inference, post-processing, pseudonymisation, training, synthetic-data generation, evaluation, and language profiling; synthetic-trained Dutch and English weights are available.Benchmarking and comparator orchestration code is maintained separately, and the study’s Belgian DEDUCE comparator is retained as a separate release.Source: Code availability · p. 19
Competing interests
The results distinguish performance across Dutch care settings from portability to synthetic English benchmarks, which do not establish performance on real English clinical text. The section also describes the shared de-identification workflow and reports no competing interests.
Competing interests
- The authors declare no financial or non-financial competing interests.Source: Competing interests · p. 25
Tables and figures
- MedDeID links real Dutch hospital and primary-care benchmarks with Dutch and English synthetic resources; character-level sub-annotations and patient or caregiver metadata are unavailable for the external English benchmarks.The cited ASQ-PHI audit covered 300 of 1,051 records and was conducted by clinicians or domain experts in the original study, not the present authors; the complete dataset was not reviewed.Source: Tables and figures · p. 26
- The systems produce candidate character spans that share post-processing: deterministic healing joins or extends detections, and metadata injection adds known patient and caregiver names.Dutch benchmarks use core PII recall; English benchmarks use annotation-character recall because the external English datasets lack sub-annotations.Source: Figure 1 · p. 27
- Hospital-trained MedDeID has the highest recall on the hospital benchmark, while synthetic-trained MedDeID leads on the synthetic and primary-care benchmarks; both retain low non-PII redaction.The table reports performance using metadata; confidence intervals, metadata-free results and the non-PII redaction decomposition are reported in supplementary tables.Source: Table 2 · p. 28, Tables and figures · p. 28
- Synthetic-trained MedDeID English has the highest recall on both external synthetic benchmarks, but the results do not establish performance on real English clinical text.After excluding explicit ages below 90 for a fair comparison, MedDeID’s non-PII redaction was 0.89% (0.75–1.03%), versus 0.55% (0.45–0.66%) for OBI RoBERTa i2b2.Source: Table 3 · p. 29, Tables and figures · p. 29
- 1.2 versus 3.0 percentage points was the synthetic-trained versus hospital-trained model’s overall average recall loss under perturbations, indicating greater stability across all three benchmarks.The overall average weights each benchmark equally; experiments with more target spans contribute more within each benchmark.Source: Figure 2 · p. 30
- MedDeID’s compact neural models occupy the high-recall, low-runtime region on the hospital benchmark, with meddeid-dutch-uza providing the strongest combination.The locally deployed Qwen3-8B was slower and less accurate than the compact models.Source: Figure 3 · p. 31
- Shared schemas and language profiles connect import or generation, annotation, training, inference, benchmark construction and evaluation through consistent JSONL and versioned model artifacts.Annotated data can support model development or benchmark creation, while models and predictions use the same local inference and evaluation workflow.Source: Figure 4 · p. 32
Supplementary Information · S1. Dataset summary · S2. Annotation scheme
The supplementary information characterizes evaluation datasets and documents the annotation scheme used to distinguish core identifying content from other material within annotated spans. It also provides extended analyses and the complete English annotation guideline, while study methods remain in the main manuscript.
MedDeID enables locally governed clinical-text de-identification from real or synthetic training data
- MedDeID’s supplementary information provides extended dataset summaries, benchmark tables, additional analyses and figures, and annotation guidelines in Dutch and English via the Zenodo archive.The complete English guideline is presented in its original styled layout; study methods are reported in the main manuscript.Source: MedDeID enables locally governed… · p. 33
S1. Dataset summary
- Dataset composition is descriptive, not an estimate of clinical identifier prevalence, because differences may reflect setting, document type, synthetic-generation design, and source-label coverage.Annotation density counts gold spans per document or query, including 219 ASQ-PHI hard-negative queries without annotations.Source: S1. Dataset summary · p. 33
- 55,379 of 579,920 hospital-benchmark characters were gold-annotated, while 42,093 formed the core PII denominator; the open synthetic benchmark contained 37,443 core PII characters among 419,578.Remaining hospital-benchmark characters form the denominator for the non-PII redaction rate.Source: S1. Dataset summary · p. 33
- 40.1 gold annotations per note made the 100-note primary-care benchmark the densest dataset, underscoring the need to test performance across datasets with differing annotation volumes and identifier mixes.Fig. S1 compares mapped gold-span counts, spans per document or query, and mapped label composition across six evaluation datasets.Source: S1. Dataset summary · p. 33, Figure S1 · p. 34
- Specialty data were available for 258 of 300 benchmark notes and 4,113 of 4,470 development notes; development-corpus enrichments mean these counts do not represent hospital documentation distributions.The benchmark included 40 mapped specialties and the development corpus 51; psychiatry and general internal medicine were deliberately enriched in the latter.Source: S1. Dataset summary · p. 35
S2. Annotation scheme
- The annotation scheme uses span-level labels for identifier type and relevant entity, then character-level sub-annotations to distinguish identifying characters from punctuation, titles, and useful context.The second layer supports defining the core PII denominator.Source: S2. Annotation scheme · p. 36
- The scheme adapts NIH/NLM Scrubber guidelines [3,4], HIPAA’s 18-identifier set [5], and GraSCCo/GeMTeX [6,7] to the Belgian and Dutch context, while minimizing Anonymize_Other.Anonymize_Other signals that a span needs further consideration rather than serving as a destination label.Source: S2. Annotation scheme · p. 36
- Core PII recall includes all character-level sub-annotation categories except formatting, additional_info, medical_info, title, and time.Literal spaces remain in full spans but are excluded from the core PII denominator; evaluation uses original character offsets.Source: S2. Annotation scheme · pp. 36, 38
S3. Full Dutch benchmark results … S7. Non-PII redaction analysis
The Dutch benchmark results report character-level PII recall and non-PII redaction, with additional analyses of category-level performance, span detection and label fidelity, input perturbations, and redaction sources.
S3. Full Dutch benchmark results
- In primary care, meddeid-dutch-synth achieved 90.3% core PII recall and 1.22% non-PII redaction, versus 87.0% and 0.29%, respectively, for meddeid-dutch-uza.Other evaluated systems had primary-care core PII recall from 37.1% to 74.9%.Source: Table S4c · p. 39, S3. Full Dutch benchmark results · p. 40
- On the Dutch synthetic benchmark, meddeid-dutch-synth had 99.8% core PII recall (95% CI, 99.5–100.0) and 0.28% non-PII redaction (95% CI, 0.21–0.35).Intervals are percentile-bootstrap confidence intervals for the metadata-enabled configuration.Source: Table S4d · p. 39, S3. Full Dutch benchmark results · p. 40
S4. Per-label performance
- For categories with more than 100 spans, meddeid-dutch-uza recall was at least 96.0%, except Organization:Healthcare at 94.7%; Profession and Organization:Other recall was 72.7% and 21.4%.The latter categories had 31 and 7 spans, respectively, so their estimates warrant caution. General-purpose OpenAI and Belgian DEDUCE detectors had 12.6% and 0.0% recall for healthcare organisations, and both had 0.0% for professions.Source: S4. Per-label performance · p. 40
S5. Span detection and label fidelity
- Span-level analysis paired predictions and annotations when core-PII characters overlapped, separating span detection from exact identifier-type and role assignment.Only gold spans containing core-PII characters entered this analysis; the hospital and primary-care sets included 4,263 of 4,269 and 4,008 of 4,010 annotation records, respectively.Source: S5. Span detection and label fidelity · p. 43
S6. Stability under input perturbations
- For meddeid-dutch-synth, perturbations caused 1.25 pp average loss, while gains reduced net loss to 0.60 pp.Metrics were averaged equally across the hospital, synthetic, and primary-care test sets.Source: Table S7b · p. 44, S6. Stability under input perturbations · p. 44
- Year-shifted dates produced the clearest consistent degradation, particularly for the hospital-trained model, while most name effects showed little or no change.Date formatting alone explained little of the larger date-stability differences.Source: Figure S7 · p. 46, Figure S8 · p. 47, Figure S9 · p. 48
S7. Non-PII redaction analysis
- On primary-care text, OpenAI and OpenMed redacted 11.46% and 8.25% of non-PII characters, versus 0.29% for meddeid-dutch-uza.Most excess redaction came from false-positive spans concentrated in a few predicted categories; boundary extensions contributed little.Source: S7. Non-PII redaction analysis · p. 49, Figure S10 · p. 49
S8. Model and timing specifications
The supplement specifies model-training and pseudonymisation procedures and reports English synthetic-benchmark results, including metadata-injection effects and limits on clinical interpretation.
S8. Model and timing specifications
- The English model used 6,700 synthetic clinical documents—3,350 en-GB and 3,350 en-US—with no real patient text and FacebookAI/roberta-base.Selected epoch counts determined final-refit duration, and benchmarks were withheld until final evaluation.Source: S8. Model and timing specifications · p. 50, Table S8 · p. 50
- Training used linear decay, a 0.01 warm-up ratio, an Apple M4 Pro GPU, and early-stopping patience of 3 with a 0.001 minimum improvement.Source: S8. Model and timing specifications · p. 50
S9. Pseudonymisation validation
- Gold-span evaluation isolated deterministic transformation, whereas predicted-span evaluation tested the full pipeline using a fixed creation date, a +371-day shift, and birthdate-to-age replacement.End-to-end failures included unredacted characters, fragmented coverage, incorrect labels, or invalid transformations; gold-transform failures arose from invalid formats or an unconverted birthdate.Source: S9. Pseudonymisation validation · p. 51, Table S10a · p. 51, Figure S11 · p. 52
- Age and birth-date values are reduced more aggressively as age rises, while finer detail is retained for early childhood and the retained levels can be customised.MedDeID Suite supports local clinical or governance requirements through customisable age groups and retained granularity.Source: S9. Pseudonymisation validation · p. 52, Table S10b · p. 52
S10. English synthetic benchmark results
- Technetium-I is a template-generated reproducibility and scale test, while ASQ-PHI analyses exclude shared unannotated age expressions from redaction numerators.For ASQ-PHI, 8,745 non-gold characters across 878 queries were excluded while retaining the original 119,651-character non-PII denominator.Source: S10. English synthetic benchmark results · p. 53, Table S11 · p. 53, Table S12 · p. 53
- 5.19 and 4.06 percentage points were the largest metadata-injection recall gains, for OpenMed Multilingual and OpenAI Privacy Filter, respectively.Injection did not affect MedDeID recall or non-PII redaction; this 14-label in-domain synthetic test does not constitute clinical validation.Source: S10. English synthetic benchmark results · p. 53, Table S13 · p. 53
Appendix A. Annotation guidelines
The guidelines define entity labels and boundary rules for identifying information in clinical text. They distinguish patient, caregiver, and other people and separate identifiable locations, organizations, dates, ages, occupations, contact details, and IDs from non-identifying context.
Name
- Personal names and initials are labeled as Name:Patient, Name:Caregiver, or Name:Other according to the person’s relationship to the record and care.Caregiver titles are included only when immediately adjacent to the name; in reproductive medicine, a partner or child may also count as a patient.Source: Appendix A · pp. 56–57
Address_Location
- Address_Location covers identifiable places from street addresses to smaller regions, with separate patient, caregiver, and other labels; countries are excluded unless part of a larger address.Caregiver locations include addresses of care providers and institutions, while other locations include places associated with external bodies, events, friends, or family.Source: Appendix A · pp. 57–58
Organization
- Organizations are divided into clinically relevant healthcare organizations and other organizations; annotate full healthcare names, while named care locations belong to Organization:Healthcare rather than Address_Location.National health authorities are excluded, and departments or institutions containing place names are classified as healthcare organizations.Source: Appendix A · pp. 58–60
Date and Age_Birthdate
- Dates include written and numeric dates, while seasons and standalone weekdays are excluded; age labels cover birth dates and ages but not gestational age.Publication, copyright, clinical-score, and external-system dates are exceptions; age annotations include the number and its time unit.Source: Appendix A · pp. 60–61
Profession
- Profession covers patients’ and relatives’ occupations, education, voluntary roles, religions, and hobbies, but not treating providers’ jobs; clearly named organizations are labeled separately.Annotate the full job title, including modifiers such as “retired.”Source: Appendix A · p. 61
Contactdetails and IDs
- Contactdetails covers communication information other than postal addresses, while ID labels cover traceable patient or caregiver identifiers, including patient-linked imaging, pathology, and study-protocol codes.IDs specific to care institutions may also be treated as ID:Caregiver; patient-specific URLs fall under ID:Patient.Source: Appendix A · pp. 62–63
Anonymize_Other
- Anonymize_Other is a catch-all for personally identifying patient or caregiver information that does not fit another label, with annotators asked to explain the choice and assess re-identification risk.Use it to flag gaps in the existing label set.Source: Appendix A · p. 64