PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data

Sina Daneshgar, Raya Namazi, Amirali Khosravi, Pouria Rouzrokh, Bardia Khosravi

arXiv:2610.33839v1cs.CLcs.CRcs.LG

TL;DR

Hosted-model de-identification requires sending unredacted clinical notes to a third party, motivating a small local model trained without real patient text. EdgeDeID learns from synthetic notes relabeled into one convention; it performs strongly on a synthetic benchmark, while real-note transfer remains limited.

  • Problem

    Hosted language-model de-identification sends unredacted clinical notes to a third party, while small local models make errors whose missed identifiers can disclose information.

    Source: §1 · p. 1

  • Method

    EdgeDeID fine-tunes a 0.6B model on notes written by three LLM teachers and re-annotated into one convention, without real PHI during training.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    0.994 micro-F1 and no missed identifier in 99% of notes, versus 0.857 and 12% for the strongest of eight off-the-shelf systems, on the synthetic benchmark.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results support synthetic-data fine-tuning of small models as a practical route to local de-identification, with quantized inference using 615 MB of CPU memory.

    Source: §6 · p. 8

    The in-domain benchmark is small, synthetic, and saturated, while MIMIC ICU notes are the only real-text evaluation and cover one note type from one institution.

    Source: Limitations · p. 9

Abstract

from arXiv · show

De-identifying clinical notes with a hosted language model discloses the information the step is meant to protect. We present EdgeDeID, a 0.6B-parameter decoder-only de-identifier that runs locally and is trained without real patient text: three LLM teachers write synthetic notes, a single labeler re-annotates them into one convention, and the model learns to emit a typed XML schema of the identifiers it finds. On a 100-note synthetic in-domain benchmark with manually verified annotations it reaches 0.994 micro-F1 and leaves no missed identifier in 99% of notes, against 0.857 and 12% for the strongest of eight off-the-shelf systems. Quantized, it runs on CPU in 615 MB. Of three additions to supervised training, only a verifier adapter helps: it separates complete from incomplete outputs of other systems at 0.94 AUROC, whereas reinforcement learning with a recall-weighted reward and a rule-based post-processing layer leave in-domain results unchanged. On MIMIC ICU nursing notes F1 falls to 0.48. The errors follow two properties our synthetic notes lack, telegraphic style and low identifier density, and three hand-written in-context examples recover 0.08 F1 without annotated target data.

1 Introduction

Clinical de-identification requires removing identifiers that general-purpose PII filters may not recognize, while hosted models require sending unredacted notes to a third party. EdgeDeID tests whether a small model trained only on synthetic notes can provide local de-identification, alongside reliability mechanisms and transfer evaluation.

1 Introduction

  • General-purpose PII filters may miss clinical identifiers because several Safe Harbor categories have no counterpart in business-text PII taxonomies.The strongest evaluated filter left at least one identifier in 88% of benchmark notes, mostly in categories its schema lacked.Source: §1 · p. 1
  • 0.994 micro-F1 and no missed identifier in 99% of notes, versus 0.857 and 12% for the strongest of eight off-the-shelf systems, on a 100-note synthetic benchmark.The benchmark annotations were manually verified.Source: §1 · p. 1
  • EdgeDeID is a 0.6B model trained on multi-teacher synthetic notes relabeled into one convention, without real PHI; quantized, it runs on a CPU in 615 MB.The paper also reports that the same backbone without fine-tuning scores 0.199.Source: §1 · p. 1
  • The verifier reaches 0.94 AUROC on outputs from nine other systems, while GRPO and rule-based post-processing do not improve the in-domain supervised model.On MIMIC ICU nursing notes, F1 drops to 0.478; three hand-written in-context examples raise it by 0.08.Source: §1 · pp. 1–2

2 Related Work

Prior clinical de-identification work spans rules, taggers, encoders, and LLMs, with performance often weaker across hospitals or note types. This work situates synthetic training, small on-premises models, and output verification within that landscape.

Clinical de-identification

  • Clinical de-identification methods have progressed from rules to recurrent taggers and fine-tuned encoders, which can perform worse on notes from another hospital or document type.Data augmentation can narrow that gap, while systems used without human review may layer rules on top of the model.Source: §2 · p. 2

LLMs for de-identification

  • LLM de-identification studies include zero-shot models, category-specific prompting, fine-tuning on annotated notes, and deployable pipelines, but nearly all rely on large models reached through an API.Prior work also examines lost clinical information and inference cost.Source: §2 · p. 2

Synthetic data and small models

  • Synthetic-data work includes generated-text training and augmentation, while compact clinical models have been explored for settings where privacy and cost matter.EdgeDeID differs from related systems by generating a typed schema and never seeing real text during training; its reported failure case is traced to register and PHI density.Source: §2 · p. 2

Reinforcement learning and verification

  • GRPO has been applied after supervised fine-tuning in other extraction and classification tasks, while prior analyses find it often sharpens existing behavior and is constrained by the KL penalty.The authors report no prior clinical de-identification application and train a dedicated adapter to judge omissions.Source: §2 · pp. 2–3

3 Method

EdgeDeID represents detected identifiers in a typed XML object and trains a small model through synthetic-data supervision, with optional reinforcement learning and post-generation reliability mechanisms. Its pipeline combines teacher-generated notes, consistent relabeling, and mechanisms that flag or add to potentially incomplete annotations.

3 Method

  • A note is represented as typed PHI spans, and the model emits them in a typed XML object; detection is evaluated, while masking and surrogate generation are out of scope.The XML must parse for the output to be usable.Source: §3 · p. 3

3.1 Synthetic Data

  • Three LLM teachers write notes with target lengths and specified PHI, then a first-pass labeler re-annotates the corpus into one convention before 37 writing-form augmentations are applied.The final corpus contains 25,972 training notes and a held-out synthetic split of 404.Source: §3.1 · p. 3

3.2 Supervised Fine-Tuning

  • The SFT prompts use instructions, examples, and a note, with worked examples in 12% of training prompts; tokenizer extensions add 69 schema tokens.The extensions shorten annotations by 37.5% on average, from 536 to 335 tokens, reducing decoding steps.Source: §3.2 · p. 3

3.3 GRPO with a Structure-Gated Reward

  • GRPO samples eight completions per prompt and scores parseable outputs with weighted presence, completeness, calibration, and length terms; unparseable completions receive zero.The completeness term is entity-level F2 against the reference, and a KL penalty anchors training to the SFT policy.Source: §3.3 · p. 3

3.4 Reliability Mechanisms

  • The self-assessment tag is trained only through GRPO calibration reward, while the rule-based QC layer repairs XML, adds pattern matches, and conditionally uses NER.The QC layer runs NER only on notes that appear unreliable, including unrepairable XML or outputs marked incomplete.Source: §3.4 · p. 4
  • The learned verifier uses a LoRA adapter as a second pass to judge candidate annotations without editing them, returning a verdict that can route outputs to human review.Its training candidates include corrupted gold annotations and outputs from four baseline systems, labeled by the zero-leak rule.Source: §3.4 · pp. 4–5

4 Experimental Setup

Evaluation combines a manually verified synthetic in-domain benchmark with MIMIC ICU nursing notes for out-of-domain testing. The authors compare local off-the-shelf systems and report entity-level metrics, paired bootstrap intervals, and decoding details.

4.1 Evaluation Data

  • The in-domain test set contains 100 separately generated synthetic notes with manually verified annotations and is treated as an upper bound because its style matches training data.Gold values were corrected to appear verbatim in their notes.Source: §4.1 · p. 5
  • The MIMIC evaluation uses 2,434 ICU nursing notes with 1,778 human-annotated PHI spans, including separate recall reporting for Safe Harbor patient or relative identifiers.About 60% of notes contain no PHI, so zero-leak rate is not reported for this dataset.Source: §4.1 · p. 5

4.2 Baselines

  • Eight local off-the-shelf systems are run with developer defaults and labels mapped to the study’s types, alongside zero-shot Qwen3-0.6B-Base as a same-size lower bound.The systems include pattern and NER tools, open-vocabulary tagging, clinical encoders, and a general-purpose privacy filter.Source: §4.2 · p. 5

4.3 Metrics and Implementation

  • Entity matches require a same-type prediction containing the full gold span, while precision accepts overlap; reported measures include micro metrics, macro-F1, F2, zero-leak rate, and XML parse rate.Paired bootstrap intervals use 1,000 note resamples; each configuration was trained once with a fixed seed, so intervals do not reflect training variance.Source: §4.3 · p. 5

5 Results

The supervised model leads the in-domain benchmark, while only the verifier adds a capability the model lacks. On MIMIC, performance drops amid whole-note failures and register errors; three in-context examples recover part of the gap, and quantization enables CPU execution in 615 MB.

5.2 Additions to the Supervised Model

  • Only the verifier adds a missing capability, separating complete from incomplete outputs at 0.938 AUROC on the nine other systems.The supervised model already reaches 0.999 recall and 0.990 precision; the verifier’s AUROC is 0.987 across all 11 systems.Source: §5.2 · pp. 5–6, Table 3 · p. 6

5.1 In-Domain Benchmark

  • 0.994 F1 and 0.999 recall, with no missed PHI in 99 of 100 notes, beat OPF’s 0.857 F1 and 0.12 zero-leak rate on the in-domain benchmark.The same results table scores all systems under one protocol, counting categories a system cannot produce against recall and zero-leak rate.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Additions to the Supervised Model

  • GRPO leaves in-domain predictions unchanged, the self-assessment tag is never emitted, and QC changes precision by −0.001 while other metrics remain unchanged.The tag is absent from nearly all sampled training completions, so its calibration term has no effect; QC adds one entity that is not PHI.Source: §5.2 · p. 6

5.3 Out-of-Domain: MIMIC ICU Notes

  • 0.478 F1 on MIMIC trails RoBERTa-i2b2’s 0.675; QC raises Safe Harbor recall from 0.666 to 0.729 at a cost in precision.The MIMIC nursing notes differ from the synthetic notes, and RoBERTa-i2b2 was trained on real i2b2 notes closer to this distribution.Source: §5.3 · p. 6, Table 4 · p. 7
  • Of 200 missed patient and relative names and dates, 155 occur in empty or unparseable whole-note outputs; clinical shorthand also triggers false identifier tags.Controlled edits recover 76% of missed names with a “Patient: name” header and 68% by keeping only the sentence containing the target; title-casing recovers 20%.Source: §5.3 · p. 7
  • Three hand-written examples raise SFT F1 from 0.478 to 0.562 and GRPO F1 from 0.523 to 0.607, while QC raises recall but reduces SFT precision from 0.407 to 0.280.On MIMIC, QC adds 1,376 entities, of which 141 match a gold span; unlike the examples, neither QC nor the verifier corrects register errors.Source: §5.3 · p. 7

5.4 Efficiency

  • 4-bit quantization runs on CPU in 615 MB and takes 3.9 seconds per note, versus 12.4 seconds for OPF on the same CPU, with little accuracy change.The quantized model scores 0.994 F1 with a 0.98 zero-leak rate, compared with 0.99 at full precision.Source: §5.4 · pp. 7–8, Table 5 · p. 8

6 Discussion

EdgeDeID is presented as a local, synthetic-data proof of concept, while failures on telegraphic, low-density notes expose limits in its training coverage. The verifier and QC layer have narrower uses, and reinforcement learning leaves the in-domain model unchanged.

Small models trained on synthetic data

  • The strongest general-purpose PII filter left an identifier in 88% of benchmark notes, while synthetic fine-tuning closed that gap locally.The authors frame multi-teacher generation, consistent relabeling, and a typed schema as a proof of concept for other note types, languages, or identifier definitions.Source: §6 · p. 8

Register and PHI density

  • MIMIC errors track telegraphic register and low PHI density rather than clinical content, and hand-written examples recover 0.08 F1.The authors identify broader synthetic coverage of note styles and PHI densities as a training-time remedy.Source: §6 · p. 8

Reinforcement learning

  • In-domain, GRPO left the model unchanged; on MIMIC, it shifted the model toward precision rather than improving what it detects.The authors interpret this as adjustment of answer behavior more than detection capability at this scale.Source: §6 · p. 8

Three reliability mechanisms

  • The verifier separates complete from incomplete outputs, while the QC layer raises MIMIC recall from 0.666 to 0.729 at a 0.13 precision cost.The verifier did not transfer to MIMIC; QC adds nothing in-domain and is framed as an optional high-recall mode for sites reviewing outputs.Source: §6 · p. 8

7 Conclusion

EdgeDeID shows that a small model trained only on synthetic clinical notes can perform strongly in-domain and run locally, but performance drops on MIMIC ICU notes. The conclusion identifies note register and PHI density as key limits, while reinforcement learning adds little to supervised training.

7 Conclusion

  • 0.994 F1 on in-domain notes and CPU execution in 615 MB show the reported performance and local-deployment result for synthetic-data fine-tuning.The model is 0.6B parameters and trained only on synthetic clinical notes.Source: §7 · p. 8
  • MIMIC ICU performance drops with telegraphic register and low PHI density, while a few hand-written examples recover part of the loss without annotated data.The conclusion identifies the registers and PHI densities represented in synthetic data as the main target for future work.Source: §7 · p. 8
  • Reinforcement learning added little to a supervised model already near its ceiling.The conclusion describes synthetic-data fine-tuning of small models as a practical route to local de-identification.Source: §7 · p. 8

Limitations

The evaluation is limited by synthetic training coverage, a narrow model comparison, and constrained real-text testing. The paper also limits its privacy claim to detection against reference annotations, not protection from re-identification.

Training data and models

  • Training on three teachers’ synthetic notes bounds coverage by what they write, with performance dropping on telegraphic, PHI-sparse MIMIC notes.The study used one backbone family at one size and did not compare a synthetic-trained encoder, so it cannot establish that a generative model best uses this data.Source: Limitations · p. 9

Reinforcement learning

  • GRPO results use one reward design and one run per setting, so findings may not hold for larger models.The authors note that at 0.6B parameters, reinforcement learning may adjust output style and format more than underlying capability.Source: Limitations · p. 9

Evaluation

  • The in-domain benchmark has 100 synthetic notes, and MIMIC is the only real-text evaluation, covering one note type from one institution.The standard i2b2/UTHealth corpora were inaccessible; MIMIC exercises seven of ten categories, and the rule-based Safe Harbor classification is not a legal determination.Source: Limitations · p. 9

Scope of the privacy claim

  • Zero-leak measures residual PHI against reference annotations; it does not guarantee protection against re-identification from quasi-identifiers.The evaluation covers detection only, not surrogate generation.Source: Limitations · p. 9

Ethical Considerations

The paper describes safeguards for using EdgeDeID, its output format and training, and analyses of reward behavior and post-processing. It also reports significance-testing and per-category result conventions.

Ethical Considerations

  • The model reduces manual de-identification effort but does not guarantee anonymity or replace safeguards; outputs should receive human review and institutional validation.Sensitive text need not leave the institution because the model runs locally.Source: Ethical Considerations · p. 9

A Output Schema

  • Every output uses the same ten <phi_entities> containers, with empty categories represented by self-closing tags.Figure 2 illustrates an annotation in the schema.Source: A Output Schema · p. 13, Figure 2 · p. 13

B Training Details

  • Training ran on NVIDIA L40S GPUs, with SFT taking about 3 hours on two GPUs and GRPO taking 59.6 hours on one GPU.The first-pass labeler took about 1 hour on two GPUs; verifier adapter stages took 0.8 and 2.2 hours on one GPU.Source: B Training Details · p. 13

C Reward Details

  • The reward combines presence, completeness, calibration, and length; completeness is entity-level Fβ with β = 2 under guarded substring matching.Length receives full credit up to 1,024 tokens and then decays linearly.Source: C Reward Details · p. 13

D Why the Self-Assessment Tag Was Never Emitted

  • The model never emits <qc_status>; because supervised targets end at </phi_entities>, GRPO’s calibration reward is usually absent and contributes no standardized advantage.The calibration term’s 0.15 reward mass was inert; suggested remedies are to include the tag in supervised targets or train a verdict-only verifier.Source: D Why the Self-Assessment Tag Was… · pp. 13–14

E QC Layer Details

  • On MIMIC, XML repair raises recall by 0.037 but lowers precision by 0.129; pattern detection adds 0.009 recall, while the NER fallback runs on one note.On the held-out training-corpus split, the layer adds 17 entities, only 3 of them PHI, and F1 changes from 0.987 to 0.986.Source: E QC Layer Details · p. 14, Table 10 · p. 14

F Significance Tests

  • Paired bootstrap differences use 1,000 note-level resamples with seed 2025; intervals excluding zero indicate significant differences.QC on SFT gives the same row as QC on GRPO because their benchmark predictions are identical.Source: F Significance Tests · p. 14, Table 10 · p. 14

G Per-Category Results

  • Per-category tables report F1 on both test sets; a dash means the system has no label for that category, not a zero score.Source: G Per-Category Results · p. 14

H MIMIC Analysis

The MIMIC analysis examines annotation-reference choice, Safe Harbor coverage, missed names and dates, and sensitivity to input casing. It finds that capitalization is an unreliable cue in nursing notes.

Reference annotation

  • Every competitive system scores 0.03 to 0.13 F1 higher against the human gold standard than against the rule-based tool output, and middle-table rankings change.About 26% of the tool output’s spans are not PHI, making it easy to mistake for the reference annotation.Source: H MIMIC Analysis · pp. 14–15, Table 14 · p. 16

Safe Harbor classification

  • Less than half of annotated spans are Safe Harbor identifiers, while a third are clinician names that some institutions remove by policy but are not patient identifiers.The classification uses rules over gold type and context and is not a legal determination.Source: H MIMIC Analysis · p. 15, Table 15 · p. 16

Why names and dates are missed

  • Of 713 patient or relative names, initials, and date spans in 373 notes, SFT misses 200; removing malformed-tag artifacts raises recall from 0.72 to about 0.78.Forty misses were tagged in outputs made unparseable by a later malformed tag.Source: H MIMIC Analysis · pp. 15–16

Input case

  • Lower-casing every MIMIC note lowers SFT recall from 0.572 to 0.442, while lower-casing mostly uppercase notes raises precision from 0.415 to 0.453 at a small recall cost.The model uses capitalization as a name cue, but that cue is unreliable in both directions on nursing notes; benchmark F1 changes by less than 0.005.Source: H MIMIC Analysis · p. 16

I Robustness to Synthetic Perturbation

The robustness harness perturbs benchmark notes while re-deriving references from inline tags. Surface-form variation has no effect, while register noise and its combination have measurable but smaller effects than the MIMIC drop.

I Robustness to Synthetic Perturbation

  • The harness rotates PHI surface forms and adds document-level register noise, including upper-casing, joined paragraphs, dropped function words, and semicolon-separated clauses.Notes and references are re-derived together; reverse edits leave benchmark recall between 99.5% and 100%.Source: I Robustness to Synthetic Perturbation · p. 16, Table 16 · p. 16
  • Surface variation has no effect; register noise costs 0.013 recall and 0.11 zero-leak, while combined perturbations cost 0.041 recall and 0.21 zero-leak.These effects are much smaller than the MIMIC drop, consistent with PHI density and note structure—not casing or telegraphic style alone—driving that difference.Source: I Robustness to Synthetic Perturbation · p. 16, Table 17 · p. 16

J Verifier Details

The verifier is evaluated as a review filter across system outputs, with results qualified by the benchmark pool’s composition. The section also reports latency context and baseline handling.

K Full Latency Results

  • The benchmark review threshold is P(incomplete) > 1/3; the zero-shot base model reaches the 1,024-token limit because it never emits its closing tag.The median prompt is about 1,060 tokens, including an 867-token fixed instruction block, so prefix caching helps on CPU.Source: J Verifier Details · p. 17, K Full Latency Results · p. 18

J Verifier Details

  • On the nine systems other than EdgeDeID, the SFT verifier adapter separates complete from incomplete outputs at 0.938 AUROC, versus 0.766 zero-shot.The pooled benchmark AUROC is dominated by EdgeDeID outputs; on EdgeDeID’s own MIMIC outputs the SFT adapter reaches 0.688 AUROC against 0.667 zero-shot.Source: J Verifier Details · pp. 17–18

L Baseline Details

Loading 2610.33839v1…