PaperlayerView exact revision on arXiv

Source-linked AI summary

Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss

Daniel Palacios, Matthew Brady Neeley, Angel Adetomike Otto, Shalini Dhamodharan, John P. Woodhouse, Chi-fan Lin, Mark Zobeck, Zhandong Liu, Hyun-Hwan Jeong

arXiv:2608.17051v1cs.CLcs.AI

TL;DR

Existing de-identification systems can miss institution-specific PHI in clinical notes, despite its regulatory importance. This study benchmarks LLM prompting for that gap and finds that contextual LLM performance and precision–recall control surpass purpose-built systems, while institution-specific prompting recovers missed PHI.

  • Problem

    Existing de-identification systems may miss institutionally situated PHI in clinical notes, although PHI removal is a regulatory requirement.

  • Method

    The study benchmarks LLM-based de-identification on 100 pediatric oncology notes from Texas Children’s Hospital.

  • Results

    LLMs beat purpose-built de-identification systems on recall by 0.20 over Stanford TiDE, while prompting recovered 79% (48/61) of missed institutional PHI categories.

  • Takeaways & Limitations

    In-context learning enables contextual PHI recovery and precision–recall control within a single LLM call, supporting institution-specific prompt development.

  • Takeaways & Limitations

    The 100-note corpus came from a single institution, limiting generalizability across institutions.

Abstract

from arXiv · show

Secondary use of electronic health records requires de-identification, yet existing systems miss \emph{institutionally situated} protected health information (PHI) such as hospital abbreviations, building names, and internal codes whose status is locally determined. We ask whether large language models (LLMs) with in-context learning (ICL) can close this gap and control the precision--recall trade-off. On 100 annotated pediatric oncology notes (5,322 PHI spans) from Texas Children's Hospital, we benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines. Each LLM ran under three prompts of increasing specificity: (1) a HIPAA-aligned baseline, (2) baseline plus the institutional PHI categories it missed, and (3) prompt 2 plus instructions against over-redacting clinical content. We then compared 14~multi-agent and ensemble configurations against the best single prompt, with recall the primary safety metric. LLMs outperformed the purpose-built systems (best F1=0.918$\pm$0.001 vs.\ TiDE 0.779), with advantages concentrated in contextual categories. Naming the missed categories recovered 79\% (48/61) of them, and discouraging over-redaction restored precision. No agentic architecture beat calibrated single-pass prompting (F1 0.906--0.907), but LLM outputs surfaced 414~candidate annotation gaps; re-annotation confirmed 227~PHI spans, against which the final prompt reached recall=0.981 (F1=0.907$\pm$0.002). Well-calibrated ICL resolves both the institutional PHI gap and the precision--recall trade-off in one LLM call per note. LLMs cost more to run than traditional methods, but that cost buys a way to audit the reference standard. LLMs are a legitimate, adaptable alternative to purpose-built de-identification systems; institution-specific prompt development should be the primary adaptation strategy.

1. Quantitative and Computational Biosciences, Baylor College of Medicine, Houston, Texas, USA … INTRODUCTION

Institutionally situated PHI creates a specification gap in clinical-note de-identification because locally meaningful identifiers are not covered by generic HIPAA categories. The study tests whether institution-specific prompting can improve recall and precision without retraining or agentic architectures.

  • 4. Data Science Center, Texas Children’s Hospital, Houston, Texas 77030, USA: The study involved Baylor College of Medicine and Texas Children’s Hospital, including pediatric hematology-oncology and data-science researchers in Houston.The listed affiliations include the Section of Hematology-Oncology and the Data Science Center at Texas Children’s Hospital.
  • INTRODUCTION: Institutionally situated PHI includes hospital abbreviations, building names, internal clinic codes, and provider naming conventions whose identifying status depends on local context.Examples include “TCH” and “Mark Wallace Tower,” which generic HIPAA prompting may not identify.
  • INTRODUCTION: 35 of 81 named patients were uniquely matched to Washington State discharge records containing hospital, diagnosis, and attending-physician data but no names or addresses.The example illustrates that combinations of noncanonical information can enable re-identification.
  • INTRODUCTION: A HIPAA-only prompt cannot know that “TCH” is identifying or that a four-digit pager number identifies staff, indicating a specification failure rather than a capability failure.Institutional note templates, abbreviations, and patient populations differ across sites, limiting reusable specifications.
  • INTRODUCTION: No agentic architecture outperformed the single-pass approach on F1, locating the bottleneck in specification rather than inference-time computation.The work directly tests dual-pass and Scrubber–Auditor pipelines motivated by prover-verifier and multi-agent approaches.
  • INTRODUCTION: The study benchmarks 8 LLMs against Stanford TiDE, OpenMed PII, pattern-based baselines, and multi-stage pipelines on 100 pediatric oncology notes containing 5,322 PHI spans.It introduces institutionally situated PHI as a failure mode common to the compared systems.
  • INTRODUCTION: Naming missed institutional categories recovers most of them, while anti-over-redaction instructions restore precision and expert re-annotation confirms gold-standard gaps as true PHI.The prompting strategy adapts de-identification through in-context learning without fine-tuning.

METHODS

The study benchmarked eight LLMs and established de-identification systems on pediatric oncology notes using progressively institution-specific prompts and multi-agent alternatives. It also expanded the annotation standard through expert review and applied explicit span-level scoring rules.

  • Systems and prompts: Eight LLMs, purpose-built systems, and two pattern-based baselines were evaluated without training or fine-tuning.LLMs ran under Baseline, Targeted, and Precision prompt conditions, progressively adding institutional categories and anti-over-redaction instructions.
  • Corpus: 100 notes from 96 patients at Texas Children’s Hospital formed the primary corpus, with 5,322 PHI spans annotated across 97 notes.Three notes contained no PHI; notes were English-language pediatric oncology records selected by pseudorandom sort.
  • Architectural comparisons: 14 multi-agent and ensemble configurations tested whether iterative, heterogeneous, or role-specific pipelines could outperform single-pass prompt engineering.Architectures included dual-pass refinement, Scrubber–Auditor pipelines, and cross-model dual-pass systems.
  • Annotation review: 227 additional PHI spans were added after expert adjudication confirmed 49 candidate instances spanning 22 unique institutional terms.Candidates came from 414 model-surfaced spans, and the enhanced standard was used only for relative comparison on 10 notes.
  • Evaluation: Recall was computed as TP/(TP+FN) and precision as TP/(TP+FP), with exact substring matching, partial masking, and short outputs treated explicitly.Outputs shorter than 50% of the note were scored as failures with all spans counted as false negatives.

RESULTS

LLMs outperformed traditional de-identification systems, especially on contextual and institution-specific PHI, while prompt design substantially improved recall and precision. Re-annotation revealed incomplete gold standards, and no multi-agent configuration surpassed calibrated single-pass prompting.

  • Overall system performance: Sonnet 4.6 achieved the best LLM performance, combining 96.0% recall and 88.3% precision for F1=0.918±0.001 across five trials.Six LLMs without failure modes exceeded TiDE’s recall by 0.15–0.22, while GPT-oss-20B fell below TiDE because of truncation.
  • Category-level performance: Averaged over the top four LLMs, recall exceeded TiDE by 0.49 on Phone and 0.48 on Other Unique ID, contextual categories where TiDE struggled.TiDE had 15.7% recall on Other Unique ID, 47.9% on Phone, and 65.0% on Geographic Data.
  • Prompt adaptation: Targeted prompting reduced Sonnet 4.6’s institutional false negatives from 61 to 13, recovering 78.7% while raising overall recall from 0.958 to 0.975.Adding anti-over-redaction examples recovered precision to 0.829 at recall 0.969 and yielded F1=0.893±0.004.
  • Enhanced gold standard: Expert re-annotation confirmed all 49 adjudicated instances as PHI and added 227 spans, exposing systematic misses of institution-specific identifiers.On enhanced gold, Precision reached recall 0.981±0.000, precision 0.844±0.003, and F1=0.907±0.002.
  • Agentic and ensemble evaluation: None of 14 multi-agent or ensemble configurations improved F1 over the single-pass Precision prompt; reproducibility-tested methods reached comparable F1 of 0.906–0.908.Cross-Model Vote increased recall from 0.981 to 0.986 but incurred a precision cost and 6× inference cost.

DISCUSSION

The discussion finds that institution-specific prompting lets LLMs outperform purpose-built systems and control the precision–recall trade-off, while multi-agent systems add discovery value but no F1 gain. Generalizability, enhanced-gold comparisons, subgroup fairness, and cross-institutional transfer remain unresolved.

  • Findings: LLMs outperform purpose-built systems, especially on categories requiring contextual reasoning.The advantage reflects contextual reasoning rather than uniformly better performance across all categories.
  • Findings: Prompt choice depends on gold-standard completeness: Baseline wins on original annotations but loses when missed institutional terms are counted.Baseline recall collapses on corrected annotations, while its Precision does not improve.
  • Findings: No multi-agent architecture improves F1 over single-pass prompting, but configurations help enumerate annotation gaps at scale.The prover-verifier framework provides no gain because the bottleneck is specifying which institutional terms to redact, not verifying a redaction.
  • Limitations: Cross-institutional generalization remains untested: the corpus contains 100 notes from one institution, and USDHUB confirms robustness only within that institution.The LLM F1 advantage narrows on canonical-PHI-dominated corpora.
  • Limitations: Enhanced-gold results are biased toward model-discrepant notes, site-specific addenda are in-sample, and subgroup recall differences remain unresolved.The study also reports GPT-oss-20B output truncation and GPT-oss-120B unstable redaction at precision 0.198.

CONCLUSION

Institution-specific prompting closes the institutional PHI gap without retraining, while calibrated single-pass LLMs outperform purpose-built systems and can audit missed annotations. Multi-agent passes add cost without improving the core approach, supporting category-level evaluation and gold-standard auditing.

  • Performance: 0.20 higher recall than Stanford TiDE shows LLMs’ advantage on institutional identifiers that pattern matching cannot reach.The advantage was observed on 100 pediatric oncology notes.
  • Prompt adaptation: 79% (48/61) of missed institutional PHI categories were recovered by naming them in the prompt.Anti-over-redaction instructions then restored precision within a single LLM call.
  • Architecture: None of 14 multi-agent configurations improved F1 over a single pass with the same prompt.Ensembles provided marginally higher recall at a precision cost and required 3 calls per note.
  • Reference-standard auditing: Expert adjudication confirmed all 49 in-context instances as true PHI, showing that LLM outputs can expose gaps in the reference standard.Cases initially resembling over-redaction were largely missed PHI annotations.
  • Implications: Evaluation should be category-specific, gold standards should be audited, and adaptation effort should favor institution-specific prompts over additional passes.LLMs cost more per note but provide adaptation without retraining and a check on the standard itself.

FUNDING · ADDITIONAL CONTRIBUTIONS

The research was supported by fellowships, cancer-informatics funds, foundations, endowments, and institutional and federal grants. Texas Children’s Hospital provided validation data, while generative AI was limited to proofreading and correction, with authors retaining responsibility for the manuscript.

  • FUNDING: Cancer-informatics support came from the Fund for Innovation in Cancer Informatics and CPRIT grant RP240131.The Cancer Prevention and Research Institute of Texas supported the research through CPRIT RP240131.
  • FUNDING: Other funders included the Chao Endowment, Huffington Foundation, and Jan and Dan Duncan Neurological Research Institute at Texas Children’s Hospital.These organizations were listed among the study’s additional sources of support.
  • ADDITIONAL CONTRIBUTIONS: The Texas Children’s Hospital Office of Research Data provided the independently de-identified USDHUB note set for within-institution validation.The validation dataset is described in Supplementary Note 4.
  • ADDITIONAL CONTRIBUTIONS: No AI-assisted tools were used for study design, analysis, or primary manuscript drafting.This restriction was stated alongside the acknowledgment of the validation dataset provider.
  • ADDITIONAL CONTRIBUTIONS: Generative AI was used only to proofread and correct typographical or grammatical errors in author-written text.The tools did not generate scientific content, analyze data, or draft substantive manuscript passages.
  • ADDITIONAL CONTRIBUTIONS: No AI or NLP tool was listed as an author, and the authors reviewed all text and accepted responsibility for its integrity, accuracy, and originality.This policy followed COPE’s position and JAMIA policy.

DATA AVAILABILITY

The evaluation pipeline and code are openly available, while the underlying clinical notes cannot be publicly shared because of patient-health-information and institutional/HIPAA restrictions. De-identified aggregate metrics are provided in the article and supplementary materials.

  • The evaluation pipeline and code are openly available at the project’s GitHub repository.
  • The underlying clinical notes cannot be shared because they are protected patient health information governed by IRB protocol H-52222.
  • The notes are unavailable for public deposition or on request because of institutional and HIPAA data-use restrictions.
  • De-identified aggregate metrics supporting the findings are provided within the article and supplementary materials.

FIGURES

The figures establish the study design, show that LLMs and prompting address institutionally situated PHI, and compare calibrated single-pass prompting with agentic alternatives on an enhanced gold standard.

  • Corpus characteristics: Date comprised 72.1 percent and Name 22.3 percent of gold-standard spans, while institutionally enriched categories comprised 3.7 percent.The latter is a lower bound because of under-annotation described in the Results.
  • Baseline performance: LLMs exceeded traditional systems on F1, while Other Unique ID and Geographic Data showed the lowest recall across systems.For Sonnet 4.6, 29 percent (61 spans) of 211 false negatives were institutionally situated PHI.
  • Prompt effects: Prompt effects were model-dependent: institutional targeting increased recall, and anti-over-redaction instructions recovered precision most for models that initially over-redacted.Models with already-high native precision showed near-zero precision gains from the latter instruction.
  • Enhanced validation and architectures: Single-pass Sonnet (Precision) matched or exceeded multi-agent variants, whose top-three methods had closely overlapping mean F1 of 0.906 to 0.908.On the enhanced gold standard, Sonnet’s Targeted prompt recovered recall to 0.980, while Precision improved precision to F1 = 0.907.

EXTENDED SUPPLEMENTARY MATERIAL · EXTENDED LIMITATIONS AND FUTURE DIRECTIONS

The extended supplementary material records limitations and future directions requiring additional experimentation, re-scoring, or external data. Key gaps concern adversarial re-identification, metric aggregation and scoring, endpoint versioning, and the omitted Spark NLP baseline.

  • EXTENDED LIMITATIONS AND FUTURE DIRECTIONS: The extensions are documented as limitations and future directions, not new results, because they require additional experimentation, re-scoring, or external data.
  • Robustness to paraphrase and residual re-identification risk: The evaluation measures span recall and clinical-content precision but not re-identification by inference, paraphrasing, or cross-referencing against external corpora.Residual quasi-identifiers could include rare diagnoses, unusual treatment timelines, and distinctive narrative phrasing.
  • Robustness to paraphrase and residual re-identification risk: An adversarial protocol applied to the LLM outputs is identified as an important direction for future work.Prior work on surrogate replacement and adversarial re-identification provides a framework for such analysis.
  • Macro-averaged and note-level metric aggregation: Micro-averaged precision, recall, and F1 are dominated by frequent categories, while macro-averaging and note-level distributions would better expose rare-category and worst-case note behavior.The reported heatmaps and per-note recall distributions offer only a partial view; full macro-averaged re-tabulation is deferred.
  • Macro-averaged and note-level metric aggregation: A stricter offset-based scheme requiring exact character-boundary agreement could penalize partial matches credited by the current text-alignment scoring.Comparing all systems this way would require normalizing LLM and traditional outputs to a common span representation; the qualitative ranking is expected to be robust but unquantified.
  • John Snow Labs Spark NLP baseline: The omitted John Snow Labs Spark NLP pipeline leaves the non-LLM comparison without a fully provisioned commercial system, potentially understating traditional-pipeline performance.Its omission resulted from licensing unavailability, and benchmarking it is left for future work.

EXTENDED DETAILED RESULTS AND METHODS

This section provides the complete prompt specifications, detailed analyses, and reproducibility and cost breakdowns supporting the summarized findings. It shows that targeted prompting recovers institutional PHI, while calibrated single-pass prompting remains the practical performance ceiling despite higher cost than local systems.

  • Prompt specifications: Targeted prompting added four institutional PHI categories: credential-adjacent staff names, pager/Voalte numbers, institution names or abbreviations, and building/facility names.Each category included WRONG →RIGHT examples.
  • Prompt performance: 48/61 institutional PHI misses were recovered under Targeted prompting, whereas Baseline treated “TCH” as a clinical abbreviation.The 13 remaining misses occurred in complex compound expressions involving institutional names.
  • Agentic and ensemble analyses: F1=0.942 was the best agentic run, but auditor restoration aggressiveness caused high variance, with F1 ranging from 0.890 to 0.942.Single-pass Sonnet Precision was near-deterministic, with SDF1=0.002 and range 0.905–0.910.
  • Agentic and ensemble analyses: Mean F1=0.907, SD=0.002 for Sonnet Precision single-pass, outperforming every self-refinement or ensemble strategy tested.Cross-model voting reached recall 0.986 but reduced precision to 0.832, while agentic and ensemble workflows required 1.7×–6× the single-pass calls without improving F1.
  • Cost and latency: $0.0009–$0.044 per note, with local systems incurring zero marginal API cost and running 3.5–27 seconds versus 3.3–61.7 minutes for LLMs.The Scrubber–Auditor pipeline cost $0.044/note versus $0.009/note for Sonnet single-pass, approximately 6× more.

EXTENDED SUPPLEMENTARY FIGURES

The supplementary figures show that institutionally situated PHI drives low-recall tails, while targeted prompting improves relevant categories and precision-focused instructions generally improve precision without substantial recall loss. Cross-dataset results largely preserve system rankings, alongside measurable cost and latency trade-offs for LLMs.

  • Per-note recall: Most top models cluster near recall 1.0, but every model has low-recall notes associated with institutionally situated PHI.These per-note distributions indicate that aggregate performance conceals a long tail of difficult notes.
  • Perfect de-identification: DeepSeek and GLM-5 achieve perfect de-identification on ∼60% of notes, while no model exceeds 65%.The limited perfect-de-identification rates reflect the persistent long tail of institutionally situated identifiers.
  • Targeted prompting: Targeted prompting concentrates recall gains in Other Unique ID and Name for responsive models, whereas MiniMax M2.5 shows broad regressions.Geographic Data exhibits mixed effects under targeted prompting.
  • Precision-focused prompting: Precision-focused instructions improve precision across most models, with GPT-oss-120B showing the largest improvement and GPT-oss-20B suffering output failures.Strong models show smaller recall deltas, supporting precision improvement without substantial recall degradation; GPT-oss-20B and MiniMax M2.5 are regression outliers.
  • Operational trade-offs: On USDHUB, Opus 4.8 achieves the highest F1 at $2.14 for the corpus, while LLMs take 3–62 minutes and local systems finish in under 1 minute.Traditional baselines incur zero marginal API cost, whereas GPT-oss-20B’s 62-minute runtime reflects retry overhead from persistent API errors.
  • Cross-dataset validation: Performance rankings are largely preserved across the primary benchmark and USDHUB, supporting generalizability across datasets.On USDHUB, TiDE reaches recall 0.976 fully unprovisioned and Opus 4.8 leads on F1 at 0.894.
Loading 2608.17051v1…