Source-linked AI summary
Mind the Gap: Robustness Risks in PII Detection Systems
Adeel Zafar, Slawomir Nowaczyk
TL;DR
PII detection benchmarks often use clean text even though deployment involves messy, shifting inputs where missed entities create privacy and security risks. This paper compares three detection paradigms on a seven-category stress test and finds degradation with distinct failure modes, motivating hybrid mitigation and OOD-aware evaluation.
Problem
Standard PII benchmarks use well-edited text and do not capture realistic distribution shifts across deployment domains, limiting evidence about production robustness.
Method
The paper evaluates encoder-based NER, rule-based hybrid detection, and generative LLM extraction on a stress test spanning seven natural distribution-shift categories.
Results
All three architectures degrade on OOD inputs, with distinct failure modes; Location recall falls from 0.725 to 0.457 across models on unstructured inputs.
Takeaways & Limitations
No single architecture is uniformly reliable across PII categories, supporting hybrid detection and OOD-aware evaluation with iterative QA-driven risk mitigation.
Takeaways & Limitations
The benchmark captures textual style shifts within a general domain, while real deployments add domain-specific vocabulary and formatting across multiple verticals.
Abstract
from arXiv · showhide
Personally Identifiable Information (PII) detection is a foundational component of data protection infrastructure where missed entities constitute direct privacy and security risks. Although modern PII systems report strong performance on standard benchmarks, we show that these evaluations mask substantial robustness failures under realistic distribution shifts encountered in deployment. Rather than comparing state-of-the-art accuracy, we study how different PII detection paradigms fail under noisy, unstructured, and informal inputs. We construct a stress test benchmark spanning seven categories of natural distribution shift and evaluate representative systems from three widely deployed architectural families: encoder-based NER (SpaCy), rule-based hybrid detection (Presidio), and generative LLM extraction (Qwen2.5-3B). All three exhibit significant degradation on out-of-distribution inputs, but with distinct and complementary failure modes. Encoder models primarily fail on unseen surface forms and boundary detection, rule-based systems fail on non-standard formats, and LLMs exhibit entity-type confusion and generation instability. These results show that aggregate benchmark scores obscure deployment-critical weaknesses and that no single architecture is uniformly reliable across PII categories. Motivated by these findings, we propose a hybrid detection pipeline with a QA-driven feedback loop for iterative risk mitigation, and release our benchmark to support OOD-aware evaluation of PII systems.
1 Introduction
PII detection systems can look strong on clean benchmarks yet fail under realistic distribution shifts, creating distinct weaknesses across encoder, rule-based, and LLM architectures.
- Motivation: PII detection is safety-critical because missed entities become data protection failures when systems redact sensitive information.Organizations use these systems before data is stored, shared, or processed.
- Motivation: Standard benchmarks report F1 scores exceeding 90% while using well-edited, formally structured text unlike messy real-world inputs.This mismatch makes benchmark confidence misleading for deployment.
- Motivation: Cross-domain deployment compounds distribution shifts because healthcare, finance, support, and social-media text differ in context, formatting, and vocabulary.Entity types may remain the same while surrounding language patterns change substantially.
- Approach: The study evaluates encoder-based NER, rule-based hybrid detection, and generative LLM extraction to characterize complementary robustness failures.It prioritizes failure analysis under realistic shifts rather than selecting a single leaderboard-leading detector.
- Results: Location recall drops from 0.725 to 0.457 across all models on unstructured inputs, while rigid SSN and IP entities remain largely unaffected.Flexible-format entities such as Location and Address suffer the most.
- Results: The same input exposes distinct failures: encoders miss locations, rule-based systems miss locations, and LLMs merge cities into ADDRESS spans.These architecture-specific errors leave PII exposed despite different internal behavior.
- Results: The LLM has the best overall F1 but lower PERSON recall than SpaCy and worse CREDIT_CARD detection than Presidio, while also producing unparseable OOD outputs.The reported comparisons are PERSON recall 0.777 vs 0.951, CREDIT_CARD recall 0.543 vs 0.700, and 2.5% unparseable outputs.
2 Related Work
Prior work establishes broad NLP robustness challenges and NER sensitivity to entity substitutions, domain shifts, and continual-learning effects, but lacks a production-realistic PII comparison across architectures.
- NER and PII Detection: NER evolved from statistical models to transformer-based token-level BIO classification evaluated on benchmarks such as CoNLL-2003 and OntoNotes 5.0.
- NER and PII Detection: PII detection extends NER with structured identifiers, motivating Presidio’s hybrid combination of pattern matching and machine-learning recognizers.
- Robustness Gap: PII types span rigid formats captured by regular expressions and flexible formats requiring contextual understanding, but this robustness spectrum has not been systematically studied.
- Distribution Shift: WILDS and Robustness Gym show that distribution shifts and multi-axis robustness failures recur across tasks and domains.
- Existing Evaluations: Prior NER audits use entity replacement and adversarial substitutions, but these controlled perturbations do not fully represent natural production variation.
- Existing Evaluations: Cross-domain NER evaluation finds substantial drops when models move between domains despite unchanged entity types, paralleling PII deployment conditions.
- Continual Learning: Continual-learning studies warn that training on new entity types can cause catastrophic forgetting, motivating regression checks during iterative improvement.
- Research Gap: No existing benchmark systematically evaluates natural production shifts or compares encoder-based, rule-based, and LLM-based PII architectures on one OOD benchmark.
3 Benchmark Design
The benchmark measures the gap between clean and shifted PII detection using a formatted evaluation set and a seven-category OOD stress test with controlled synthetic annotations.
- Benchmark Structure: The benchmark contains two evaluation sets designed to measure in-distribution versus out-of-distribution performance.
- Set A: Set A contains 100 well-formatted examples, 350 entity annotations, and nine PII types generated from 10 manually authored templates.
- Set B: Set B covers seven distribution-shift categories with 80 examples per category, totaling 560 examples and 2,330 entity annotations.Gold spans are obtained automatically because the inserted synthetic PII values are controlled during generation.
- Dataset Statistics: Benchmark statistics show that mixed-context PII has the highest entity density, whereas overlapping entities has the lowest.
4 Experiments
All evaluated architectures degrade on OOD inputs, but their robustness profiles differ by entity format, distribution shift, and failure mechanism. Aggregate performance therefore conceals deployment-critical weaknesses, especially for flexible entities such as locations.
- Overall robustness: 0.073: The LLM achieves the highest overall F1 on both sets, but also has the largest robustness gap; Presidio has the smallest gap at 0.057.SpaCy falls between them with a gap of 0.064.
- Architecture-specific failures: Encoder models primarily miss unseen entities and absorb neighboring text into spans, while the LLM misclassifies or merges entities and Presidio fails on non-standard formats.Examples include unseen synthetic locations, unpunctuated boundary absorption, and credit-card numbers falling outside expected patterns.
- Category analysis: LLMs handle typos best, reaching F1 0.869 versus 0.522 for SpaCy, but generative extraction introduces entity-type confusion and malformed outputs.The LLM's typo advantage is attributed to exposure to misspellings during pretraining.
- Shared failure modes: Location failures recur across architectures because synthetic or name-like city names are missed, misclassified as persons, or absorbed into broader ADDRESS spans.This shared failure suggests a structural limitation in handling geographic entities within unstructured contexts.
5 Discussion and Conclusion
The findings indicate pervasive OOD robustness gaps, with architectures failing differently and clean benchmark scores understating production exposure. The paper therefore motivates hybrid detection, QA-driven iteration, and broader evaluation while acknowledging important benchmark limitations.
- Risk Assessment: Clean-data performance understates deployment risk because the benchmark covers only textual style shifts within a general domain, whereas real deployments add domain-specific vocabulary and formatting.The observed robustness gaps are therefore likely a lower bound on production conditions.
- Risk Assessment: 21.3% of PII entities remain undetected even by the best OOD model, Qwen, which achieves F1 0.848.Location entities have a 54% miss rate, while the LLM produces zero output on 2.5% of inputs.
- Hybrid Architecture: Different architectures fail on different inputs, motivating systems that combine encoders, regex layers, LLM fallbacks, and entity merging.The proposed hybrid assigns well-formed text, rigid-format PII, and ambiguous inputs to complementary components.
- Iterative QA: QA teams can categorize production failures and route them to architecture-specific mitigations, including fine-tuning, new regex patterns, prompt refinement, or targeted LLM tuning.The pipeline then verifies improvements through regression tests that include both previously failing and previously passing cases.
- Limitations: The benchmark is synthetic, limited to English and one general domain, and evaluates one representative model per architectural family.Future work should test naturally occurring multi-domain text, multilingual settings, stronger models, larger LLMs, and fine-tuned variants.