PaperlayerView exact revision on arXiv

Source-linked AI summary

Broad and Efficient PII Detection with Unified Labels

Andy Hsu, Konstantin Berlin

arXiv:2610.22597v1cs.CLcs.CR

TL;DR

PII detection must find identifiers across varied contexts with high recall at low false-positive rates, but datasets disagree on categories and labeling decisions. This paper relabels public datasets under shared contextual criteria and trains a compact detector. It reports leading low-FPR recall against larger baselines with substantially lower compute costs.

  • Problem

    PII datasets have differing taxonomies and example-level labeling decisions, while many models use only a few synthetic datasets and complex architectures raise deployment cost.

    Source: §1 · pp. 1–2

  • Method

    The authors define contextual criteria for 12 structured PII categories, use GPT-5 Mini to relabel data under category-specific constitutions, and train a compact detector on the unified corpus.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    At a low-FPR operating point, the 141M-parameter detector has the highest recall in 11 of 14 comparisons under unified labels and 9 of 14 under original labels, with 2–10× higher throughput and 94–97% less peak GPU memory than larger baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The detector can serve as a first-stage scanner so more expensive detection stages need to run only on a smaller set of candidates.

    Source: §1 · p. 2

    Only 9 of the 12 supported categories have comparable benchmark and baseline coverage, and operating points calibrated on WildChat may require evaluation and recalibration for other traffic distributions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Personally Identifiable Information (PII) detection models must efficiently find identifiers across diverse contexts while maintaining a low false-positive rate (FPR) on high-volume traffic. Recent detectors increasingly rely on complex architectures that raise deployment cost, yet many are fine-tuned on only a few synthetic datasets that limit their coverage. Combining public datasets is challenging because their taxonomies differ, and even after categories are mapped to a common schema, labels can disagree on what counts as a detection, producing inconsistent training and evaluation signals. To address these problems, we present a practical 141M-parameter encoder-based detector trained on 1.12 million examples from 77 public datasets, relabeled under a detailed definition of broad contextual detection for 12 structured PII categories. Evaluated on the AI4Privacy and Nemotron-PII benchmarks at a low-FPR operating point, our model achieves the highest recall against larger baselines in 11 of 14 comparisons under our labels and 9 of 14 under the original labels. Its largest recall gains include useful additional coverage, such as multilingual disclosures and IP addresses embedded in URLs. It achieves this with 2-10x higher throughput and 94-97% less peak GPU memory than the baselines, delivering leading detection performance at substantially lower cost.

1 Introduction

PII detectors need high recall at low false-positive rates, but training data and label disagreements limit coverage while complex models raise deployment costs. This work unifies contextual criteria across datasets and trains a compact detector with strong recall and efficiency.

1 Introduction

  • Training data often comes from a few synthetic datasets, while differing taxonomies and labeling decisions create inconsistent signals and newer complex models raise cost and latency.Disagreements include whether to label a card number shown only in its last four digits or a business phone number.Source: §1 · pp. 1–2
  • The approach defines broad contextual criteria for 12 categories and relabels datasets at the example level before combining them into a training corpus.The criteria span countries, formats, and languages, and the process can add benign examples without original PII labels.Source: Figure 1 · p. 1, §1 · p. 2
  • The work relabels two benchmark datasets and 77 public datasets, producing about 1.12M training examples under unified labels.The relabeled benchmarks remove positives lacking category context while adding coverage such as identifiers within larger spans.Source: §1 · p. 2
  • At a low-FPR operating point, the 141M-parameter detector achieves the highest recall in 11 of 14 comparisons under unified labels and 9 of 14 under original labels.It also delivers 2–10× higher throughput and uses 94–97% less peak GPU memory than larger baselines.Source: §1 · p. 2

2 Related Work

PII datasets differ in taxonomy and annotation practice, so combining categories alone does not make their example-level labels consistent. Prior work explores taxonomy alignment, LLM labeling, and flexible detection architectures, while this paper relabels diverse datasets under one definition.

2.1 PII Datasets

  • PII datasets vary in category names, breadth, and granularity; B2NER and PIIBench align taxonomies, but preserve differences in how examples were labeled.B2NER combines 54 datasets into a taxonomy of over 400 entity types, while PIIBench maps over 80 labels from ten datasets into 48 categories.Source: §2.1 · p. 2

2.2 Labels and Labeling Definitions

  • LLM labeling work uses written guidelines or labels previously unlabeled text, whereas this approach relabels many datasets and replaces existing decisions.Benchmark labels can follow different methodologies or contain errors, motivating relabeling complete benchmarks and examining disagreement patterns across models.Source: §2.2 · p. 3

2.3 PII Detection Models

  • PII detectors range from fixed-category span models to flexible GLiNER variants and privacy-focused classifiers, whose detailed behavior can add inference cost.Newer architectures still depend on their training data and task definitions; fine-tuning XLM-RoBERTa with 100 to 1,000 labeled examples surpassed Privacy Filter zero-shot performance in one analysis.Source: §2.3 · p. 3

3 Method

The method defines broad contextual detection for 12 structured PII categories, applies category-specific written constitutions to relabel diverse data, and trains a compact shared-encoder model to classify categories and extract spans. Its joint global-and-token cascade filters span outputs, while increasing model size yielded little recall gain.

3.1 Task Definition

  • The task covers 12 structured identifier categories and surfaces every identifier whose category is clear from context; spans exclude descriptive words and may overlap across categories.A value presented as a credit card number remains positive even if it fails a checksum, and partially masked numbers can also be retained for review.Source: §3.1 · pp. 3–4

3.2 Constitutions and Data

  • Category-specific constitutions specify what belongs in each category, required evidence, span boundaries, and common edge cases; GPT-5 Mini applies them to produce category and span labels.Samples are inspected during development and constitutions refined when recurring deviations appear; these annotations are called constitutional labels.Source: §3.2 · p. 4
  • The training corpus contains approximately 1.12M relabeled examples from 77 public datasets, including varied AI, security, and PII contexts.Evaluation uses relabeled AI4Privacy validation and Nemotron-PII test splits, with no evaluation examples used for training.Source: §3.2 · p. 4

3.3 Model

  • Joint training improved span-extraction recall over training the token classifier alone, and the cascade lets token extraction use a lower threshold while filtering likely overflags.Training takes 25 minutes on four NVIDIA RTX PRO 6000 GPUs.Source: §3.3 · p. 4
  • Training the same recipe on 307M-parameter mmBERT-base improved average recall by less than one percentage point while increasing memory and latency, so larger size yielded no significant performance gains.Source: §3.3 · p. 4
  • Two classifiers share one mmBERT-small encoder: a global classifier predicts input-level categories, while a token classifier scores spans across all 12 categories.The global classifier gates which token-classifier spans are retained, as shown in the architecture overview.Source: §3.3 · p. 4, Figure 2 · p. 5

4 Results

Across the two benchmarks, the analysis finds substantial label disagreement in selected categories, then compares detection recall, span extraction, and inference cost under the study’s evaluation settings.

4.1 Label Disagreement Analysis

  • AI4Privacy has 59–86% original-only positives in five categories, while Nemotron mostly agrees except for constitutional-only IP Address examples.In the AI4Privacy audit, two judges agreed with the constitutional labels on 68% and 72% of sampled disagreements; they agreed on 100% and 98% of sampled Nemotron IP additions. Recognition of less common international formats as sufficient context remains future work.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Detection Performance

  • At 0.1% WildChat FPR, the model has the highest recall in 11 of 14 comparisons under constitutional labels and 9 of 14 under original labels.It detects more shared positives across all eight AI4Privacy languages, and detects embedded IP addresses that GLiNER baselines largely miss.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Span Extraction

  • The model achieves higher test F1 in five of six benchmark–baseline pairs, including all three Nemotron comparisons.Span extraction uses exact-match micro F1 against constitutional labels on each benchmark’s test split.Source: §4.3 · p. 7, Table 4 · p. 8

4.4 Inference Efficiency

  • 2–10× higher throughput and 94–97% less peak GPU memory distinguish the model from the baselines.Measurements cover the full inference path, from tokenization to span decoding.Source: §4.4 · p. 8, Table 5 · p. 8

5 Discussion

At the same FPR limit as the baselines, the model flags genuine disclosures they miss, including multilingual examples and embedded IP addresses. The discussion also notes that broad coverage can include detections that are not useful to every application.

5 Discussion

  • At the same FPR limit as the baselines, the model correctly flags many genuine disclosures that they miss.On AI4Privacy, baselines often assign these identifiers the wrong category; on Nemotron, additional coverage primarily involves embedded IP addresses.Source: §5 · p. 8
  • The model recovers multilingual disclosures that baselines misclassify, including a Hindi tax identification number labeled as a bank routing code.German credit-card and French driver-license examples are also assigned incorrect categories by baselines.Source: §5 · p. 8
  • Category misrouting matters because guardrails may monitor phone numbers but block tax identification numbers.Source: §5 · p. 8
  • On Nemotron, additional coverage primarily involves embedded IP addresses, which NVIDIA GLiNER appears to omit after training on that dataset’s training split.Source: §5 · p. 8
  • Broad detections such as organizational phone numbers may not matter to every application, but retaining them lets applications adjust coverage without retraining a narrower detector.The paper gives OpenAI Privacy Filter’s private_phone definition as an example of narrower coverage.Source: §5 · p. 8

6 Conclusion

The paper concludes that a compact detector trained on public datasets relabeled under a single written definition can outperform larger baselines at lower cost. It also describes planned releases and future work to refine definitions and extend category coverage.

6 Conclusion

  • The compact detector, trained on public datasets relabeled under a single written definition of contextual detection, outperforms larger baselines at much lower cost.Source: §6 · p. 8
  • The authors plan to open-source the model, inference code, example constitutions, and sample relabeled data.They hope these artifacts support research on model sizes, architectures, and detection tasks for privacy guardrails.Source: §6 · p. 8
  • Future work will refine the constitutions and extend coverage to additional identifier categories.Source: §6 · p. 8

Limitations

The selected benchmarks and baselines offer comparable coverage for only 9 of the detector’s 12 structured categories; other traffic distributions may also require evaluation and recalibration.

Limitations

  • Only 9 of the detector’s 12 structured categories have comparable coverage in the selected benchmarks and baseline models.The remaining 3 require additional evaluation.Source: Limitations · p. 9
  • GPT-5 Mini may apply the constitutions incorrectly or inconsistently despite iterative review during their development.Source: Limitations · p. 9
  • Operating points are calibrated on WildChat and evaluated on AI4Privacy and Nemotron, so other traffic distributions may require evaluation and recalibration.Source: Limitations · p. 9

Ethical Considerations

The paper describes the detector as a component for flagging identifiers, not a standalone privacy system, and reports safeguards around its data use. It also identifies the four aspects covered by its example credit-card constitution.

Ethical Considerations

  • WildChat conversations are used only to calibrate aggregate false-positive rates, and raw content is not reproduced; qualitative examples come from synthetic benchmarks.The data come from public research datasets or synthetic sources, and WildChat users opted into anonymous research-data collection.Source: Ethical Considerations · p. 9
  • The detector assigns category scores and spans without generating text, but it can miss identifiers and should be only one part of a larger privacy system.Source: Ethical Considerations · p. 9
  • Generative AI assistants supported paper editing and code development, and the authors reviewed the work and remain responsible for it.Source: Ethical Considerations · p. 9

A Constitution Examples

B Training Configuration

The training configuration uses a jointly trained encoder and classifiers, while the following sections specify contextual matching, category mappings, and inference procedures.

B Training Configuration

  • The pretrained encoder and both classifiers are trained together for one epoch, with the final checkpoint used; each classifier outputs 12 independent category scores.An input or token may be assigned to multiple categories.Source: B Training Configuration · p. 12, Table B1 · p. 12

C Context Words

  • Context matching normalizes text with Unicode NFKC, lowercasing, and non-alphanumeric replacement, then checks complete word sequences from eight languages.Direct words are category-specific, while broad context words are shared across categories.Source: C Context Words · p. 12

D Category Mappings

  • Mapped labels are prompted together, and scores from multiple labels mapped to one category are combined by taking their maximum.The mappings cover benchmark and baseline labels; a dash marks categories deemed not comparable under the methodology.Source: D Category Mappings · p. 12

E Model Inference

  • Inference uses overlapping, model-specific input chunks; Privacy Filter processes whole inputs and derives category scores from maximum marginal BIOES-tag probabilities.Repeated chunk scores are merged by maximum, and repeated span candidates are deduplicated after offsets are mapped to the full input.Source: E Model Inference · p. 12

F Operating-Point Sensitivity

The section compares recall at two WildChat FPR limits and examines calibration based on labeled FPR versus a label-free flag rate.

F Operating-Point Sensitivity

  • Every detector and category is recalibrated independently at each WildChat FPR limit while benchmark inputs and model scores remain fixed.Table F1 and Table F2 report recall at 0.25% and 0.5% WildChat FPR, respectively.Source: F Operating-Point Sensitivity · p. 12, Table F1 · p. 15, Table F2 · p. 15
  • At 0.25% FPR, our model has the highest constitutional recall in 12 of 14 comparisons and original-label recall in 10 of 14; at 0.5%, the counts are 14 of 14 and 10 of 14.The tables report constitutional/original recall on the same benchmark examples, with thresholds independently calibrated at each stated limit.Source: Table F1 · p. 15, Table F2 · p. 15, F Operating-Point Sensitivity · p. 16
  • At a 0.5% flag rate, label-free calibration gives our model the highest constitutional recall in 12 of 14 comparisons and original-label recall in 9 of 14, versus 14 and 10 under labeled calibration.Treating every WildChat flag as a false positive makes this calibration conservative toward higher-recall detectors; at 0.1%, the budget is below the prevalence of categories such as Phone Number and no threshold satisfies it for several baselines.Source: F Operating-Point Sensitivity · p. 16

G Training Datasets

The training corpus draws examples from 77 sources, with two datasets contributing separately listed subsets.

G Training Datasets

  • The training corpus comprises examples from 77 sources, with TyDi QA and Hermes Function Calling each contributing two separately listed subsets.The listed sources include the corresponding TyDi QA and Hermes Function Calling entries.Source: G Training Datasets · pp. 16–17

Related papers

Loading 2610.22597v1…