Protegrity-PII Technical Report

1 Introduction

• AI agents handling workflows in healthcare, banking, and commerce can expand the aggregation surface for personal information.The introduction highlights risks including identity theft, fraud, and discrimination, alongside organizations’ need to use PII.Source: §1 · pp. 1–2

1.1 PII detection problem definition

• PII detection remains difficult to define because competing formulations differ on span boundaries, multiword entities, and what text qualifies as PII.The task also requires context-sensitive detection beyond pattern matching, while taxonomies vary across industries, jurisdictions, and cultures.Source: §1.1 · pp. 2–3
• A real-world detector must balance high recall, scale, latency, broad generalizability, and support for structured, semi-structured, and unstructured text.The authors advocate recall = 99.999% across all PII types as an arbitrary criterion for legal defensibility.Source: §1.1 · p. 3

1.2 Our contributions

• The paper presents and open-sources Protegrity-PII for fast, accurate PII detection across diverse contexts and data formats.The introduction describes the model as built to detect and sanitize PII in unstructured text and secure agentic workflows.Source: §1 · p. 2, §1.2 · p. 3
• The authors propose PII-SCOPE, a benchmark for systematically evaluating fine-grained entities in semistructured and unstructured text.They also report issues with public benchmarks and analyze practical trade-offs across PII detectors and benchmark datasets.Source: §1.2 · p. 3

2.2 Rule-based, learned, and hybrid approaches

• Figure 3 maps models and datasets over time; the text traces rule-based identifiers through sequence models, neural taggers, and transformer encoders.Early rule-based systems used templates and specialized knowledge; later methods included HMMs, conditional random fields, BiLSTM-CRF taggers, and pretrained Transformers.Source: Figure 3 · p. 5, §2.2 · pp. 5–6

2.1 PII detection, entity classification, and redaction

• PII workflows distinguish locating spans, assigning taxonomy-dependent entity labels, and removing, masking, or replacing them; NER overlaps only slightly and misses many identifiers.A public figure or city can be a named entity without being PII, while account, health insurance, and government IDs may be PII but absent from NER.Source: §2.1 · p. 5

2.2 Rule-based, learned, and hybrid approaches

• Later approaches include GLiNER’s natural-language entity descriptions, LLM-based detection, and hybrids combining deterministic regex detectors with learned or language-model components.The text notes that LLM-based efforts face output-alignment and inference-latency limits, while hybrids may run each component on every input or merge outputs by voting.Source: §2.2 · p. 6

2.3 Synthetic datasets and evaluation benchmarks

• Real-text PII corpora are scarce and access-controlled, while synthetic benchmarks vary in taxonomy and boundary conventions and often emphasize F1 over recall.The text notes that missed PII is more critical than false positives, so it argues evaluation should report recall.Source: §2.3 · pp. 6–7

2.4 Where this work differs

• Protegrity-PII uses a product-driven 69-type native taxonomy, a compact token-level classifier, and threshold calibration for precision under a high-recall floor, reporting typed and untyped metrics.The authors note that its taxonomy and label mappings reflect downstream product requirements.Source: §2.4 · p. 7

3.1 Formal problem statement

• PII classification takes free-form text as input and an entity mask as output, with each annotation specifying a span and its entity type; overlapping annotations are forbidden.Source: §3.1 · p. 7

3.2 Challenges with datasets

• Synthetic datasets can contain mode collapse, incorrect entity masks, and degenerate or incomplete text, while instruction-following limitations in LLM generators make them imperfect training data.Source: §3.2 · p. 7

3.3 PII-SCOPE

• PII-SCOPE is a public benchmark of 8,030 rows spanning varied fields, and Protegrity-PII is trained on a roughly 100× larger scaled version.Its scenarios include human resources, finance, healthcare, legal/government, retail, insurance, and information technology; synthetic generation also faces a chicken-and-egg challenge because generators must detect the PII they label.Source: §3.3 · p. 8

3.4 Overgeneration

• PII-SCOPE has the lowest overall overgeneration among the considered benchmarks, while SPY shows significant overgeneration that can unfairly penalize precision and F1.The proxy annotations aggregate five detectors, counting a missing mask entity when three detect it; regular expressions are also used for fixed-pattern entities.Source: §3.4 · p. 9, Figure 6 · p. 10

3.5 Richness

• Roughly 1.8× the entity density of the next densest dataset, PII-SCOPE also has the greatest richness on both entity density and type diversity.Richness refers to the number of entities and distinct entity types represented in dataset rows.Source: §3.5 · pp. 9–10, Table 1 · p. 10

4.1 Architecture

• The model uses a pretrained XLM-RoBERTa encoder, a linear classification head over token representations, and Viterbi decoding to produce entity masks [22].Source: §4.1 · p. 10

4.2 Token-level formulation

• Each token receives a categorical distribution over tags, with two tags per entity type plus one non-PII tag.The tag sequence is decoded into entity-mask triplets at inference time.Source: §4.2 · p. 11

4.3 BIO tagging

• BIO tagging represents non-PII tokens with O and each entity type with B-t and I-t tags.Source: §4.3 · p. 12

4.4 Training objective

• Training minimizes token-level multiclass cross-entropy, a surrogate not aligned with downstream metrics that cannot encode the preference for recall over precision.The model therefore relies on post-training decision-threshold calibration to achieve the desired behavior.Source: §4.4 · p. 12

4.5 Pruning and distillation

• About 97% of the full model’s recall is retained by a model roughly 25% smaller on disk that processes rows about 25% faster on an RTX 6000.The model is pruned using block-importance and activation-based criteria, then distilled from the unpruned teacher.Source: §4.5 · pp. 12–13

4.6 Model calibration

• Post-training calibration replaces the most-probable-tag rule with a user-calibrated threshold, injecting a preference for recall over precision without retraining.Source: §4.6 · p. 13

5.1 Benchmark models

• Comparators include open-weight classifiers, a promptable GLiNER model, rule-based Presidio, Rampart, and a frontier LLM reference.The frontier reference is Claude Fable 5; GLiNER accepts entity types as input, unlike models with a fixed native taxonomy.Source: §5.1 · pp. 13–14

5.2 Metrics

• Typed false negatives include misses and label confusions; untyped scoring counts only misses, allowing comparisons across native taxonomies.Typed results use the paper’s taxonomy, with mappings applied to each dataset as necessary.Source: §5 · p. 13, §5.2 · p. 15

5.2.1 Metric preference and justification

• Recall is prioritized because missed PII costs more than false positives, and it avoids penalizing correct models on overgenerated public benchmarks.The paper reports recall alongside precision and F1 for completeness.Source: §5.2.1 · p. 15

5.3 Benchmark results

• On PII-SCOPE’s maximal spanning entity sets, Protegrity-PII achieves superior strict typed F1 to every baseline.The comparison evaluates the maximal spanning set of entities across models’ native taxonomies.Source: §5.3 · p. 16, Figure 9 · p. 16
• Across considered datasets, Protegrity-PII dominates benchmark models on most metrics, especially recall and F1; it leads all metrics on AI4Privacy and Nemotron.The full benchmark results cover typed and untyped recall, precision, and F1.Source: §5.3 · p. 17, Table 4 · p. 18
• 3.79s and $0.0035 per million characters make Protegrity-PII the fastest and cheapest model considered, roughly 4× faster and cheaper than next-best self-hosted models.It is roughly 8× faster and cheaper than Presidio; Figure 11 excludes Rampart and caps its axes.Source: §5.3 · p. 19, Figure 11 · p. 19
• Training on Protegrity-PII’s curated dataset yields greater total out-of-distribution generalization across tested models than other benchmark datasets.The analysis fixes model architecture, trains independently on each dataset, and uses untyped recall for evaluation.Source: §5.3 · p. 20, Figure 12 · p. 20

6 Conclusion

• Protegrity-PII shows strong out-of-distribution generalization and detection rates across evaluated benchmarks, within its taxonomy and under taxonomy-agnostic evaluation.The paper also releases PII-SCOPE to support more rigorous, standardized benchmarking across diverse scenarios and entity types.Source: §6 · p. 20
• Extremely rare or unseen PII types and the quality and representativeness of training data may constrain performance.The authors recommend monitoring and updating the model according to the downstream use case.Source: §6 · p. 20

A Supplementary results

• Table 6 lists the top five confused entities for all models across all datasets.Source: Table 6 · p. 21
• Figure 13 compares typed recall under exact, containment, and overlap boundary criteria.Containment counts a predicted span that contains the ground-truth span; overlap counts one that overlaps it.Source: Figure 13 · p. 21
• Figures 14 and 15 break down model performance by common sensitive types and common languages, respectively.Figure 14 reports typed recall; Figure 15 reports untyped recall.Source: Figure 14 · p. 22, Figure 15 · p. 22

B Evaluation label mappings

• Table 7 links Appendix B.1–B.6 to the typed-recall computation.Source: Table 7 · p. 23
• Table 8 defines a model’s taxonomy coverage by whether at least one native label maps onto an entity type in the paper’s taxonomy.Source: Table 8 · p. 23
• Table 9 lists the five-label intersection of the native taxonomies of Protegrity-PII, AWS Comprehend, OpenAI Privacy Filter, Microsoft Presidio, and Rampart.Source: Table 9 · p. 23

• The supported-entity inventory includes identifiers, contact and location details, health and biometric data, and sensitive personal attributes.The listed labels include EMAIL, GEOCOORD, MEDICAL_RECORD_NUMBER, NEURAL_DATA, and POLITICAL_OPINION.Source: Table 10 · p. 23, §B.1 · p. 24
• The appendix identifies AI4Privacy label mappings in Table 11.Source: Table 11 · p. 24
• The appendix identifies Nemotron, Gretel, and SPY label mappings in Tables 12, 13, and 14.Source: Table 12 · p. 25, Table 13 · p. 26, Table 14 · p. 27

• Entity descriptions cover identity and account labels, including PASSPORT, PASSWORD, PHONE, PIN, and USERNAME.Source: Table 16 · p. 27, §B.2 · p. 28
• The descriptions include sensitive attributes such as POLITICAL_OPINION, RACE_ETHNICITY, RELIGIOUS_BELIEF, and SEXUAL_ORIENTATION.Source: §B.2 · p. 28
• The described labels also include address and tracking categories, from SECADDRESS and STREET to TRACKING_IDENTIFIER and ZIPCODE.Source: §B.2 · p. 28

• Table 17 covers AWS Comprehend output mappings to the evaluation taxonomy.Source: Table 17 · p. 28
• The listed labels include DATE_TIME, DATE, DRIVER_ID, DRIVERLICENSE, EMAIL, IP_ADDRESS, and IP.Source: §B.3 · p. 29
• Additional listed labels include ADDRESS, STREET, URL, USERNAME, VEHICLE_REGISTRATION, and ZIPCODE.Source: §B.3 · p. 29

B.4 OpenAI Privacy Filter

• OpenAI Privacy Filter mappings pair native labels with evaluation labels, including secret SSN, private_address STREET, and private_url URL.Source: Table 18 · p. 30, §B.4 · p. 31

B.5 Microsoft Presidio

• Microsoft Presidio mappings include labels such as UK_NHS, PERSON, US_PASSPORT, and US_SSN alongside evaluation-taxonomy labels.Source: Table 19 · p. 31, §B.5 · p. 32

B.6 Rampart

• Rampart mappings list variants including DRIVERS_LICENSE, GIVEN_NAME, SECONDARY_ADDRESS, and ZIP_CODE alongside taxonomy labels.Source: Table 20 · p. 32, §B.6 · pp. 33–34

C Redaction prompt

• The redaction prompt requires exact contiguous source substrings, one allowed label per occurrence, smallest complete spans, and structured output only.Source: C Redaction prompt · p. 36