PaperlayerView exact revision on arXiv

Source-linked AI summary

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

Vishnu Prasad Vijaya Kumar, Santhosh Venkatesh, Ivan P. Yamshchikov

arXiv:2609.02207v1cs.CVcs.CL

TL;DR

Document-image PII redaction remains difficult to evaluate because OCR errors, layout, and visual noise can leave a page unsafe when one identifier is missed. LeakageBench addresses this gap with a 500-page, 11,954-annotation benchmark and shared evaluation across OCR-dependent and OCR-free systems. Tool assistance improves GPT-5.5 localization, but critical page-level leakage remains high.

  • Problem

    Existing PII benchmarks are mostly text-centric and do not measure spatially grounded, page-level release safety for document-image redaction.

  • Method

    LeakageBench evaluates OCR-dependent detectors and OCR-free vision-language models through a shared image-space interface using entity-level F1 and document-level leakage metrics.

  • Results

    Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but GPT-5.5 with Code Interpreter still has DocLeakcrit = 0.968.

  • Takeaways & Limitations

    Stronger detection and tool assistance improve localization without making most pages safe for release.

  • Takeaways & Limitations

    LeakageBench is a focused challenge set from public business documents and does not represent all redaction settings; error attribution is also limited by missing artifacts.

Abstract

from arXiv · show

Real-world personally identifiable information (PII) redaction often operates on document images---scans, screenshots, and PDF renderings---where OCR errors, layout structure, and visual noise determine whether sensitive information is actually removed. Existing PII benchmarks are mostly text-centric and do not measure document-level redaction risk: a page remains unsafe if even one identifier is missed. We introduce LeakageBench, a challenge set of 500 document images with 11,954 GDPR-aligned PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces. We evaluate generic OCR pipelines, commercial and task-adapted OCR-dependent detectors, and OCR-free vision-language models using entity-level F1, group-wise leakage, and document-level leakage metrics. Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but critical page-level leakage remains 0.968. These results show that stronger detection and tool assistance improve localization without making most pages safe for release. LeakageBench provides a diagnostic benchmark for high-recall, spatially grounded PII redaction in document images.

1 Introduction

LeakageBench frames document-image PII redaction as a release-safety problem, where OCR, layout, repetition, and visual noise make missed identifiers consequential. It combines entity-level quality with document-level leakage evaluation and diagnoses why systems remain unsafe.

  • Motivation: Text-centric PII benchmarks and key-field document benchmarks do not measure whether any identifier remains visible on a page.Document-image redaction requires both entity localization and residual page-level leakage assessment.
  • Motivation: A single false negative can make a page unsafe, whereas false positives unnecessarily obscure useful content.Document-level leakage therefore complements, rather than replaces, localization and typing F1.
  • Benchmark: LeakageBench contains 500 document-page images and 11,954 localized PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces.The benchmark uses a GDPR-aligned schema and a shared image-space interface.
  • Contributions: The benchmark reports release-oriented document-level leakage alongside entity-level F1 and diagnoses failures across coverage, OCR, detection, interface validity, and spatial grounding.These diagnostics apply to both OCR-dependent and OCR-free systems.

2 Related Work

Prior work covers text-based PII detection, document understanding, document models, and privacy leakage, but does not combine exhaustive spatial PII localization with a page-level release criterion.

  • Text-based PII detection: Text-based PII benchmarks establish span-level detection and anonymization but exclude document-image artifacts, spatial localization, and page-level release safety.Their evaluation is centered on text rather than image-grounded redaction.
  • Document-image understanding: Document-understanding benchmarks evaluate forms, receipts, OCR, question answering, and selected business-document fields rather than exhaustive PII localization and residual leakage.Their task-specific extraction goals differ from release-oriented redaction evaluation.
  • Document models: OCR-based models combine recognized text with layout, while OCR-free models jointly learn visual reading and spatial localization.LeakageBench evaluates both through a common image-space prediction interface.
  • Privacy evaluation: Existing leakage-oriented privacy studies examine residual exposure, probing, or downstream inference but lack a localized document-image benchmark with a page-level release criterion.LeakageBench supplies that image-grounded evaluation setting.

3 Task and Evaluation Protocol

The task requires page-local, spatially grounded identification and typing of every visible PII region, evaluated with one-to-one IoU matching, entity-level F1, and worst-case page leakage.

  • Task: Given a page image, systems identify every visible PII region, assign a schema type, and optionally extract its value.Gold instances contain image-space boxes, fine-grained types, risk groups, and optional verbatim values.
  • Task: A page remains unsafe when even one in-scope identifier is missed, and inference provides no cross-page context.DocLeak is therefore computed over independently processed pages.
  • Prediction format: Each prediction contains a pixel bounding box, PII type, optional value string, and optional confidence score in a unified interface.The format supports both OCR-assisted pipelines and OCR-free vision-language models.
  • Matching: Predictions and gold instances are greedily matched one-to-one by descending IoU, with typed matches additionally requiring the correct PII type.Value correctness is assessed only for matched pairs when values are available.
  • Metrics: Headline DocLeak uses type-aware matching, while diagnostic attribution also counts correctly localized but mistyped boxes as spatially masked.DocLeak is reported with F1loc and F1type because it does not penalize extra predictions.
  • Metrics: Entity-level metrics measure average detection quality, whereas document-level leakage measures whether any PII remains unmatched on a page.F1loc uses type-agnostic matching, while F1type requires IoU ≥τ and the correct type, with τ = 0.75 unless stated otherwise.

4 Dataset: Collection and Composition

LeakageBench is a focused 500-page challenge set of public operational-style business documents, designed to stress-test PII redaction across heterogeneous layouts, identifier groups, and dense or degraded pages.

  • Collection: LeakageBench samples 500 document-page images from public operational-style business documents rather than representing all redaction settings.Its scope emphasizes heterogeneous layouts, dense tables, scanned correspondence, mixed acquisition channels, and visual degradation.
  • Sources: Source materials include archived heterogeneous pages, table-heavy invoices and receipts, and free-form emails, letters, memos, and scanned correspondence.These sources place identifiers across tables, narrative text, signatures, headers, and footers.
  • Schema: Linkage keys such as policy numbers, case IDs, and customer IDs are in scope because they can resolve to people through common workflows.The benchmark treats identifiability as extending beyond strings that directly resemble names.
  • Schema: The schema reports direct identification, lookup/linkage, and contextual inference groups in addition to fine-grained labels.Contextual fields capture re-identification surfaces that amplify risk when combined with direct identifiers or linkage keys.
  • Composition: 11,954 PII instances are annotated across 500 pages, averaging 23.90 instances per page, including 4,315 critical Direct and Linkage identifiers averaging 8.63 per page.Pages with many identifiers create disproportionate pressure because one missed identifier causes leakage.
  • Annotation: Annotations use tight visible-span boxes and verbatim value transcription, with independent annotation and adjudication procedures for quality control.The benchmark also reports fine-grained labels and aggregated Direct, Linkage, and Contextual groups.

5 Experimental Setup

The evaluation compares OCR-dependent and OCR-free systems through a shared image-space redaction protocol, using localization, leakage, schema, and validity measures. It also diagnoses how document-image constraints and unsupported schema types affect redaction assessment.

  • Evaluation protocol: All systems are converted to a common prediction format and evaluated with entity-level localization, localization-plus-typing, document-level leakage, schema coverage, and supported-type leakage.No system is trained or tuned on LeakageBench.
  • OCR-dependent baselines: OCR-dependent systems extract text and word boxes, detect typed PII spans, and merge corresponding boxes into image-space redaction regions.The evaluated family includes a generic Presidio pipeline and stronger commercial or task-adapted detectors.
  • Benchmark diagnostics: The benchmark reports spatial PII density for all annotated fields and for critical Direct+Linkage identifiers.Figure 2 aggregates identifier overlap across page regions, while Table 2 reports source composition and label volume.
  • OCR-free vision-language baselines: OCR-free vision-language models receive page images directly and return PII types, values, confidence scores, and bounding boxes.Qwen3-VL-32B and InternVL3-38B are evaluated under the same localization-and-typing protocol.
  • Evaluation protocol: OCR-free systems have schema coverage 1.0 by configuration because the full LeakageBench taxonomy is supplied in the prompt.This allowance does not imply reliable detection; empirical failures are measured separately.
  • Scope: LayoutLM-style trained baselines are excluded because the challenge set requires off-the-shelf systems that return PII types with image-space boxes.Such models typically require supervised fine-tuning, task-specific heads, OCR-token alignment, and a train/dev/test split.

6 Results

LeakageBench evaluates document-image PII redaction with localization, typing, schema, and page-level leakage metrics. Stronger systems improve entity-level performance, but critical page-level leakage remains high because coverage, OCR, detection, and spatial grounding failures persist.

  • Overall safety picture: DocLeakcrit remains 0.968 for both Amazon Comprehend PII and GPT-5.5 with Code Interpreter despite their stronger detection results.Amazon Comprehend PII achieves F1loc = 0.304, while GPT-5.5 with Code Interpreter achieves F1type = 0.119.
  • Leakage severity: Critical leakage is severe: leaking pages retain 5.3–8.7 critical identifiers on average, with the lowest-severity system missing approximately 66% across applicable pages.For that system, the median is 4 missed identifiers and P90 is 11.
  • Schema coverage: Restricted systems cover only 20.8%–25.0% of benchmark types, while full-schema systems still show near-total Linkage leakage.Restricted taxonomies miss most Linkage keys, and dense tables, sparse lookup keys, OCR fragmentation, and layout variation remain difficult even with full label coverage.
  • Supported-schema leakage: Supported-schema evaluation still finds leakage on more than 94% of Direct pages and more than 96% of Linkage pages for full-schema OCR-dependent systems.Residual exposure reflects OCR errors, repeated identifiers, span fragmentation, and box reconstruction, not only taxonomy mismatches.
  • OCR-dependent failures: Box and alignment failures account for 45.9–84.5% of attributable leaks in seven of nine OCR-dependent systems, while detector misses account for 15.5–48.8%.Textract misses 623/4,280 value-bearing critical annotations (14.6%), compared with 765/4,280 (17.9%) for Tesseract.
  • Tool-assisted localization: GPT-5.5 Code Interpreter raises F1loc from 0.090 to 0.249 and F1type from 0.050 to 0.119, but Linkage leakage remains 0.964.DocLeakcrit falls from 0.990 to 0.968, so tool-assisted spatial grounding improves without providing release-level safety.

Ethical, Legal, and Release Protocol

LeakageBench is intended for defensive redaction-safety research, but releasing real document images with fine-grained PII annotations creates privacy and dual-use risks. Access is therefore governed by a privacy- and license-aware Data Use Agreement, and the benchmark does not define legal compliance.

  • Combining real document images with fine-grained PII annotations introduces privacy and dual-use risks.
  • The document images and annotations will be available for research use under a privacy- and license-aware Data Use Agreement.
  • LeakageBench is intended for evaluating privacy-preserving redaction, not re-identification or PII mining.
  • Its GDPR-aligned schema supports comparative evaluation but does not define legal compliance or prescribe universal redaction requirements across jurisdictions or deployments.

Limitations

LeakageBench has focused coverage and page-local evaluation, while its conservative matching and incomplete artifacts constrain interpretation of results. Its annotation protocol also defines a safety-driven, schema-specific scope for visible PII.

  • Dataset scope: LeakageBench is a 500-page challenge set from public business documents, not a representative sample of all redaction settings.
  • Dataset scope: The benchmark excludes heavily handwritten, medical, legal-discovery, multilingual, and non-U.S. documents.
  • Evaluation scope: Evaluation is page-local, provides no cross-page context, and does not measure multi-page contextual inference.
  • Schema scope: Direct identifiers and linkage keys are primary release-safety targets, whereas masking contextual identifiers depends on applicable policy.
  • Metric limitations: DocLeak uses conservative type-aware matching at IoU τ = 0.75, so a correctly localized but wrongly typed box counts as a leak.
  • Metric limitations: Axis-aligned boxes and IoU do not directly measure pixel coverage for skewed, handwritten, or irregular regions.
  • Metric limitations: The benchmark penalizes extra predictions through F1 but defines no document-level over-redaction or utility metric.
  • Attribution limitations: Stored artifacts prevent fully separating box and alignment errors, and OCR-free results reflect configurations whose API behavior may change.

B Baseline Reproducibility Details

Baseline evaluations convert diverse systems into a shared image-space prediction format and apply fixed schema, validation, and scoring rules. OCR-dependent pipelines recover boxes from OCR spans, while OCR-free models emit normalized boxes directly from page images.

  • Common interface: All baseline outputs use the same format: PII type, optional value, confidence score, and image-space bounding box.
  • OCR-dependent conversion: OCR-dependent systems detect PII over recognized text, align spans to OCR word boxes, and use minimal enclosing rectangles for localization.
  • OCR-dependent conversion: Unaligned OCR-dependent detections are discarded before scoring.
  • Schema mapping: Predictions are mapped to the LeakageBench schema; under full-schema evaluation, unsupported gold types remain in scope and count as misses.
  • OCR-free conversion: OCR-free VLMs receive page images directly and output normalized boxes converted to pixel coordinates, with the full benchmark schema supplied in the prompt.
  • Validation: Malformed outputs, unknown labels, invalid boxes, and out-of-bounds boxes are scored as empty prediction sets without model-specific repair.
  • Prompting: The shared prompt header requires detecting and tightly localizing all visible PII in business document images using allowed types and confidence scores.

G IoU Threshold Sweep

The IoU threshold sweep tests how changing box-tightness requirements affects entity-level and document-level evaluation. Lower thresholds relax alignment requirements, but document-level leakage remains high across thresholds.

  • Threshold design: The sweep evaluates IoU thresholds τ ∈ {0.50, 0.75, 0.95}, from relaxed to tighter box-alignment requirements.
  • Results: Entity-level F1 changes substantially with the IoU threshold, whereas document-level leakage remains high across thresholds.

I Additional Error Diagnostics

The additional diagnostics characterize leakage severity, attribute OCR-dependent failures, measure OCR recoverability, and document evaluation settings across privacy groups, thresholds, sources, and entity types.

  • Diagnostic breakdown: Tables 13–15 separate leakage severity, OCR-dependent attribution, and detector-independent OCR recoverability.Table 13 uses the headline type-aware matcher; Table 14 uses type-agnostic matching; Table 15 uses gold values and stored OCR text.
  • OCR recoverability: Table 15 measures OCR recoverability for 4,280 value-bearing critical annotations.The 4,315-annotation total excludes 35 annotations lacking a verbatim gold value; partial denotes unrecoverable values with OCR-window similarity ≥0.8.
  • Evaluation settings: Table 9 reports conditional document leakage for each privacy group at IoU τ = 0.75 under full-schema scoring.Pages are included when they contain at least one ground-truth identifier from the relevant group; unsupported types and invalid outputs count as misses.
  • Evaluation settings: Table 10 sweeps IoU thresholds, with lower values relaxing box alignment and higher values requiring tighter localization.Shaded metric cells mark the main-paper threshold τ = 0.75.
  • Dataset diagnostics: Tables 11 and 12 report corpus, entity-category, and entity-type statistics by source.Categories include Direct Identifiers, Contextual Identifiers, Linkage Keys, Signature, and Operational References; casing is standardized to Title Case while acronyms remain uppercase.
  • Leakage severity: Table 13 summarizes missed Direct and Linkage identifier severity using DocLeak, missed counts, and distribution statistics at IoU τ = 0.75.Mean, median, and P90 are computed over leaking pages, while mean proportion uses all pages containing critical identifiers; bold marks the best value within each system family.
  • Failure attribution: Table 14 attributes leaked Direct and Linkage identifiers in OCR-dependent systems using type-agnostic matching at IoU τ = 0.75.Exclusive failure shares exclude value-unavailable instances from attribution percentages, OCR partial is secondary, and dashes indicate unavailable vendor OCR.
Loading 2609.02207v1…