PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Quality in RVL-CDIP: A Systematic Audit

Stefan Larson, Kevin Leach

arXiv:2610.32126v1cs.CVcs.CL

TL;DR

With RVL-CDIP scores near saturation, it is unclear whether benchmark gains measure document understanding or exploitation of dataset defects. This paper audits five defect axes and re-benchmarks models after joint remediation; the 13 retrained models are 0.9 to 2.9 points more accurate, and defect effects are not additive.

  • Problem

    Near-saturated RVL-CDIP accuracy may reflect document-classification ability or exploitation of benchmark defects, while OCR choices also make scores across papers non-comparable.

    Source: §1 · p. 2, §9 · p. 33

  • Method

    The paper audits five dataset and evaluation-protocol axes, then evaluates models after remediating the defects jointly.

    Source: §1 · p. 2

  • Results

    After all defects were remediated, the 13 retrained models were 0.9 to 2.9 points more accurate, and defect effects were not additive.

    Source: §14 · p. 59

  • Takeaways & Limitations

    Comparing validation and test duplication can distinguish corpus-assembly defects from partitioning defects and indicate whether re-splitting is a remedy.

    Source: §12.5 · p. 56

    The 35.2% duplication rate is a lower bound because detection recall is 82.9%; code-removal accuracy decreases also understate the codes’ contribution.

    Source: §13 · p. 57

Abstract

from arXiv · show

RVL-CDIP is the de facto standard benchmark for document classification, and reported accuracy on it is now 98.8%. In this paper, we audit RVL-CDIP for data quality issues and find that approximately 10.5% of its 400,000 documents have erroneous labels and that 35.2% of the test set has a duplicate or near-duplicate in the training set. Identification codes on the pages are sufficient to classify at 69% accuracy, and a state-of-the-art model loses 11 points without them. Image-only models trained on RVL-CDIP lose 16 to 28 points on documents from other sources, and the choice of OCR engine changes accuracy by 4.1 to 8.3 points. We find that nearly 16,000 RVL-CDIP documents contain sensitive personal data, and produce de-identified versions of these documents. We re-benchmark 25 models, and with all defects remediated, the 13 that we retrain are 0.9 to 2.9 points more accurate than in the original condition; the most accurate of them, LayoutLMv3, is 96.5% accurate. We release the corrected corpus, the annotations and the out-of-distribution data.

1 Introduction

RVL-CDIP scores are often treated as evidence of progress in document understanding, but the benchmark may reward learning its own defects. This audit examines those defects individually and together, including sensitive-data exposure and the consequences of remediation.

1 Introduction

  • The audit treats RVL-CDIP accuracy as a composite of classification ability and models’ ability to learn benchmark defects across five axes.The axes are label validity, train–test integrity, spurious visual cues, distributional breadth, and OCR evaluation protocol.Source: §1 · p. 2
  • The paper jointly remediates all five axes, tests whether defects interact, and releases annotations, corrected corpus variants, and evaluation data.Source: §1 · pp. 3–4
  • 15,956 documents contain sensitive personal data, and synthetic replacement leaves DiT predictions unchanged on 445 resumes.The authors run the subsequent audit on the de-identified corpus.Source: §1 · pp. 2–4
  • Approximately 10.5% of RVL-CDIP labels are erroneous, with category error rates ranging from 1.5% for resumes to 31.3% for letters.The audit measures errors against published category criteria absent from the original release.Source: §1 · p. 4
  • Approximately 35.2% of test documents have a near-duplicate in training, and the similar validation rate indicates leakage during corpus assembly rather than splitting.Source: §1 · p. 4
  • Identification codes appear on 92% of the corpus; code-only features achieve 69% accuracy, and removing codes costs a state-of-the-art model 11.0 points.Source: §1 · p. 4
  • On out-of-distribution documents, RVL-CDIP-trained image-only models lose 14 to 28 points, compared with 3 to 4 points for text models.Their ability to distinguish in-domain from out-of-domain inputs also approaches chance.Source: §1 · p. 4
  • Changing only the OCR engine changes accuracy by 4.1 to 8.3 points, exceeding the 0.19 to 0.82-point differences between successive LayoutLM generations.Source: §1 · p. 4

2 Background and Related Work

This section introduces RVL-CDIP’s source and benchmark use, surveys reported performance and evaluation practice, and reviews prior work on dataset defects and their assessment.

2 Background and Related Work

  • Section 2 describes RVL-CDIP and its source collection, then reviews prior work on label errors, duplication, shortcuts, out-of-distribution evaluation, and sensitive data.Source: §2 · pp. 4–5

2.1 RVL-CDIP and IIT-CDIP

  • RVL-CDIP contains 400,000 single-page grayscale images across 16 categories, sampled from the roughly seven-million-document IIT-CDIP litigation archive.The archive spans mainly scanned paper from one industry and was assembled for legal discovery, not machine-learning evaluation.Source: §2.1 · p. 5

Performance on RVL-CDIP

  • Reported accuracy rose from 88%–91% for early image-only models to 98.80% for LayoutLLM, while leading-model differences are often below one point.Twenty-nine of 34 surveyed papers evaluate classification only on RVL-CDIP or datasets derived from IIT-CDIP.Source: Performance on RVL-CDIP · pp. 6–7

2.3 Related Work

  • Related work documents label noise and train–test overlap, shortcut cues, missing document-classification OOD evaluations, and privacy risks in public corpora.Prior work on RVL-CDIP identification codes and black-box redaction also motivates this paper’s analyses and utility-preserving de-identification.Source: §2.3 · pp. 7–8

3 Sensitive Personal Data and De-Identification

The paper audits sensitive personal data in RVL-CDIP, replaces sensitive values with synthetic ones, and uses the de-identified corpus for later experiments. It finds 15,956 affected documents and reports that localized de-identification leaves classifier predictions unchanged on 445 resumes.

3 Sensitive Personal Data and De-Identification

  • The corpus was de-identified before the audit because the authors say they cannot responsibly release the data with sensitive personal information in place.The paper reports aggregate statistics and does not release the locations or values of sensitive entities.Source: §3 · pp. 8–9

3.1 Problems with Automated Detection

  • Four evaluated PII detectors have narrow coverage, and their document-level recall for Social Security numbers ranges from 0.70 to 0.97.Only Social Security number and age are supported by all four detectors; seven of the eleven recorded entity types are detected by none.Source: §3.1 · p. 9

3.2 Annotation

  • The authors hand-annotated sensitive entities because their locations are needed for redaction, unlike document-level detections.The annotation recorded bounding boxes so each entity’s position on the page could be used in subsequent redaction.Source: §3.1 · p. 9, §3.2 · p. 10

3.3 Prevalence of Sensitive Personal Data

  • 15,956 documents (4.0% of RVL-CDIP) contain sensitive personal data, including 2,342 US Social Security numbers.Dates and places of birth, home addresses, citizenship and nationality statuses, and marital, parental, and spousal statuses are among the most frequent entity types.Source: §3.3 · p. 10

3.4 De-Identification with Synthetic Replacement

  • Synthetic replacement preserves entity types and formats while substituting plausible values; redaction rectangles can remove answers needed for other document tasks.The authors use the replaced corpus as RVL-CDIP’s default distribution and run later experiments on it.Source: §3 · p. 9, §3.4 · pp. 12, 14
  • The replacement pipeline leaves some regions blank: four categories, covering 3.8% of annotated regions, are erased and refilled but not rewritten.The authors state that claims about replacement preserving answerability do not apply to those regions.Source: §3.4 · p. 14

3.5 Effect of De-Identification on Downstream Models

  • Synthetic replacement is slightly closer to originals than white redaction by cosine similarity for 61.6% of the 445 documents.Black redaction produces the least similar documents, and similarity decreases as the redacted area increases.Source: §3.5 · p. 14
  • Across three treatments, no document changed its predicted label on 445 resumes; de-identification therefore has almost no effect on classification.Mean absolute confidence changes were 0.0036 for black redaction, 0.0022 for white redaction, and 0.0024 for synthetic replacement.Source: §3.5 · pp. 14–15

4 Label Guidelines for RVL-CDIP Categories

The original RVL-CDIP release did not define its categories, so the authors developed operational criteria for auditing labels and annotating out-of-distribution data. Their review found heterogeneous categories and set conventions for ambiguous, multi-page, and non-English documents.

4 Label Guidelines for RVL-CDIP Categories

  • The authors wrote operational criteria for all 16 categories because the original release provided no definitions, criteria, or annotation rules.They began with a review of 16,000 training documents and revised the criteria during the exhaustive review of all 400,000 labels.Source: §4 · p. 15

4.1 Method

  • Most categories contain several distinct sub-types, making a category label less precise than its name suggests.The criteria address shared boundaries empirically; checks appeared in 8.8% of reviewed budget documents and 3.8% of invoice documents, so checks are assigned to budget.Source: §4.1 · pp. 15–16
  • The criteria exclude recurring documents with no semantic relation to their assigned category, including placeholder pages marked “IMAGE NOT AVAILABLE ONLINE.”All 54 such test pages were marked unknown in the review and removed from the fixed conditions.Source: §4.1 · p. 16

4.2 Scope Conventions

  • For documents plausibly belonging to two categories, the authors follow consistent corpus resolutions and mark unresolved cases as mixed.The benchmark’s single-label format cannot represent multiple valid categories.Source: §4.2 · p. 17
  • Most images show a document’s first page, while charts, tables, and presentation slides may come from any position.Source: §4.2 · p. 17
  • Category assignment follows document type regardless of language; legible examples include German, Dutch, French, Spanish, Portuguese, Italian, Japanese, Chinese, Arabic, and Hebrew.Most legible documents are in English.Source: §4.2 · p. 17

5 Axis I: Label Validity

An exhaustive review found that about one in ten RVL-CDIP labels is erroneous, with substantial differences across categories. Because label errors are frequent and systematic, automated detection methods that assume mostly correct labels cannot establish the error rate; the authors relied on human review.

5.1 Automated Error Detection

  • Automated label-error methods assume errors are rare, but systematic errors from unspecified category boundaries recur across all three splits and can receive high model confidence.The authors conclude that establishing the error rate requires human review.Source: §5.1 · p. 19

5.2 Annotation Protocol

  • Seven volunteer undergraduate researchers manually inspected all 400,000 documents using category criteria, with the project lead reviewing every “no” and “unsure” judgment.Documents marked “no” received corrected labels where possible, and those corrections form the remediated corpus.Source: §5.2 · p. 19

5.3 Findings

  • Approximately 10.5% of RVL-CDIP is erroneously labeled: 5.17% unknown, 3.84% wrong, and 1.49% mixed.The rate is 11.48% if unsure cases are counted as errors.Source: §5.3 · p. 20, Table 6 · p. 21
  • Error rates vary from 31.26% for letter to 1.48% for resume, and the letter–memo boundary is frequently unclear.Of relabeled letter documents, 83.9% were memos; relabeled invoice documents were 86.7% budgets, and relabeled scientific_publication documents were 91.8% news articles.Source: §5.3 · pp. 20–21

5.4 Sampled and Exhaustive Error Rates

  • The exhaustive training-split review found 8.97% errors without mixed documents and 10.45% with them, versus earlier sampled estimates of 8.1% plus 1.7% multiple-label cases.The authors conclude only that a sampled estimate can understate the error rate.Source: §5.4 · pp. 21–22

6 Axis II: Split Integrity

RVL-CDIP’s test split substantially overlaps training because both exact copies and shared-template documents recur across splits. The audit’s filter-and-refine method finds widespread overlap, with rates that vary by category and are nearly uncorrelated with label errors.

6.1 Definition of Duplicates

  • Template matches count as leakage because shared layout and structure expose the discriminative features needed for document-type classification.The two cases are copies differing in scan conditions and distinct instances of the same form or letter with different filled fields.Source: §6.1 · p. 22

6.2 Limitations of Perceptual Hashing

  • Perceptual hashing misses same-page copies affected by scan noise or distortion and cannot detect template matches that differ in filled content.Using image hashing for these cases would substantially undercount duplicates.Source: §6.2 · p. 22

6.3 Filter-and-Refine Detection

  • The filter-and-refine pipeline proposes candidate pairs with CLIP similarity, then checks image keypoint matches and recognized-text sequence similarity; its calibrated operating point has 100% precision and 82.9% recall.A pair is judged duplicate when sequence similarity is at least 0.2 and feature-match count is at least 100.Source: §6.3 · pp. 22–23

6.4 Findings

  • 35.2% of RVL-CDIP’s test split has a duplicate or near-duplicate counterpart in training.The reported duplication rate is a lower bound because the detection pipeline prioritizes precision and misses roughly one pair in six.Source: §6.3 · p. 23, §6.4 · p. 24, Table 7 · p. 25
  • 82.20% of specification test documents overlap training, while file_folder, handwritten, and email have overlap rates of 1.98%, 4.03%, and 6.96%, respectively.The paper links higher rates to recurring standardized templates and lower rates to structurally unique documents.Source: §6.4 · p. 25, Table 7 · p. 25

6.5 Duplication and Label Error

  • Label-error and duplication rates are almost uncorrelated across categories (Pearson r = −0.03; Spearman ρ = 0.14), but this category-level evidence is treated as a hypothesis about independence.The authors test the question at the document level in Section 11.Source: §6.5 · pp. 25–26

7 Axis III: Spurious Cues from Identification Codes

Identification codes stamped onto RVL-CDIP pages correlate with document categories and provide a strong predictive cue unrelated to document type. The audit measures their distribution and predictive signal, then detects and removes them to create code-free corpus versions.

7.1 Identification Codes

  • The codes are Bates numbers or comparable litigation identifiers, and their numbering correlates with document category, though the release does not establish why.The original file paths also encode the signal: filename features alone yield 68.8% accuracy, and adding the folder identifier raises it to 70.3%.Source: §7.1 · p. 26, §7.3 · pp. 28–29

7.2 Annotation and Distribution

  • 92.0% of the annotated RVL-CDIP sample contains codes, whose positions and content vary by category; 99.9% of handwritten codes begin with 5.The first character alone is nearly sufficient to identify one of the sixteen categories.Source: Figure 12 · p. 26, §7.2 · p. 27, Figure 14 · p. 27

7.3 Predictive Signal of Identification Codes

  • 69.2% accuracy on RVL-CDIP’s 16-way task comes from code features alone, rising to 69.9% with a convolutional network on the code region.The code-feature classifier exceeds zero-shot CLIP by 25.5 to 30.0 points on RVL-CDIP.Source: §7.3 · p. 28, Table 8 · p. 28

7.4 Detection and Removal at Scale

  • A YOLOv8 detector trained on 8,000 annotated RVL-CDIP documents achieves 97.1% F1 on RVL-CDIP and produces code-free versions of both corpora.It was applied to the remaining 388,800 RVL-CDIP documents; the paper evaluates the effect of removal in Section 10.Source: §7.4 · p. 29

8 Axis IV: Distributional Narrowness

RVL-CDIP covers old scanned documents from a narrow industry and does not establish how models handle contemporary, born-digital, or out-of-category documents. The paper constructs two evaluation sets for these distinct out-of-distribution questions and assesses abstention using confidence-based separability.

8.1 Distributional Coverage of RVL-CDIP

  • RVL-CDIP is limited to pre-2006 documents, mostly scans of physical paper, from a single industry and largely one litigation episode.Of 34 surveyed papers, 23 evaluate document classification only on RVL-CDIP.Source: §8.1 · p. 29

8.2 Out-of-Distribution Evaluation: RVL-CDIP-N and RVL-CDIP-O

  • RVL-CDIP-N contains 1,002 documents from different sources that fit RVL-CDIP categories, while RVL-CDIP-O contains 2,915 documents outside all sixteen categories.The former evaluates generalization under changed input distributions; the latter evaluates whether models can abstain on out-of-category documents.Source: §8.2 · p. 30, Figure 15 · p. 31, Figure 16 · p. 32

8.3 Measuring Abstention

  • Abstention is evaluated by how well confidence separates in-domain from out-of-domain documents, reported as AUC across thresholds.The N-O setting is the more realistic and harder comparison because neither set resembles the model’s training data.Source: §8.3 · pp. 32–33

9 Axis V: OCR Engine Choice

For text and layout models, “accuracy on RVL-CDIP” is not a single quantity because papers may use materially different OCR text. The audit finds substantial disagreement between OCR engines and inconsistent reporting of engine choice and quality.

9.1 Evidence from Prior Work

  • 6.6 points separated LayoutLMv2’s 88.7% with Tesseract from its authors’ 95.3% with Microsoft Azure OCR on the same architecture, weights, and test images.The comparison was not a controlled experiment.Source: §9.1 · p. 33

9.2 OCR Reporting in Prior Work

  • None of 23 surveyed papers reported RVL-CDIP OCR quality, and only UDoc varied engines, scoring 93.42%, 93.96%, and 94.10% with Tesseract, EasyOCR, and Google OCR.When papers do not name the engine, their reported results cannot be reproduced from model code and benchmark alone.Source: §9.2 · pp. 33–34

9.3 Agreement Between OCR Engines

  • Textract produced 84.6 million words versus Tesseract’s 69.3 million, 18% fewer; only 4,253 pages were empty for both engines.The comparison measures disagreement, not which engine is correct; RVL-CDIP has no ground-truth transcription.Source: §9.3 · p. 34
  • On handwritten pages, the engines shared only 9% of their words, while character distance and word overlap varied substantially across categories.For letters, memos, email, and resumes, median character differences were 8–11% and word overlap was 85–89%; for news articles and scientific publications, character distance was 64–71% and word overlap 49–71%.Source: §9.3 · p. 35
  • The authors release both OCR text layers with bounding boxes and per-document engine versions, allowing later work to fix an OCR input or test sensitivity across engines.Without shared text, a necessary model input is left to each user’s discretion, making comparisons unsound to that extent.Source: §9.3 · p. 35

10 Re-Benchmarking RVL-CDIP

This section re-benchmarks models under remediated and corrected-label conditions and compares their performance across OCR text layers, including tests of identification-code removal. It also examines whether trained models reliably abstain on contemporary out-of-scope documents.

10.1 Models

  • Twenty-five models are evaluated in seven groups, spanning fine-tuned image, text, and multimodal models, linear baselines, public checkpoints, and zero-shot classifiers.The zero-shot models were not trained on RVL-CDIP and serve as controls for changes across corpus conditions.Source: §10.1 · p. 36, §10.2 · p. 37

10.2 Corpus and Protocol Variants

  • Test-side remediation measures each defect’s effect on models trained on the original data, while retrained conditions measure performance after the same remediation is applied to training and validation.The fully remediated condition combines corrected labels, code removal in training and testing, and test pages without detected training duplicates.Source: §10.2 · p. 37

10.3 Joint Remediation

  • 2.7 to 4.5 points: retraining on corrected labels raises retrained-model accuracy over the original condition, whereas test-side label correction alone brings smaller gains or losses.Removing detected training duplicates lowers accuracy by 1.8 to 4.2 points for retrained models and public checkpoints on the remaining test pages. Under original labels, removing codes costs image models 3.1 to 4.8 points; much of the loss falls on 3,553 unknown, mixed, and unsure pages.Source: §10.3 · pp. 38–41
  • 0.9 to 2.9 points: retrained models are more accurate than in the original condition after full remediation.LayoutLMv3 reaches 96.5% accuracy; the TF-IDF-plus-SigLIP SVM reaches 96.0%, between LayoutLM at 95.2% and LayoutLMv3.Source: Table 12 · p. 39, Figure 19 · p. 42, §10.3 · pp. 42–43

10.4 Accuracy on RVL-CDIP-N

  • 10.0 to 15.7 points: corrected-label training raises accuracy for the seven image models on RVL-CDIP-N, from 61.4 to 71.2% to 73.1 to 86.9%.LayoutLMv3 rises from 83.7 to 97.2%; under corrected labels, the gap between RVL-CDIP test accuracy and RVL-CDIP-N nearly disappears for LayoutLM, LayoutLMv3, and the TF-IDF-plus-SigLIP SVM, but image models remain 9.0 to 19.7 points lower on RVL-CDIP-N.Source: Table 13 · p. 43, §10.4 · pp. 43–44

10.5 Abstention on RVL-CDIP-O

  • 0.594 to 0.673: in the N-O setting, the five CNNs’ abstention AUC falls near chance, while the best of thirteen trained models still accepts 59.5% of out-of-scope documents at a threshold retaining 95% of RVL-CDIP-N.The text models’ separation changes little between settings; zero-shot vision-language models show the reverse pattern, separating N-O better than T-O under maximum softmax probability.Source: §10.5 · pp. 45–46, Table 14 · p. 46

10.6 OCR Engine Comparison

  • Table 16 compares accuracy across OCR engines while holding architecture fixed, covering three text-only models and LayoutLM; every reported Textract–Tesseract difference is significant.Tesseract is the authors’ 4.1.1 run, and the table distinguishes original labels from corrected labels.Source: §10.6 · p. 47, Table 16 · p. 48
  • The comparison trained identical models on Tesseract 4.1.1 and AWS Textract text layers, then evaluated original and corrected labels and the RVL-CDIP-N test set.The text layers included tokens and bounding boxes, and the same architectures, hyperparameters, schedules, and seeds were used.Source: §10.6 · p. 48
  • Textract raises every model’s accuracy by 4.1 to 8.3 points, while BERT and RoBERTa differ by at most 0.16 points in any column.The OCR gains exceed the 0.19 to 0.82 points separating successive LayoutLM generations; label correction raises all scores but changes the OCR gaps by less than half a point.Source: §10.6 · p. 48
  • For TF-IDF SVM, Textract’s 9.1-point accuracy advantage remains with codes removed, while code removal at test time alone costs 0.1 to 0.2 points.On Textract, removing codes at test time costs pretrained models 1.4 points for BERT and 1.8 for RoBERTa under original labels, and 0.2 to 0.4 points for BERT, RoBERTa, and LayoutLM under corrected labels; codes can account for at most about a third of the OCR gain, though the direct Tesseract code-free retraining test was not run.Source: §10.6 · p. 49

11 Interaction Between Defects

The section tests whether audit defects interact and finds that their effects are not simply additive. Label errors and duplicates are associated, while label correction does not remove the OCR-engine effect or consistently change other defect effects.

11.1 Pairwise Associations

  • 28.1% versus 36.1%: test pages with label errors or unsure labels have training duplicates less often than correctly labeled pages (odds ratio 0.69).The relationship varies across categories, and the lower duplication rate among unknown, mixed, and unsure pages differs from the higher rate among relabeled pages.Source: §11.1 · pp. 49–50, Table 17 · p. 49
  • 5.88 and 5.75 points: BERT’s Textract advantage under original and corrected labels changes little after label correction, showing the label and OCR effects are nearly independent.Across the first three reported models, correcting labels reduces the OCR effect by at most 0.41 points; for LayoutLM it increases the effect by 0.45 points.Source: §11.1 · p. 50
  • 0.014 to 0.106 AUC: corrected-label training raises N-O separability in eight of nine model pairs, but does not make models more likely to reject out-of-scope documents.The increase comes from greater confidence on the in-domain side, including RVL-CDIP-O; LayoutLM is the exception in N-O separability.Source: §11.1 · p. 50, Table 18 · p. 50
  • At most about a third of pretrained text-model OCR advantage comes from reading identification codes better; the comparison does not account for any TF-IDF SVM OCR advantage.The direct comparison on Tesseract text was not run for the pretrained text models.Source: §11.1 · p. 51

11.2 Additivity of Confounds

  • 1.2 to 1.8 points: fully remediated accuracy exceeds the sum of the three single-remediation changes, so separate defect adjustments underestimate corrected-benchmark accuracy.Most of the difference comes from label correction plus de-duplication (0.9 to 1.4 points); their interaction reflects the higher share of wrong labels among pages without a training duplicate.Source: §11.2 · pp. 51–52, Table 19 · p. 51
  • 1.8 to 4.2 points: de-duplication lowers accuracy under original labels, versus 0.9 to 3.1 points under corrected labels, partly because pages without duplicates have more wrong labels.Pages without a training duplicate have a 13.1% wrong-label share, compared with 9.4% for pages with one.Source: §11.2 · pp. 51–52

12 Discussion

The discussion argues that RVL-CDIP scores combine task performance with several benchmark defects, whose effects cannot be corrected independently. It recommends using remediated variants with out-of-distribution evaluation while building a better-curated successor.

12.1 Interpreting RVL-CDIP Scores

  • Published RVL-CDIP accuracy combines classification ability with benchmark defects; because their effects differ in sign and are not additive, scores cannot be corrected by adjusting for each defect separately.The paper therefore measures accuracy with all defects remediated at once.Source: §12.1 · p. 52
  • 0.19 to 0.82 points: successive LayoutLM generations differ by less than the several-point effects of quantified defects, so RVL-CDIP does not establish which is more accurate.A 98.8% score also reflects matching labels that are approximately 10.5% erroneous; the paper reports that contemporary-document classification is not close to solved.Source: §12.1 · p. 52

12.2 Guidance for Users of RVL-CDIP

  • 1.48% for resume versus 31.26% for letter: category label error varies sharply; 1.5% of documents have multiple valid labels, and 35.2% of test pages have a training duplicate.The validation split has a similar duplication rate, so random repartitioning does not remove the duplicates.Source: §12.2 · p. 53

Identification codes.

  • A code-only classifier reaches 69% accuracy, while the corpus supports claims only about English-language scanned tobacco-industry documents produced before 2006; OCR engine choice also materially affects text and layout results.The public Donut checkpoint drops from 95.2% to 84.2% without codes; the paper advises naming the OCR engine and preferably using a published text layer.Source: Identification codes · p. 53
  • Sensitive personal data makes redistribution a concern, and many published results cannot be re-scored because fine-tuned classifiers are unavailable.The authors prefer de-identified versions and recommend releasing fine-tuned classifiers or, at minimum, per-test-document predictions.Source: Identification codes · pp. 53–54, Table 20 · p. 54

12.3 Should RVL-CDIP Still Be Used?

  • The authors recommend retaining RVL-CDIP in remediated form, pairing in-distribution results with out-of-distribution evaluation, and building a successor benchmark.The released variants address four defect axes directly, provide two text layers for the fifth, and are de-identified by default.Source: §12.3 · pp. 54–55

12.4 Requirements for a Successor Benchmark

  • A successor should provide published annotation guidance, measured low label error, multi-label annotations, verified splits, privacy checks, and documented provenance artifacts.These requirements address category interpretation, label quality, multiple valid document types, duplication, sensitive data, and collection-added shortcuts.Source: §12.4 · p. 55

12.5 Lessons for Benchmark Auditing

  • Benchmark audits should examine corpus provenance and defect rates across splits, use zero-shot controls, and distribute canonical converted inputs; exhaustive label review took seven annotators about two years.Automated label-error detection can miss pervasive errors when category boundaries were never specified; a successor should also include contemporary and born-digital documents, broader categories, multiple languages, and paired out-of-distribution data.Source: §12.5 · pp. 56–57

13 Limitations

The audit’s limitations include criteria-dependent label judgments, lower-bound estimates, restricted model and dataset coverage, and residual risks in de-identification. The authors also limit broader methodological conclusions to evidence from one corpus.

Label review

  • 10.5% is erroneous with respect to the authors’ published criteria; different teams may draw category boundaries differently, especially between letters and memos.The review took about two years, with substantial agreement (Fleiss’s kappa 0.74); the authors judge the error rate more likely understated than overstated.Source: §13 · p. 57

Lower bounds and definitions

  • 35.2% is a lower-bound duplication rate because duplicate detection has 82.9% recall, while code-removal accuracy decreases are also lower bounds.Preserving the positions and dimensions of erased code regions leaves predictive information; counting unsure labels as errors raises the label error rate from 10.51% to 11.48%.Source: §13 · pp. 57–58

Models and evaluation data

  • Model coverage is incomplete, pretraining exposure to test pages was not measured, and the two small out-of-distribution sets yield uncertain per-category estimates.The evaluated OCR comparison includes Tesseract and Textract only; a third engine could produce accuracy outside their observed range.Source: §13 · p. 58

De-identification

  • De-identification offers no formal privacy guarantee, and missed entities remain in the released data while realistic replacements make them harder to locate.Format matching can disclose approximate field width and format, so users should disable it when format itself is sensitive.Source: §13 · p. 58

Scope of the conclusions

  • Claims that confident learning fails under systematic error and provenance predicts defects remain hypotheses supported by a single corpus.The category-level analysis is ecological and based on sixteen aggregates; document-level analysis is preferred when both address the same question.Source: §13 · p. 58

14 Conclusion

The conclusion reports that correcting RVL-CDIP’s defects changes model accuracy and that their effects are not additive. It releases corrected and out-of-distribution resources for evaluating the benchmark.

14 Conclusion

  • 0.9 to 2.9 accuracy points: the 13 retrained models are more accurate with all defects remediated, and defect effects are not additive.The published differences among leading models are smaller than the measured defect effects, so those results do not establish which leading model is more accurate.Source: §14 · p. 59
  • Approximately 10.5% of labels are wrong, 35.2% of test pages have a training duplicate or near-duplicate, and identification-code-only classification reaches 69% accuracy.These are distinct benchmark defects reported in the conclusion.Source: §14 · p. 59
  • 16 to 28 points: image-only models lose this much accuracy on documents from other sources; OCR-engine choice changes text and layout model accuracy by 4.1 to 8.3 points.Nearly 16,000 documents contain sensitive personal data.Source: §14 · p. 59
  • The release includes de-identified corpora with and without identification codes, corrected labels, duplicate pairs, two OCR outputs, and out-of-distribution sets RVL-CDIP-N and RVL-CDIP-O.The authors recommend evaluating on corrected data and reporting out-of-distribution results alongside in-distribution results.Source: §14 · p. 59

Broader Impact Statement

The paper frames de-identification as its most important outcome because the public corpora contain personal information, and distributes de-identified versions by default. It also describes released artifacts and their provenance.

Broader Impact Statement

  • Personal records in widely mirrored litigation corpora concern people who did not consent; the authors make de-identified corpora the default release.They consider reducing this exposure the work’s most important outcome.Source: Broader Impact Statement · p. 59

A.1 Inventory

  • The release inventory includes de-identified images and OCR text, annotations, duplicate pairs, code detections, out-of-distribution sets, and trained classifiers.The annotations include original and corrected labels where available, mixed-category judgments, and annotator notes, with personal-data notes removed.Source: Broader Impact Statement · p. 59, §A.1 · p. 73

A.2 Release Policy for Sensitive Data

  • The de-identified release changes none of the predictions in the reported evaluation, and mean confidence changes are of order 10^-3.The authors withhold sensitive-entity locations, types, and values, reporting only aggregate statistics.Source: §A.2 · p. 74

A.3 Provenance and Licensing

  • Artifacts are keyed to original RVL-CDIP file identifiers, and annotations are versioned because some supersede figures reported in earlier papers.This lets readers join artifacts to the distributed corpus and identify which version produced a number.Source: §A.3 · p. 74

Appendix B. Datasets Used by the Surveyed Papers

These appendices document surveyed datasets and released materials, detail the de-identification pipeline, and evaluate confident learning and identification-code features. Confident learning finds only a minority of reviewed label errors under its default settings.

Appendix B. Datasets Used by the Surveyed Papers

  • The surveyed 34 papers all evaluate document classification on RVL-CDIP; Table B.1 also catalogs their datasets for other tasks and document-corpus pretraining.The table caption defines IE, VQA, layout analysis, table structure, OCR, and NLI abbreviations and marks unavailable datasets and absent pretraining.Source: Appendix B · p. 74, Table B.1 · p. 75

Appendix C. Sensitive Entity Types and Automated Detection; Appendix D. Sensitive Data in Datasets Derived from IIT-CDIP

  • Seven of eleven entity types are unsupported by all four tested PII detectors, and sensitive entities recur across datasets derived from shared IIT-CDIP source pages.For Social Security numbers, reported recall ranges from 0.70 for Azure to 0.97 for Presidio; the appendix counts affected documents across five derived datasets.Source: Appendix C · p. 76, Table C.2 · p. 77, Appendix D · p. 78

Appendix E. The De-Identification Pipeline; E.1 Replacement Values; E.2 Typeface Matching

  • Replacement values are consistent across repeated regions on a page, and doc-redact matches replacement typefaces using nearby measurable words when needed.The pipeline links repeated address parts and related marital-status or child-count values; typeface matching ranks rendered candidates by chamfer distance, with fallback faces when no word can be measured.Source: Appendix E · p. 79, §E.1 · p. 79, §E.2 · pp. 79, 81

E.3 Checking Annotations Before Rendering; E.4 Encoding and Use of OCR

  • Before rendering, OCR text is checked against annotated entity categories; released pages are JPEGs re-encoded from quality-75 copies to quality 95.Textract supplies words and boxes for replacements, but both Textract and Tesseract are separately run over replaced pages for model evaluation.Source: §E.3 · p. 81, §E.4 · p. 82

Appendix F. Confident Learning on RVL-CDIP

  • Less than a quarter of reviewed wrong labels are found by confident learning with default settings, which underestimates the test-split error rate by a factor of two to six.Across models, the method estimates 2.0–6.0% wrong labels against 11.8% in the review; Table F.1 reports estimates and flag precision and recall.Source: Appendix F · pp. 82–83, Table F.1 · p. 83

Appendix G. Original and Corrected Labels

  • Table G.1 cross-tabulates original labels by row against corrected labels by column for 15,366 relabeled documents across all splits.Documents judged unknown, with no corrected label, are excluded.Source: Appendix G · p. 84, Table G.1 · p. 85

Appendix H. Identification Codes: Leading-Digit Distribution

  • The leading-digit table reports character frequencies by category as row percentages, omitting non-digit leading characters; the figure maps detected-code positions by category.The number beneath each figure panel is the share of category pages with at least one detected code.Source: Appendix H · p. 85, Table H.1 · p. 86, Figure H.1 · p. 86

Appendix I. Identification Codes: Features of the Code-Only Classifier

  • The code-only classifier combines code locations and transcriptions with features encoding string patterns, nearby numbers, and OCR legibility.Its 330 RVL-CDIP features include whether a code is absent; label-dependent features are computed without using a document’s own training label.Source: Appendix I · p. 87, Table I.1 · p. 88

Appendix J. Identification Codes: Supplementary Results for the Code-Only Classifier

The supplementary results show that identification-code features and file metadata predict RVL-CDIP categories well, while code formats and predictive accuracy vary substantially across categories. Per-condition analyses also detail how model accuracy changes with corrected labels, code removal, OCR and evaluation set.

Appendix J. Identification Codes: Supplementary Results for the Code-Only Classifier

  • 70.3% accuracy on the RVL-CDIP test split results when folder-name target encodings are added to file-name features, versus 68.8% without them.The folder name alone yields 41.4%; nearest-number features may exploit near-duplicates and multi-page documents split across training and test.Source: Appendix J · p. 89
  • 95.9% email recall, 94.0% handwritten recall, 91.1% resume recall and 86.0% specification recall contrast with 37.3% for memo and 36.4% for letter.Category-specific code formats help explain the variation: 98% of handwritten codes use a two-part format, while 47% of resumes have no code.Source: Appendix J · p. 89
  • 69.9% accuracy is reached on RVL-CDIP when code features are combined with the convolutional network, with code content providing the largest gains.Code format and the first one to five characters raise accuracy to 58.4%; nearest-number features add 8.1 points to 66.5%.Source: Appendix J · p. 89, Figure J.1 · p. 90

Appendix K. Joint Remediation: Per-Condition Detail

  • 0.3 to 1.8 points more accuracy is observed for zero-shot models on pages without codes, except Jev with the page image, which is 0.6 points less accurate.Figure K.1 displays paired training variants; Figure K.2 shows test accuracy lost when codes are removed.Source: Appendix K · p. 91, Figure K.1 · p. 92, Figure K.2 · p. 93
  • With corrected labels, the SVM using TF-IDF text and SigLIP embeddings reaches 97.1% on RVL-CDIP and 97.3% on RVL-CDIP-N.On RVL-CDIP-N, the three text-image fusions score 91.9%, 95.6% and 97.3%; text-only TF-IDF scores 85.7%, and image-only ResNet-50, CLIP and SigLIP score 62.1%, 88.3% and 95.6%.Source: Table K.1 · p. 92, Appendix K · p. 93, Table K.2 · p. 94
  • 6.2 points more accuracy comes from giving Jev the page image rather than OCR text under corrected test labels, with handwritten recall rising from 3% to 49%.The two inputs produce the same prediction on 84.1% of test pages; image input also raises file_folder accuracy from 63% to 83%.Source: Appendix K · p. 94
  • 1.6 to 9.6 points is the increase in macro accuracy for every retrained model on RVL-CDIP-N after training on corrected labels.Macro accuracy exceeds micro accuracy for 13 of 15 trained models and public checkpoints under original labels, by up to 7.2 points for Donut and 6.3 for LayoutLMv3.Source: Appendix K · p. 95, Table K.5 · p. 96

Appendix L. Out-of-Distribution Supplementary Results

The appendix examines out-of-distribution detection across confidence scores, models, and prompts, including how models reject out-of-scope documents and whether results change when unparseable responses are excluded. It also reports the effects of corpus-grounded definitions and documents the models’ fine-tuning setup.

Appendix L. Out-of-Distribution Supplementary Results

  • Under every alternative score, CNNs separate N-O worse than text and multimodal models, while corrected-label retraining raises N-O AUC in eight of nine pairs.For zero-shot models, GEN is best and energy worst in N-O; at the 95% threshold, LayoutLM’s T-O acceptance of RVL-CDIP-O ranges from 58.1% to 83.4% across scores.Source: Appendix L · pp. 97–98, Table L.2 · p. 98, Table L.3 · p. 99
  • Removing identification codes lowers T-O AUC by 2.3 to 8.1 points for image models trained with original labels and 1.0 to 3.3 points with corrected labels.For the same architectures trained without codes, AUC changes by −0.9 to +0.7 points; at a fixed operating point, six documents without OCR text do not materially change results.Source: Appendix L · p. 99, Figure L.1 · p. 100

Appendix M. Zero-Shot Classification Prompts

  • The three zero-shot prompts use category names alone, corpus-grounded definitions, or generic definitions to distinguish benchmark annotation conventions from general category knowledge.Each prompt uses the same category list and response structure; the corpus-grounded definitions explicitly describe subtypes included by RVL-CDIP annotators.Source: Appendix M · p. 100

Rejection Variant for Out-of-Scope Detection

  • Out-of-scope rejection prompts add a seventeenth none_of_the_above option and instruct models not to use it merely to express uncertainty between categories.Explicit and out-of-vocabulary labels count as rejection, while unparseable replies count as not rejected; pages without OCR text are excluded from denominators.Source: Rejection Variant for Out-of-Scope… · pp. 103–104

Input of the Decision Models

  • The decision models select the highest-probability option from sixteen category names without definitions, using OCR text except that Jev receives the page image.Jev’s image input uses the de-identified released corpus, with out-of-range page dimensions rescaled to the API’s accepted range.Source: Input of the Decision Models · p. 104

Appendix N. Prompt Sensitivity: Statistical Analysis

  • Prompt comparisons use paired McNemar tests and Cochran’s Q, with Holm–Bonferroni correction and Wilson intervals; RVL-CDIP-N’s smaller sample makes null results weak evidence of no effect.Because accuracy is about twenty points higher on RVL-CDIP-N, cross-dataset prompt effects are compared on the log-odds scale.Source: Appendix N · p. 105

Transfer of the Corpus-Grounded Prompt

  • Prompt 2 removes 19.1% of Maverick’s and 16.9% of gpt-oss-20b’s RVL-CDIP errors, but Qwen’s 3.4% error reduction becomes a 26.0% increase on RVL-CDIP-N.Qwen’s reversal has odds ratio 0.22 (95% CI [0.12, 0.43]); Prompt 2 encodes corpus-specific annotation conventions, while generic Prompt 3 is statistically indistinguishable from no definitions for Maverick and gpt-oss-20b.Source: Transfer of the Corpus-Grounded Prompt · pp. 105–107, Table N.3 · p. 106

Sensitivity to Unparseable Responses

  • Unparseable replies range from 9 documents for gpt-oss-20b to 2,771 for Mistral 7B, but complete-case analysis leaves comparison signs and significance verdicts unchanged.The authors count unparseable responses as errors in the main analysis, noting that this conflates classification accuracy with instruction-following.Source: Sensitivity to Unparseable Responses · p. 107, Table N.4 · p. 107
  • Removing unparseable responses changed no comparison’s sign or significance; Qwen’s Prompt 1–Prompt 2 contrast fell from +1.28 to +0.79 pp, while Mistral 7B’s preference widened from −2.40 to −2.90 pp.Mistral 7B’s accuracy increased by roughly three points for every prompt, but its prompt ordering remained unchanged.Source: Sensitivity to Unparseable Responses · p. 108

Out-of-Scope Detection on RVL-CDIP-O

  • 37.2%–64.2% of documents were rejected by gpt-oss-20b and Qwen 3 32B, versus 3.1%–20.1% by Llama 4 Maverick and Mistral 7B; Prompt 3 yielded the highest detection for all four.Every omnibus test rejected, and all twelve pairwise contrasts remained significant after correction; without false-rejection measurements, detection rates do not establish separation of out-of-scope from in-distribution documents.Source: Out-of-Scope Detection on RVL-CDIP-O · p. 108, Table N.6 · p. 109
  • Llama 4 Maverick and gpt-oss-20b used only explicit rejections, while Mistral 7B often assigned its own labels and Qwen 3 32B used both rejection routes.Mistral’s invented labels accounted for 53–64% of its detections; unparseable replies occurred only for Mistral 7B and Llama 4 Maverick, and counting them as detections left the model grouping unchanged.Source: Out-of-Scope Detection on RVL-CDIP-O · pp. 108–109, Table N.7 · p. 109

Appendix O. Models and Training Details

  • The fine-tuned models used publication or prior-work settings, with conditions differing only in label set and pages; image models used grayscale 224-by-224 inputs, while text and layout models used Textract inputs.BERT and RoBERTa used text truncated to 512 tokens; LayoutLM used Textract words and boxes, and LayoutLMv3 added the page image.Source: Appendix O · pp. 109–110, Table O.1 · p. 110

Related papers

Loading 2610.32126v1…