PaperlayerView exact revision on arXiv

Source-linked AI summary

HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries

Shangxuan Tian, Yanhui Chen, Carlos Queiroz

arXiv:2608.21792v1cs.AIcs.CVcs.IRcs.LG

TL;DR

Regulated document classification lacks labels at cold start, must preserve data residency, and faces costly review under taxonomy drift. HIRA combines training-free multimodal retrieval, local LLM verification, and selective human feedback, reaching 0.9423 Macro-F1 on corrected Tobacco-3482, 17.4 points above zero-shot.

  • Problem

    Regulated document classification must operate with bespoke, long-tailed taxonomies, few labels, data-residency constraints, and drifting templates.

  • Method

    HIRA uses a training-free retrieval cascade with multimodal fusion, local LLM verification, and human corrections stored in evolving retrieval memory.

  • Results

    0.9423 Macro-F1 on corrected Tobacco-3482, 17.4 percentage points above the LLM zero-shot baseline, while matching the fully labelled pool oracle after labelling 24.8% of the pool.

  • Takeaways & Limitations

    HIRA provides a practical alternative to repeated model retraining for long-tail document classification in regulated deployments.

  • Takeaways & Limitations

    The evaluation omits a fully supervised LayoutLMv3 baseline on Financial-80 because collecting sufficient labels is infeasible under deployment constraints.

Abstract

from arXiv · show

Document classification in regulated industries is constrained by data residency, limited cold-start labels, scarce review capacity, and costly model-governance procedures. We present HIRA, a training-free, on-premises retrieval-augmented cascade for document classification in regulated deployments that combines BM25 over OCR text, dense text embeddings, and image-level representations through validation-calibrated weighted reciprocal-rank fusion. Confident documents are classified directly by retrieval; uncertain or visually confusable documents are passed to a locally hosted LLM verifier, which receives the OCR text, retrieved exemplars, label descriptions, and confusion-specific terms. When the verifier remains uncertain, the document is sent to human review. Each correction is stored as a margin-weighted retrieval exemplar and updates a Dirichlet-smoothed confusion graph, letting the system improve without updating model weights. On a private 80-class trade-finance corpus, HIRA processes the full 30,233-document production stream while requesting human correction for only 1,945 documents (6.4%), improving Macro-F1 from 0.6218 to 0.8548. On the corrected Tobacco-3482 benchmark, HIRA reaches 0.9423 Macro-F1 with a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier, 17.4 percentage points above the zero-shot LLM baseline, while invoking the verifier for only about 40% of documents and reducing LLM calls by approximately 60%. With 518 human corrections (24.8% of the pool), HIRA matches the fully labelled pool oracle, in which all 2,086 pool documents are indexed with their ground-truth labels. These results show that selective human feedback and retrieval-memory adaptation can be a practical alternative to repeated model retraining for long-tail document classification in regulated deployments.

1 Introduction

HIRA addresses regulated document classification when data cannot leave deployment environments, labelled data is unavailable at cold start, and annotation capacity is limited. It uses a training-free retrieval cascade with selective local LLM verification and human correction, achieving strong results with limited feedback.

  • 1 Introduction: The method targets deployments where confidential documents cannot use cloud LLM APIs, bespoke taxonomies lack labelled training data, and templates or categories change after deployment.Human annotation is described as the dominant marginal cost in this setting.
  • 1 Introduction: HIRA fuses BM25 over OCR text, dense sentence embeddings, and image-level representations through validation-calibrated weighted reciprocal-rank fusion.The cascade routes uncertain or visually confusable cases to a verifier receiving OCR text, retrieved exemplars, label descriptions, and distinguishing terms.
  • 1 Introduction: Macro-F1 improves from 0.6218 to 0.8548 on Financial-80 while human correction covers only 1,945 of 30,233 documents (6.4%).HIRA processes the full private 80-class trade-finance production stream.
  • 1 Introduction: Macro-F1 reaches 0.9423 on Tobacco-3482 [17], 17.4 percentage points above the LLM zero-shot baseline with a 40% verifier call rate.The result uses a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier and matches the fully labelled pool oracle.

2 Related Work

Related work spans supervised document-image classification, retrieval-augmented classification, and cost-aware cascade inference. HIRA is positioned against the cold-start and taxonomy-adaptation constraints of fine-tuned models while combining retrieval and selective routing.

  • Overview: Existing document-classification research is organized around fine-tuned document-image models and retrieval-augmented or cascade inference approaches.This taxonomy frames HIRA’s relationship to the three related-work areas discussed in the section.
  • Fine-tuned document-image classification: Fine-tuned document models including LayoutLM [25], LayoutLMv2 [26], LayoutLMv3 [8], DiT [16], Donut [11], and UDOP [20] require labelled training data and retraining when taxonomies change.The latter adds a GPU pass and compliance review, creating limitations for cold-start regulated deployments.
  • Retrieval-augmented classification: Retrieval-augmented methods supplement learned parameters with non-parametric retrieval, supporting nearest-neighbour label aggregation, LLM prompting, verifier-based classification, and long-tail legal-text classification.Examples include RAG [14], dense passage retrieval [9], kNN-LM [10], kNN-Prompting, Class-RAG [2], and SRA [18].
  • Cascade inference and cost-aware routing: Cascade inference routes inputs across models of increasing capacity to balance cost and quality, with FrugalGPT [3] and CALM [19] reducing expensive computation through uncertainty or confidence signals.The cited approaches assume either no retrieval or a fixed retrieval stage, whereas HIRA uses tiered thresholds.

3 Problem Setting

Regulated document classification must operate with customer-specific, long-tailed taxonomies and only tens of labelled documents per class, while avoiding supervised fine-tuning and cloud-LLM defaults. HIRA therefore evaluates accuracy per human label and updates retrieval memory and a lightweight confusion graph without changing model weights.

  • 3 Problem Setting: Customer-specific, long-tailed taxonomies and only tens of labelled documents per class create a cold-start problem under deployment drift.These conditions jointly preclude supervised fine-tuning and cloud-LLM defaults in regulated deployments.
  • 3 Problem Setting: Accuracy per human label is the appropriate metric because the data are long-tailed and human review capacity is limited.
  • 3 Problem Setting: Retrieval-augmented classification fits the setting because HITL corrections update only retrieval memory and a lightweight confusion graph, leaving model weights untouched.

4 The HIRA System

HIRA is a training-free, two-stage retrieval-augmented cascade that combines weighted multi-modal retrieval, local LLM verification, and HITL review. Human corrections update only the retrieval index and Dirichlet-smoothed confusion graph through EDRM, guiding later decisions without model-weight updates.

  • Deployment and evaluation: All components remain training-free after deployment: only the retrieval index and confusion graph change, while evaluation fixes stream order and reruns the full cascade from frozen states.Methods independently select HITL documents but process the same document stream and are evaluated on the same held-out test set at correction checkpoints.
  • Cascade and Tier 2: Tier 1 returns its top class when the score clears τ_t1; otherwise, Tier 2 verifies the case or routes it to human review when verification fails.The verifier receives OCR text, retrieved exemplars, category descriptions, and confusion-specific terms for high-confusion class pairs.
  • Tier 1: HIRA combines three parallel retrievers—BM25 over OCR, MiniLM OCR embeddings, and SigLIP2 image embeddings—using deployment-calibrated weighted reciprocal-rank fusion.Fusion uses k_rrf = 10 and per-retriever weights calibrated by validation Macro-F1; the paper reports that fusion improves over any single retriever.
  • EDRM: EDRM makes each correction an adaptive retrieval update: difficult or low-margin examples receive larger clipped weights, while stale entries decay and are pruned.Misclassified and low-margin corrections receive larger weights than easy confirmations; γ = 0.1, w_min = 0.1, and w_max = 10.
  • EDRM: EDRM also maintains a Dirichlet-smoothed confusion graph whose high-confusion pairs supply distinguishing OCR tokens to future Tier-2 prompts.Confusion counts decay with a deployment-set half-life, allowing outdated confusion patterns to be discarded.

5 Experimental Setup

Experiments evaluate HIRA on an imbalanced private trade-finance corpus and a corrected Tobacco-3482 benchmark using shared retrieval splits, cold-start baselines, and deployment-oriented metrics. DeepSeek-R1-Distill-Qwen-32B is selected as the Tier-2 verifier, while main results use a fixed seed after a low-variance check.

  • Datasets: Financial-80 is a private, skewed 80-category trade-finance corpus containing many near-synonymous document classes requiring fine-grained disambiguation.Only aggregate metrics are reported because of the data-sharing agreement.
  • Datasets: Corrected Tobacco-3482 contains 2,737 documents across 10 categories, with 100 seed documents, 2,086 streamed pool documents, and 279 held-out test documents.Tier-1 RRF weights and its threshold are calibrated on the validation split.
  • Baselines: Baselines share HIRA’s retrieval index, pool, and test split, covering single-feature retrieval, RRF, naive HITL, zero-shot DeepSeek, and matched-label-count DiT fine-tuning.Full-training-set supervised fine-tuning is excluded as a primary baseline because cold-start labels are unavailable and updates require model-governance review.
  • Verifier selection: DeepSeek-R1-Distill-Qwen-32B [5] is selected as the verifier after achieving Macro-F1 0.8819, versus 0.8452 for Qwen-VL2.5 [1] and 0.8233 for Llama-3.1-8B [7].It also runs faster per document on the same A100.
  • Evaluation protocol: Macro-F1 is the primary metric for the imbalanced setting, supplemented by accuracy, group F1, queue and verifier call rates, and Tier-1/Tier-2 invocation fractions.All HITL-curve results use the frozen-state, full-cascade protocol.
  • Evaluation protocol: A three-run Financial-80 check at N=1,945 produced Macro-F1 values 0.8523, 0.8535, and 0.8586, with mean 0.8548 and a 0.63-point range, so main results use one fixed seed.The observed variance was an order of magnitude below the smallest method gap.

6 Results

On corrected Tobacco-3482, HIRA’s final cascade reaches 0.9423 Macro-F1 while invoking the verifier for about 40% of documents and matching the fully labelled oracle after 518 human corrections. Retrieval-memory adaptation improves performance without weight updates and reduces human-review and LLM costs.

  • Cascade versus zero-shot: 0.9423 Macro-F1 is HIRA’s final score on 2,086 Tobacco-3482 documents, +17.4 points over DeepSeek zero-shot, with approximately 40% verifier calls versus 100%.The result uses a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier under the enum5 configuration.
  • HITL learning trajectory: At 518 corrections, HIRA labels 24.8% of the pool yet reaches 0.9423 Macro-F1, matching the fully labelled pool oracle.The oracle indexes all 2,086 documents with ground-truth labels at uniform weight 1.0.
  • EDRM versus naive correction injection: Margin-weighted exemplar injection raises Macro-F1 from 0.8857 to 0.9135, while the full cascade reaches 0.9423 versus 0.9135 for naive HITL.The confusion graph augments the verifier’s distinguishing-term prompt, linking retrieval-memory adaptation to the additional gain.
  • Configuration and cascade efficiency: V3 reaches the highest cascade-final Macro-F1, 0.9423, with 518 corrections versus 862 for V0, while about 60% of documents stop at Tier 1 and 18.6% enter human review.The queue rate decreases as the index matures, and the verifier accepts the remaining Tier-2 cases without human review.

7 Financial-80: Production Deployment Results

On the harder, imbalanced 80-class Financial-80 production corpus, HIRA raises Macro-F1 from 0.6218 to 0.8548 while requesting human correction for only 1,945 of 30,233 pool documents. The cascade combines retrieval, verifier escalation, and exemplar updates to reduce human workload while resolving near-synonym confusions.

  • Dataset and protocol: Financial-80 contains 32,911 documents across 80 categories, with 30,233 unlabelled pool documents and aggregate-only reporting because the taxonomy is confidential.The HITL experiments use 648 seed documents, 316 validation documents, and 1,714 test documents.
  • Static baselines: The best seed-only baseline reaches 0.622 Macro-F1 versus 0.774 on Tobacco-3482, while RRF reaches 0.502 versus 0.391 for the best single retriever.DS zero-shot shows a 0.945 Head-F1 and 0.557 Tail-F1, producing a 38.8-point Head–Tail gap.
  • Dynamic queue rate: At N=1,945, Tier-1 retrieval accepts 61.6% of documents, Tier 2 receives 38.4%, and only 6.2% enters the human-review queue.The dynamic queue rate falls from approximately 18.8% initially to about 6.2% as corrections accumulate, while the verifier absorbs most uncertain cases.
  • HITL efficiency: 0.8548 cascade-final Macro-F1 exceeds the 0.6218 DS zero-shot baseline by 23.3 points after scanning all 30,233 pool documents and routing 1,945 documents to correction.This corresponds to 6.4% of the production stream; one reviewer completed the corrections in roughly six working days.
  • EDRM and cascade contribution: The full cascade reaches 0.8548 Macro-F1 versus 0.7920 for Tier-1-only retrieval at N=0, a +6.3-point gain from verification and exemplar injection.Near-synonym invoice and certificate subtypes dominate confusion clusters, motivating distinguishing-term handling.

8 Limitations

The evaluation is dominated by training-free baselines, with DiT serving only as a limited-label supervised reference on Tobacco-3482. A fully supervised LayoutLMv3 baseline is omitted on Financial-80 because collecting enough labels is infeasible under deployment constraints.

  • Evaluation scope: Most baselines are training-free, limiting comparison with fully supervised methods.The paper frames this choice around regulated deployment constraints.
  • Evaluation scope: DiT baselines provide only limited-label supervised references on Tobacco-3482.
  • Evaluation scope: The study excludes a fully supervised LayoutLMv3 baseline on Financial-80 because collecting sufficient labels is infeasible under deployment constraints.

9 Conclusion

HIRA is a training-free, on-premises retrieval-augmented cascade for regulated document classification that combines multimodal retrieval, local LLM verification, and selective human-feedback memory updates. It reaches Macro-F1 0.8548 on Financial-80 while routing 6.4% of documents to human correction, and Macro-F1 0.9423 on corrected Tobacco-3482.

  • Conclusion: Macro-F1 0.8548 on Financial-80, up from 0.6218, while only 6.4% of the 30,233-document stream is routed to human correction.This summarizes HIRA’s reported Financial-80 outcome and selective-review scope.
  • Conclusion: Macro-F1 0.9423 is achieved on corrected Tobacco-3482, with a reported +17.4-point improvement above the LLM zero-shot baseline.The supplied passage states the benchmark result and improvement, but its final comparison text is truncated.
  • Conclusion: HIRA combines multimodal RRF retrieval, a locally hosted LLM verifier, and selective HITL correction stored as retrieval memory and confusion-graph updates.The cascade is presented as training-free and on-premises for document classification in regulated industries.
Loading 2608.21792v1…