Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics

Maksim Evdokimov; Matvey Ivanov; Dmitrii Tsiupin; Olga Tsymboi; Anatolii Potapov; Aleksandr Ivanov

Source-linked AI summary

Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics

Maksim Evdokimov, Matvey Ivanov, Dmitrii Tsiupin, Olga Tsymboi, Anatolii Potapov, Aleksandr Ivanov

arXiv:2609.01575v1Sep 1, 2026cs.CL

TL;DR

Regulated document extraction remains difficult because privacy, latency, and serving-cost constraints limit existing automation. The paper develops a difficulty-aware, production-adapted MoE VLM and evaluates its quality-adjusted deployment economics. The deployed model outperforms larger baselines while offering substantially lower costs.

Problem
Regulated document extraction at hundreds-of-millions scale is limited by privacy rules, strict latency requirements, costly cascades, and expensive quality-clearing open-source VLMs.
Method
The paper fine-tunes a 35B-total, 3B-active MoE VLM on in-house data and Difficulty-Aware-curated open-domain documents for prompt-driven heterogeneous workflows.
Results
The model averages 0.814 across evaluation groups and beats the strongest deployable baseline by 53% in quality-adjusted cost.
Takeaways & Limitations
Quality-adjusted economics identify the model as the most economically viable solution for regulated document-flow deployment.
Takeaways & Limitations
The training corpus lacks multi-step reasoning examples, so performance may be lower on compositional reasoning and long-range evidence aggregation workflows.

Abstract

from arXiv · show

Extracting structured fields from hundreds of millions of documents annually remains costly in regulated industries: bespoke OCR cascades cover only a fraction of workflows, privacy rules preclude external models, and existing open-source VLMs that clear quality thresholds cost more to serve than human annotation. We present a deployed document-understanding system built on a Mixture-of-Experts VLM (35B total, 3B active), fine-tuned on in-house production data mixed with open-domain documents curated by a Difficulty-Aware pipeline for layout diversity, fact-extractability, and cross-model consistency. Fitting on a single H100 and serving heterogeneous workflows via prompting, the model leads all deployable (non-reasoning) baselines up to an order of magnitude larger. A quality-adjusted cost analysis, with confirmation and correction costs calibrated from production telemetry, shows it reduces expected costs by over 80% against the human baseline and by more than 50% against the best competing open-source model, while larger baselines remain economically unviable.

1 Introduction

The paper targets costly document automation in regulated settings with a unified, privacy-compatible VLM and difficulty-aware data curation. Its deployed model combines strong benchmark performance with quality-adjusted economic viability.

Motivation: Regulated document processing is constrained by privacy rules, costly large VLM serving, strict latency limits, and maintenance-heavy OCR cascades.The target platform processes hundreds of millions of Cyrillic-language documents annually, including judgments, invoices, tax certificates, and receipts.
Approach: The proposed MoE VLM has 35B total and 3B active parameters, replacing separate workflow pipelines with one prompt-driven model on a single H100.Training mixes in-house production data with open-domain PDFs curated to retain hard, information-rich samples.
Results: 0.814 average score across evaluation groups exceeds the base checkpoint by 8.4 points and the strongest reasoning-mode baseline by 5.3 points.The reasoning-mode baseline is ten times larger.
Deployment economics: The model handles production traffic within latency constraints, while large baselines require more GPUs than economically justified against replaced human work.The quality-adjusted analysis reports a 53% advantage over the strongest deployable baseline.
Deployment economics: The methodology links per-field accuracy to deployment economics using coefficients calibrated from production telemetry and workflow-specific recalibration.Reported coefficients include workload distribution, GPU utilization, routing policy, and per-action costs.

2 Related Work

Prior work establishes rapid VLM progress, production deployment, and data-curation benefits, but leaves unified document extraction and quality-adjusted economics insufficiently formalized. The paper positions its system as an end-to-end, single-VLM alternative to cascaded pipelines.

VLM scaling: Scaling improves benchmark performance for frontier VLMs, but 70B–400B models require multi-GPU serving with prohibitive per-document cost.
Production systems: Recent production systems consolidate workflows or use structured synthesis, while document systems commonly rely on cascaded OCR–LLM pipelines.
Document extraction: The paper replaces document OCR–LLM cascades with end-to-end extraction from rendered pages through a single VLM.
Deployment economics: Prior work uses confidence routing and pay-per-token query cascades, but asymmetric confirmation-versus-correction costs and per-field economics remain largely unformalized.

3 Framework

The framework builds a single-interface document VLM from production-aligned supervision and curated open-domain PDFs. Its curation pipeline filters for reliable, structurally useful, visually grounded, and cross-model-consistent training examples.

Framework: The system serves field extraction, document classification, visual validation, and freeform VQA/OCR through one instruction-following interface.Supervision combines in-house workflow data with open-domain documents.
In-house supervision: In-house annotations arise from routine workflows, matching production semantics and taxonomies without additional annotation effort.Prompts encode field semantics, layout cues, disambiguation rules, formatting constraints, and absent-value handling.
Difficulty-aware curation: The DADC pipeline filters 300K multilingual Common Crawl PDFs because internal layouts and taxonomies saturate while unfiltered open-domain samples are often too easy.
Difficulty-aware curation: Text-layer validation compares embedded PDF text with internal OCR on rendered pages and retains 40% of input pages.
Difficulty-aware curation: Two-stage VLM filtering rebalances prose-heavy pages toward forms, tables, dense layouts, and fact-extractable documents.Fact-extractability rescoring recovers structurally plain but fact-dense pages discarded by the structural filter.
Grounding and verification: Generated text-based supervision is grounded in rendered document pages, then verified across model families to filter hallucinations and schema violations.A separate refinement stage rewrites annotations into target formats and adds explicit formatting rules.
Visual augmentation: Sample-serving augmentation applies photometric, geometric, and elastic transformations to narrow the gap between pristine PDFs and camera captures, scans, and screenshots.

4 Experiments

Experiments evaluate the model across internal and public document benchmarks, data-mixture ablations, base-model scales, and deployment constraints. The model leads deployable baselines across benchmark groups while avoiding reasoning-mode latency and cost.

Benchmark results: 0.767 is the best average over non-fine-tuned benchmarks, 2.4 points above the strongest open-source reasoning baseline and 3.4 points above the base model.The model also achieves competitive multi-page performance at 0.764.
Benchmark results: 0.764 matches the 0.777 multi-page score of the much larger reasoning model without test-time reasoning.Reasoning traces improve the reference model by decomposing dispersed visual evidence, but production latency SLAs preclude such inference.
Deployment constraints: Deployable configurations show the model leading every benchmark group, including multi-page at 0.764 versus 0.745 for the non-reasoning 397B baseline.Reasoning-mode models are treated as reference points rather than deployable candidates because of latency constraints.
Base-model scaling: SFT gains generalize beyond its coverage, while post-trained Qwen3.5-35B-A3B-Base retains the lead across all groups.The comparison uses identical SFT data across base-model scales.
Data scaling: Internal data saturates quickly, whereas open-domain data continues improving performance; a hybrid mixture provides the best trade-off.The adopted strategy uses internal data for early exploration and targeted gap-filling, while open-domain synthetic data carries most scaling.
Cross-lingual transfer: DADC-trained data improves every public benchmark, with DocVQA rising from 82.14 to 90.10 and InfoVQA from 51.10 to 66.30.These gains transfer to English despite predominantly Russian training data; OCRBench and OCRBench v2 also improve by +15% and +1.1%.

5 Quality-adjusted Cost Analysis

The quality-adjusted cost framework converts model quality and serving requirements into per-document economics, showing why smaller models can be more viable despite competitive quality. It routes fields by accuracy and precision thresholds and compares costs against manual annotation.

Framework: Per-document economics combine inference cost, field-level quality, workflow characteristics, and the fully manual baseline for unified model comparison.The framework treats field extraction as the dominant cost driver and can extend to classification and criteria validation.
Inference Cost: GPU footprint is the primary cost driver because compact MoEs may require 1 GPU while dense 400B models require 8.Inference cost also depends on utilization, monthly GPU cost, and profiled throughput for each model–workflow pair.
Field-level Costs: Confirmation costs 0.3x and correction costs 1.1x relative to manual annotation, reflecting asymmetric assisted-mode effort.Expected assisted-field cost depends on precision because precision determines the confirmation–correction split.
Routing Thresholds: θauto = 0.95 bounds residual error on fully automated fields, while θassist = 0.125 discards suggestions whose correction cost exceeds fresh annotation.Deployment is economically justified when ρw(m) > 0.
Results: Our model achieves the highest cost reduction ratio in both workflow groups and nearly 5× higher throughput than Qwen2.5-VL-72B.Large-model serving can produce negative cost reduction ratios when compute exceeds displaced annotation effort, especially as field count decreases.

6 Conclusion

The paper presents a domain-adapted 35B-A3B VLM trained with difficulty-aware curation on mixed production and open-domain data. It argues that quality-adjusted economics identify the model as the most viable solution for regulated document deployment.

Conclusion: The 35B-A3B VLM combines in-house production data with open-domain PDFs curated through a difficulty-aware pipeline.The pipeline is intended to produce domain-adapted supervision for document understanding.
Conclusion: The model outperforms baselines up to an order of magnitude larger on production benchmarks.The conclusion distinguishes benchmark-leading quality from the additional requirement of deployment viability.
Conclusion: The quality-adjusted cost framework makes the quality–cost trade-off explicit and identifies the proposed model as the most economically viable solution.The paper presents this framework as a template for grounded model selection in deployment economics.

Limitations

The paper identifies unresolved limits in reasoning coverage, cross-lingual evaluation, and post-training performance. These boundaries motivate future work beyond the current SFT recipe and evaluation suite.

Reasoning and reinforcement learning: The model may underperform on compositional reasoning and long-range evidence aggregation because the corpus lacks multi-step reasoning examples.The authors defer reinforcement learning until SFT data scaling yields diminishing returns, partly because verifiable rewards require unambiguous ground-truth answers.
Cross-lingual coverage: Cross-lingual transfer remains unclear because the study does not include a per-language ablation.Training includes multiple Cyrillic languages alongside Russian, but their individual effects are not isolated.
Post-training ceiling: SFT has not pushed the model to its performance ceiling, which the authors primarily attribute to pre-training limitations.Planned extensions include broader data diversity, reasoning coverage, and higher-quality supervision.

Ethical Statement

The study uses in-house human annotation and keeps sensitive data within secured institutional infrastructure. Its deployment framework is explicitly designed for human-in-the-loop operation in regulated settings.

Human annotation: Human annotation, quality control, and benchmark construction were performed by trained in-house operators under full-time contracts, without external crowdsourcing.The assisted-mode cost study used aggregated, anonymized telemetry from 50 annotators over three months.
Data privacy: Privacy constraints motivated on-premise fine-tuning and deployment, with internal data protected by access controls, encryption, and audit logs.Internal documents, derived samples, and checkpoints trained on internal data were not transferred outside the institution’s secured infrastructure.
Responsible use: The model is intended for regulated enterprise document understanding, not unconstrained consumer-facing deployment without further safety review.The cost framework requires recalibration before transfer, and routing decisions must be revisited when workflows or document distributions materially change.
Benchmark construction: Internal benchmarks use overlapping annotation by trained annotators, with disagreements adjudicated and items lacking majority consensus excluded.This procedure is described as part of benchmark construction rather than external crowdsourced evaluation.

A.2 Contamination Control

The curation pipeline controls contamination and progressively filters open-domain PDFs for valid text, challenging document structure, extractable facts, and cross-model consistency. Prompt variation and entity-level partitioning further target generalization beyond memorized entities or templates.

A.2 Contamination Control: Evaluation documents are partitioned by unique entities and templates so reported metrics measure generalization rather than memorization.The procedure separates company names, personal names, taxpayer identifiers, account numbers, and document structures between training and evaluation.
A.2 Contamination Control: Entity-level partitioning also separates Court Case Files from Short Court Rulings, making the latter an out-of-distribution judicial document test.The two benchmarks differ in document structure, schema focus, and parties or identifiers.
B.1 Prompt Template: Training prompts combine task descriptions with output schemas, while semantically equivalent phrasings and randomized key-value order reduce template overfitting.The generalized field-extraction template is shown in Figure 1.
C.2 Text Layer Validation: PDF text is validated by comparing the embedded text layer with internal OCR, discarding pages below a fixed consistency threshold.This addresses malformed encodings, rasterized pages, and transparent OCR layers that can corrupt supervision.
C.2 Text Layer Validation: 40% of input pages survive text-layer validation, with matched precision 0.80, matched recall 0.90, and average normalized Levenshtein distance 0.05.The remaining pages are discarded as malformed or dependent on low-quality OCR text layers.
C.3 Document Sampling: The document-sampling funnel admits visually structured pages directly and rescues plain-text pages when visual complexity or QA potential reaches at least 5.Stage 1 uses taxonomic labels, while Stage 2 complements it by recovering pages with concrete extractable facts despite sparse structure.
C.5 Consistency Verification: At least two of three independent verifiers must agree with a generated answer for acceptance, retaining approximately 35–40% of candidate samples.Mismatched samples are discarded because disagreement may indicate hallucination or an unsolvable question; many rejections involve table or sparse-layout artifacts.

C.7 DADC Component Ablation

The DADC ablation tests progressively added curation components and shows that filtering recovers the in-house baseline, while cross-model verification provides the improvement from open-domain data. The resulting accepted pool is shifted toward harder visual examples.

Ablation design: DADC ablation progressively adds layout and fact-extractability sampling, then cross-model verification, while holding the training budget fixed.Table 9 distinguishes unfiltered, sampled, and verified open-data variants, with augmentation enabled throughout.
Ablation results: Unfiltered open-domain data degrades the in-house-only baseline, whereas layout and fact-extractability sampling recover parity.Naively scraped PDFs displace production-relevant signal before filtering is applied.
Ablation results: Cross-model consistency verification is the only stage that strictly improves over in-house-only training, turning open data into a positive scaling factor.The verification stage filters hallucinations and schema violations through agreement across model families.
Difficulty distribution: After DADC, the retained pool shifts toward higher visual difficulty while preserving the relative proportions among difficulty levels.The raw distribution is heavily skewed toward easy examples.

D Offline Experiments

Offline experiments compare full-parameter fine-tuning with LoRA and describe the data-serving infrastructure used for rendered-document augmentation. LoRA achieves comparable downstream metrics with substantially less compute, while the augmentation pipeline is decoupled from training nodes.

Fine-tuning configuration: LoRA reaches metrics within noise of full fine-tuning at a fraction of the compute.The LoRA configuration supports rapid hypothesis testing and data-mixture or prompt-format ablations.
Fine-tuning configuration: Full-parameter fine-tuning converges substantially more slowly and requires larger GPU FLOPs than the subsequently evaluated LoRA configuration.Both configurations use identical data mixtures but their own optimal hyperparameter setups.
Data pipeline: Rendered-document augmentation is served asynchronously through a framework-agnostic stack decoupled from training nodes.The service handles retrieval, augmentation, and preprocessing, while independent horizontal scaling helps maximize GPU utilization.
Data pipeline: The data pipeline separates image storage from training nodes and isolates augmentation dependencies from the training environment.These design choices remove per-node storage coupling and reduce dependency conflicts.

E Latency Analysis

Latency experiments evaluate per-document serving under increasing concurrency and calibrate the quality-adjusted human-work coefficients from production telemetry. The proposed model remains within the 10-second SLA on one H100, unlike both larger baselines.

Serving latency: P95 latency for the proposed model stays below the 10-second SLA through five workers, reaching 7.4 seconds at the tested maximum.Measurements use approximately 6K input tokens, including 5K visual tokens, and up to 0.5K output tokens.
Serving latency: Qwen2.5-VL-72B breaches the SLA at two workers with P95 = 10.96s, while Qwen3.5-397B-A17B-FP8 crosses it at three workers with P95 = 10.77s.The latter requires an 8×H100 replica under expert parallelism.
Serving latency: The proposed model and Qwen2.5-VL-72B are benchmarked on one H100, whereas Qwen3.5-397B-A17B-FP8 requires eight H100s to fit.Higher concurrency levels for the larger baseline were infeasible under the SLA constraint.
Human-work calibration: Production telemetry from 50 annotators yields a confirmation coefficient of 0.30 and a correction coefficient rounded to 1.1.The coefficients are based on several million field interactions collected over a rolling three-month window.