PaperlayerView exact revision on arXiv

Source-linked AI summary

State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models

Mingxu Chai, Chenyu Liu, Ziyu Shen, Jiazheng Zhang, Kaidi Zhang, Ruoyu Chen, Jun Long, Jihua Kang, Tao Gui, Qi Zhang

arXiv:2608.28698v1cs.CV

TL;DR

Document VLMs repeatedly access globally compressed visual tokens even though each prediction typically needs localized, decoding-state-dependent evidence. SCVER addresses this mismatch with coarse-to-fine state-conditioned retrieval, stabilized by SGLO, and improves robustness and accuracy–efficiency trade-offs, especially at reduced input resolution.

  • Problem

    Existing document VLMs use global visual tokens for both coarse semantics and fine-grained details, although prediction evidence is localized and varies by decoding step.

  • Method

    SCVER uses compact global tokens for coarse structure and retrieves relevant high-resolution regions conditioned on the current decoding state, while SGLO provides weak spatial guidance during training.

  • Results

    SCVER consistently improves robustness and achieves a better accuracy–efficiency trade-off on document parsing benchmarks, especially under reduced input resolution.

  • Takeaways & Limitations

    Decoding-time visual evidence retrieval provides a practical direction for improving fine-grained perception in future VLMs.

  • Takeaways & Limitations

    SGLO increases training-time computation and may reduce training throughput because it requires an additional frozen-base forward with extracted attention maps.

Abstract

from arXiv · show

Compared with typical vision-language tasks, document parsing places stronger demands on fine-grained visual perception. Existing vision-language model (VLM)-based parsing approaches rely on globally compressed visual tokens, where fine-grained details are entangled within a single representation and repeatedly accessed during decoding. However, we observe that the visual evidence for each prediction is typically localized and conditioned on the current decoding state, whereas such representations must be accessed in full at every decoding step, resulting in inefficient computation. To address this mismatch, we formulate perception as state-conditioned visual evidence retrieval (SCVER) during autoregressive decoding. The model operates on a compact global representation for coarse structure and retrieves a small set of relevant high-resolution regions conditioned on the current token state. This coarse-to-fine design enables on-demand access to fine-grained visual cues, relieving globally shared representations from encoding all fine-grained details. We further find that learning such state-conditioned retrieval in VLMs is challenging and unstable. To stabilize this process, we introduce a Spatially-Guided Learning Objective (SGLO) to guide the retrieval process. Experiments on document parsing benchmarks show that SCVER improves robustness under reduced input resolution and achieves a better accuracy-efficiency trade-off, demonstrating the effectiveness of on-demand visual evidence retrieval for fine-grained perception.

1 Introduction

Document parsing requires fine-grained visual perception, but existing VLMs repeatedly access globally compressed representations even though each prediction uses localized, state-dependent evidence. SCVER retrieves relevant high-resolution regions on demand, while SGLO stabilizes this retrieval learning.

  • Document parsing relies more on precise visual perception and fine-grained feature modeling than typical vision–language tasks.
  • Existing methods encode entire images into fixed visual tokens, forcing one global representation to capture all details and be fully accessed at every decoding step.This mismatch creates redundant computation that worsens as input resolution and visual-token count increase.
  • SCVER combines compressed global tokens for coarse structure with state-conditioned retrieval of relevant high-resolution regions during decoding.This coarse-to-fine design reduces dependence on high-resolution inputs while maintaining efficient decoding.
  • Naive SCVER training is unstable because retrieval locations are latent spatial variables only indirectly supervised by token-level losses.The resulting selection process can be underconstrained and degenerate.
  • SGLO stabilizes retrieval by using visual-token attention from an independent frozen-base forward as a weak spatial prior, without additional annotations.
  • Over 70% computation is reduced with negligible accuracy loss when SCVER is integrated with MinerU2.5, while MonkeyOCR PubTabNet accuracy rises from 86.9 to 91.2.

2 Related Works

Related work improves document parsing through modular, end-to-end, and lightweight pipelines, but gives limited attention to fine-grained visual information use during decoding. Other approaches address resolution, multi-scale features, adaptive access, sparse attention, or token pruning without explicitly implementing state-dependent retrieval as the core mechanism.

  • Document parsing evolved from modular systems with specialized models toward end-to-end image-to-structured-text approaches and lightweight detect-then-recognize pipelines.
  • These parsing paradigms mainly optimize overall system design, with limited attention to fine-grained visual information utilization during decoding.
  • Dynamic-resolution encoders handle aspect-ratio variation but do not explicitly enhance fine-grained perception, while multi-scale methods improve representation capacity at an efficiency cost.
  • Adaptive visual-access methods remain within standard attention, where evidence is implicitly aggregated rather than explicitly retrieved in a state-dependent manner.
  • Sparse attention and token-pruning methods reduce interactions or redundant visual tokens, but they differ from explicit retrieval of task-relevant spatial evidence.

3 Method

SCVER keeps compressed visual tokens for global structure while retrieving localized high-resolution evidence conditioned on decoding state. SGLO stabilizes this retrieval by guiding predicted reference points with spatial targets from a frozen base stream.

  • Motivation: Existing compressed-token decoding faces a trade-off between fine-grained information and quadratic attention cost O((nv + nt)^2).Increasing nv preserves more detail but increases computation, whereas reducing nv sacrifices fine-grained information.
  • State-Conditioned Visual Evidence Retrieval: SCVER augments compressed global representations by retrieving relevant high-resolution visual regions conditioned on the current decoding state.This avoids requiring compressed tokens to encode every fine-grained visual detail upfront.
  • SCVER Operator: The retrieval operator uses token-conditioned sparse sampling to select spatial locations, offsets, and weights from pre-downsampling features.Sampling parameters vary with the autoregressive token representation, enabling different decoding states to access different regions.
  • SCVER Operator: SCVER provides token-specific access to fine-grained evidence while retaining a compact global representation and remaining compatible with KV caching.The retrieved evidence is combined with the original token through a residual connection.
  • Spatially-Guided Learning Objective: Naive state-conditioned retrieval is unstable because spatial retrieval variables receive only weak constraints from token-level supervision.Predicted locations can therefore become diffuse or degenerate during optimization.
  • Spatially-Guided Learning Objective: SGLO uses a frozen no-gradient base stream to produce detached visual-attention centroids that guide reference points predicted by the SCVER stream.Both streams share frozen backbone weights, while the final objective combines SCVER token-level cross-entropy with an alignment term.

4 Experiments

Experiments show that SCVER improves document-parsing accuracy and robustness under reduced resolution while preserving a favorable accuracy–efficiency trade-off. It also benefits text-rich VQA and gains from SGLO-stabilized retrieval training.

  • Main results: SCVER improves performance across standard and constrained document-parsing settings, including PubTabNet from 85.9 to 91.9 and B-MOD from 82.7 to 94.0 on Dolphin-1.5.
  • Robustness under reduced resolution: At 0.5 resolution, SCVER recovers MonkeyOCR-Pro PubTabNet accuracy from 58.3 to 92.8 and B-MOD accuracy from 62.4 to 91.3.This setting uses about 25% of the original visual tokens.
  • Accuracy–efficiency trade-off: At roughly 2 GFLOPs, SCVER raises formula recognition from about 61% to 91%, table recognition from about 43% to 80%, and text recognition from about 69% to 89%.
  • Accuracy–efficiency trade-off: SCVER reaches about 89% text-recognition accuracy at roughly 1.8 GFLOPs, while the baseline requires more than 6 GFLOPs for a similar level.Throughput remains comparable to the baseline despite additional retrieval operations.
  • Generalization to text-rich VQA: On text-rich VQA, SCVER remains competitive with DocVLM on DocVQA and TextVQA and performs better on InfoVQA under a limited token budget of 320.SCVER achieves these results without relying on an external OCR system.
  • SGLO analysis: With SGLO, the PubTabNet performance gain increases from 2.5% to 6.9%, while training becomes more stable and downstream performance improves.A weight of λ = 0.1 provides a better balance between stability and performance than the tested alternatives.
  • Retrieval frequency: Applying SCVER every four decoder layers reduces computational cost by approximately 10% with only a 0.7% accuracy drop versus applying it at every layer.
  • Retrieval behavior: Retrieved locations are highly localized and vary across tokens, with fine-grained tokens focusing on corresponding local regions.

5 Conclusion

The paper concludes that SCVER combines coarse-to-fine retrieval with SGLO to provide token-specific fine-grained visual evidence during decoding. Experiments indicate improved robustness and accuracy–efficiency trade-offs, especially at reduced input resolutions.

  • SCVER formulates perception as retrieval of token-specific visual evidence conditioned on the current decoding state.
  • The method combines coarse-to-fine retrieval with SGLO for efficient access to fine-grained visual evidence during autoregressive decoding.
  • Experiments show consistently improved robustness and a better accuracy–efficiency trade-off on document-parsing benchmarks, especially under reduced input resolution.

Limitations

SGLO adds training-time computation and implementation requirements through an auxiliary frozen-base forward and explicit attention extraction, but its auxiliary stream is disabled during inference.

  • SGLO requires an additional no-gradient frozen-base forward and explicit visual-token attention maps from selected decoder layers during training.
  • This auxiliary stream increases training-time computation and may reduce training throughput, while introducing no SGLO-specific inference overhead.

A Training Data

The training data combine benchmark-derived supervision with multilingual supplements and normalized formula representations. These additions address multilingual evaluation coverage and inconsistencies in LaTeX ground truth.

  • Training data are mainly derived from benchmark training splits covering layout understanding, formula recognition, and text recognition.
  • Existing open-source datasets are mainly English, creating a mismatch with multilingual OmniDoc evaluation data that may limit multilingual robustness.
  • The authors add multilingual formula and text-recognition data to better support multilingual document parsing.
  • Divergent LaTeX representations can introduce semantic ambiguities and inconsistencies in formula ground truth.
  • The dataset pipeline standardizes divergent LaTeX forms and rerenders them to verify alignment between input images and output sequences.
  • An additional 0.5M multilingual formula samples are incorporated from oleehyo_latex_formulas_80M.

A.2 Text Recognition Dataset

The paper constructs a scientific-document-oriented text recognition dataset to address limitations in existing layout annotations, using a pipeline that aligns detected image regions with extracted paragraphs.

  • Dataset motivation: DocLayNet provides 11 layout classes across six document categories, but represents mathematical content only as a general Formula category.This granularity does not separately label inline formulas embedded in textual regions.
  • Dataset construction: The authors build a complementary paragraph recognition dataset from the page-level annotated Text_Completion_arXiv corpus.The dataset targets scientific-document text recognition with block-level image–text pairs.
  • Dataset construction: Page images are segmented into layout elements, cropped by detected bounding boxes, and processed by a text recognition model to generate textual predictions.Paragraph-level ground truth is extracted using double-newline separators, then aligned with recognized text using edit-distance filtering.
  • Dataset outcome: The resulting pipeline produces a large-scale, high-quality element-level dataset tailored for robust scientific-document text recognition.

B.3 Layout Detection

The layout-detection study evaluates SCVER within MinerU2.5 and finds that its gains build on a strong vision–language baseline while improving fine-grained visual structure modeling. Additional analyses indicate that dynamic state-conditioned sampling, guided by balanced SGLO weighting, is more important than simply adding high-resolution features.

  • Experimental setup: SCVER is adapted to MinerU2.5 for layout detection because detector errors can limit page-level parsing improvements.The evaluation uses DocLayNet and D4LA with unified label mapping across their different taxonomies.
  • Baseline comparison: MinerU2.5 significantly outperforms YOLO-based and Detection Transformer detection models across layout element categories.The authors attribute this advantage to joint training on layout detection and layout element recognition.
  • SCVER results: SCVER further improves layout detection by helping the generative vision–language model use fine-grained visual information during decoding.The reported benefit is better capture of detailed visual structures.
  • Hyperparameter analysis: Using additional high-resolution feature maps brings only marginal accuracy gains while substantially increasing computation.With one feature map, increasing sampling points per head already approaches peak performance.
  • Hyperparameter analysis: The main benefit of SCVER comes from dynamic, state-conditioned sampling rather than simply aggregating more high-resolution visual features.
  • SGLO: Token-level loss alone provides no direct supervision for sampling locations, causing predicted sampling points to collapse toward nearly uniform global patterns.SGLO is introduced to explicitly guide the sampling process.
  • SGLO: A balanced SGLO weight is necessary because insufficient guidance yields marginal gains, whereas excessive weighting may constrain semantic representation learning.

B.6 Data Leakage Ablation Study

The data-leakage ablation examines whether performance gains could instead reflect training-data distribution differences. Fine-tuning on evaluation datasets produces mostly negligible or negative changes, while SCVER remains more beneficial than additional data on B-MOD.

  • Study motivation: Because the backbone’s training data are unavailable, the experiments test whether distribution differences could explain the observed improvements.
  • Ablation results: Fine-tuning the backbone yields limited improvements, with negligible changes on most benchmarks and slight degradation on some.B-MOD is the exception, showing a moderate gain.
  • Interpretation: On B-MOD, additional training data produce smaller gains than the proposed perception mechanism, indicating that the improvements are unlikely to be explained solely by data scaling.The results emphasize explicitly modeling fine-grained visual perception in generative vision–language models.

C Study Case

The case study examines how SCVER changes recognition behavior when visual evidence conflicts with language priors. Across text, formulas, and tables, the examples illustrate more faithful visual following and fewer structural or symbol-level errors, while the language-prior observation remains insufficiently investigated.

  • Text recognition: In the text-recognition example, the base model follows a semantically plausible language prior even though the image contains an incorrect spelling.
  • Text recognition: SCVER instead tends to follow the image’s visual information more faithfully rather than correcting it according to linguistic expectations.The authors suggest this behavior may mitigate hallucination caused by language-prior dominance.
  • Scope boundary: Whether recognition systems should correct spelling errors or reproduce image text exactly remains an open question in this case study.The phenomenon is not the work’s primary focus and is not systematically investigated.
  • Formula recognition: Formula examples compare SCVER and non-SCVER outputs on simple, matrix, degraded, and Chinese-text expressions, highlighting structural and symbol-fidelity failures without SCVER.
  • Table recognition: Table examples show that SCVER helps preserve complex multi-row, multi-column, and merged-cell structure compared with outputs containing boundary, association, and text errors.
Loading 2608.28698v1…