PaperlayerView exact revision on arXiv

Source-linked AI summary

InSight-doc: Agentic Visual Perception for Long-Document Understanding

Kaican Li, Weiyan Xie, Lewei Yao, Jiannan Wu, Lanqing Hong, Yongxiang Huang, Nevin L. Zhang

arXiv:2608.10628v1cs.CVcs.CLcs.LG

TL;DR

Long-document understanding must handle visually rich pages while limiting costly inference and context rot. InSight-doc uses retriever-free, coarse-to-fine active perception with learned zoom-in trajectories, and its SFT+RL model improves document VQA accuracy while moving the accuracy-efficiency frontier.

  • Problem

    Long-document understanding requires reasoning over many visually rich pages, but long contexts make inference costly and are associated with context rot.

  • Method

    InSight-doc starts from low-resolution document views and adaptively zooms into relevant high-resolution regions, appending the resulting evidence during end-to-end reasoning without external retrievers.

  • Results

    InSight-doc-8B achieves 66.9% average accuracy at r = 0.25, outperforming Qwen3-VL-8B by 16.4 points across four document VQA benchmarks.

  • Takeaways & Limitations

    The framework improves accuracy while reducing hallucination, context length, and end-to-end inference latency across document and high-resolution VQA benchmarks.

  • Takeaways & Limitations

    The study evaluates SFT+RL only with Qwen3-VL-8B-Instruct and does not test advanced RL methods or reward designs.

Abstract

from arXiv · show

Long-document understanding often requires reasoning over many visually rich pages, making inference costly and prone to context rot. In this work, we propose InSight-doc, an agentic visual perception framework that treats visual resolution as an adaptive reasoning-time resource. InSight-doc starts from low resolution and selectively zooms into high-resolution regions for finer evidence, without relying on any external retriever. To train such an agent, we construct an active-perception corpus of 17.9K high-quality SFT examples with region-level zoom-in trajectories, accompanied by 19.2K hard RL examples. Through SFT+RL, InSight-doc-8B improves the baseline by 4.3--16.4 accuracy points over document VQA benchmarks. On long documents, it reduces hallucination by more than 40% and inference latency by 41%--68% while maintaining an accuracy lead. Our code, datasets, and model are released at https://github.com/m-Just/InSight-doc .

1 Introduction

Long-document understanding is costly and vulnerable to context rot, especially for visually rich, multi-page documents. InSight-doc addresses this with adaptive, retriever-free visual perception that starts coarse and selectively zooms into relevant regions during reasoning.

  • Motivation: Long-document MLLM inference incurs O(N) space and O(N^2) time costs, while longer prompts can worsen performance through context rot.The paper attributes context rot to diluted attention over long contexts and limited genuinely long-context training data.
  • Motivation: Multi-page document understanding requires answering questions over lengthy, visually rich documents such as research papers and financial reports.The paper identifies this as a vital real-world application of multimodal large language models.
  • InSight-doc: InSight-doc starts from a low-resolution overview, then appends cropped high-resolution regions as visual evidence through interleaved thought and zoom-in actions.This coarse-to-fine workflow dynamically acquires multi-scale evidence without relying on a fixed resolution or external retriever.
  • Contribution: InSight-doc adaptively acquires visual evidence during multi-round reasoning over long documents and visually rich images.The framework is presented as an end-to-end agentic approach to long-document understanding.
  • Training: 17.9K multi-hop zoom-in SFT trajectories and 19.2K hard RL examples form the framework’s training corpus.These examples are designed to teach adaptive evidence acquisition during multi-round reasoning.
  • Results: The method improves the accuracy-efficiency Pareto frontier over the baseline in extensive experiments.The paper’s Figure 1 summarizes concurrent reductions in hallucination, sequence length, and latency alongside improved long-document VQA accuracy.

2 Related Work

Prior work spans end-to-end high-resolution processing, scripted or detector-based visual search, and emerging think-with-images methods. However, dynamically gathering evidence across distant pages of multi-page documents remains underexplored.

  • Existing paradigms: End-to-end methods preserve fine-grained visual information by feeding high-resolution pages into MLLMs, but task-irrelevant pages can create large visual-token contexts.The related-work discussion contrasts this approach with methods designed to reduce context burden.
  • Visual search: Visual search has progressed from external detectors and scripted zoom workflows toward MLLMs that internalize zoom and crop operations.The paper describes reinforcement learning and curated trajectories as routes to this internalized behavior.
  • Research gap: Existing visual search systems are mainly evaluated on natural photographs or single-page, text-rich images, where models often locate one salient region.These settings differ from multi-page document search.
  • Research gap: Multi-page documents require searching across collections of pages with evidence scattered across distant and non-adjacent pages.Joint reasoning and dynamic acquisition of multi-region evidence under a strict token budget remain largely underexplored.

3 InSight-doc

InSight-doc starts with downsampled document pages and adaptively zooms into selected regions during a reasoning-action loop. Its cost analysis models the tradeoff between reduced visual-token input and multi-turn tool-use overhead, reporting bounded sequence lengths and latency.

  • 3.1 Implementation: The process repeats until the model answers or reaches the tool limit, enabling iterative evidence acquisition within the reasoning loop.The approach therefore replaces a fixed-image assumption with dynamic visual access during inference.
  • 3.1 Implementation: InSight-doc initializes visual context with pages resized by r ≤1, then uses reasoning to decide whether a region needs zooming.The initial pages are downsampled from high-resolution sources before the model selects regions for closer inspection.
  • 3.1 Implementation: At each step, zoom_in selects an image, region description, and bounding box, then crops that region from the corresponding high-resolution source.The crop may be resized by zoom factor c and appended to the visual context for subsequent reasoning.
  • 3.2 Inference Cost Analysis: 7.8% to 45.0% is the modeled upper-bound range for total sequence length relative to baseline across long-document VQA parameter ranges.The range assumes r ∈ [0.25, 0.50], n(r) ∈ [1, 3], δ ∈ [0.01, 0.05], λ ∈ [0.10, 0.50], and κ ∈ [50, 200].
  • 3.2 Inference Cost Analysis: ∼32.5% of the no-resize baseline is the latency bound at r = 0.35 with two tool calls under representative parameters.The analysis models latency from input and generated-token counts, with α and β capturing prefill and decoding costs while omitting minor overheads.

4 Data Construction

The data-construction pipeline builds multi-source, multi-page, multi-hop document QA with explicit zoom-in trajectories, then filters items to create SFT and RL partitions. The resulting corpus contains 37,149 QA instances and averages 18.51 document pages overall.

  • Data sources and QA construction: The corpus combines multi-source, multi-page, multi-hop document QA with multiple question types and explicit zoom-in CoT trajectories.Documents come from six sources, including arXiv, DUDE, DocVQA, InfographicVQA, Paper2Poster, and MapTab.
  • Filtering and trajectory construction: A three-stage cascade removes prior-only and zoom-free questions before constructing zoom-in CoTs for the remaining items.The first two stages filter questions answerable at 20 DPI or at 50–100 DPI without zoom.
  • Filtering and trajectory construction: Correct CoT answers judged by GPT-5-nano become SFT data, while the remaining items are reserved for RL.This partitions successful and failed trajectory-generation outcomes into distinct training objectives.
  • Multimodal CoT: The two-agent generator uses vReasoner to request regions and vSearcher to localize them, then merges reasoning, boxes, and crops into a flat multimodal CoT.The merged trajectory serves directly as the SFT imitation target.
  • Dataset statistics: 37,149 QA instances comprise the final corpus, including 17,913 SFT trajectories and 19,236 RL prompts.The average document length is 18.51 pages overall, while SFT trajectories average 2.61 assistant CoT/tool-use rounds.

5 Experiments

Experiments evaluate InSight-doc across standard, long-document, and general high-resolution VQA, showing accuracy gains, stronger abstention, and improved accuracy–efficiency trade-offs. Analyses attribute these results to shorter contexts and more stable evidence-acquisition trajectories.

  • Document VQA: 66.9% average accuracy at r = 0.25 exceeds Qwen3-VL-8B by 16.4 points across four document VQA benchmarks.The gains are 17.2 points on DUDE, 18.3 on MP-DocVQA, 17.1 on MMLongBench-Doc, and 12.8 on LongDocURL.
  • Document VQA: 72.6% average accuracy at r = 0.5 improves over Qwen3-VL-8B by 4.3 points, while remaining competitive with proprietary models.At r = 0.25, InSight-doc outperforms all GPT variants; at r = 0.5, it remains comparable with GPT and Gemini models.
  • General high-resolution VQA: InSight-doc improves general high-resolution VQA over Qwen3-VL-8B without zoom by 7.2 and 3.7 points on MME-RealWorld-Lite, and 1.8 and 8.7 points on O3-Bench.The figures correspond to r = 0.25 and r = 0.5, respectively, for each benchmark.
  • Comparison with related methods: InSight-doc achieves 57.8% on MMLongBench-Doc and 65.6% on LongDocURL, exceeding strongest previously reported results by 15.7 and 9.3 points.These are cross-paper comparisons with differences in backbones, training data, input resolution, and evaluation protocols, so they are contextual rather than fully controlled.
  • Efficiency analysis: At 70 DPI, InSight-doc outperforms the 140-DPI baseline by 1.2 points on average while reducing latency by 54%.On the longest-document subset, it reduces latency by 71% while improving accuracy by 3.0 points; across benchmarks, it also uses substantially shorter sequences.
  • Unanswerable questions: At r = 0.25, InSight-doc improves unanswerable F1 over Qwen3-VL-8B without zoom by 24.6 points on DUDE and 25.9 points on MMLongBench-Doc.It consistently outperforms the zoom-enabled baseline, indicating stronger identification of insufficient document evidence.

6 Conclusion

InSight-doc treats visual resolution as an adaptive reasoning-time resource, selectively zooming from low-resolution pages into relevant regions. Across document and high-resolution VQA benchmarks, this coarse-to-fine approach improves accuracy while reducing hallucination, context length, and latency.

  • InSight-doc treats visual resolution as an adaptive reasoning-time resource for long-document understanding.
  • Selective zooming from low-resolution pages into relevant regions supports coarse-to-fine visual reasoning.The framework begins with low-resolution pages and inspects relevant regions at higher resolution.
  • Results across document and high-resolution VQA benchmarks demonstrate the effectiveness of coarse-to-fine visual reasoning.The reported improvements include accuracy gains alongside reductions in hallucination, context length, and end-to-end inference latency.

Limitations

The paper’s evaluation and related-work scope highlights constraints around long-document visual reasoning, including computational cost, retrieval dependence, and the limited breadth of the authors’ experimental setup.

  • The experiments use SFT+RL only with Qwen3-VL-8B-Instruct, the proposed framework, and the proposed dataset.The authors identify evaluation on additional recent models and models from other providers as a direction for more complete evaluation.
  • End-to-end methods: End-to-end high-resolution processing preserves fine-grained visual information but incurs substantial training and inference overhead from large visual-token counts.Many pages may also contain task-irrelevant content.
  • Visual retrieval-based methods: Fixed top-k visual retrieval can miss relevant evidence, remain sensitive to k, and recover poorly from initial retrieval errors.These limitations are especially relevant for multi-hop questions whose evidence falls outside the retrieved subset.
  • Coarse-to-fine methods: Doc-V⋆ primarily relies on the external Colqwen2.5 retriever, introducing retrieval errors and indexing or retrieval overhead.The cited reliance is reported as 94.0–99.8%.
  • Visual search: Earlier visual-search workflows typically depend on external detectors or scripted procedures and often operate in a single rigid search round.

B.2 Proof of Proposition 2

The proof analyzes latency for interleaved tool-use executions by comparing them with a hypothetical single-turn ordering. Prefix caching and β ≥ α establish that the actual interleaving does not increase the relevant cross-term contribution.

  • The proof models an execution at resize ratio r with m = n(r) tool calls, distinguishing initial input, tool-return inputs, intermediate responses, and the final response.U0 is the initial input; Uj are tool-call returns; Gj are generated responses, with Gm as the final response.
  • The hypothetical single-turn execution places all Pr input tokens before all Rr generated tokens before applying the latency expression.
  • The actual execution interleaves inputs and generated responses as U0, G0, U1, G1, …, Um, Gm.
  • With prefix caching, previously processed tokens are not recomputed, so the ordering difference concerns earlier generated chunks Gi paired with later tool-return chunks Uj.The hypothetical ordering contributes 2βGiUj, whereas the actual ordering contributes 2αGiUj.
  • Because β ≥ α and token counts are nonnegative, the comparison implies the stated latency bound for the interleaved execution.

C.5 RL Data Construction

The RL dataset is assembled from filtered source questions and targeted answerable or unanswerable additions, then constrained by prompt-length recovery and weighted sampling. The final RL parquet contains 19,236 rows before weighted sampling.

  • Hard questions and negative examples remaining after filtering and SFT-trajectory construction form the starting pool for RL.The pipeline applies source selection, duplicate removal, SFT-overlap removal, and prompt-length capping.
  • RL prompts are capped at 24k estimated prompt tokens, with r = 0.5 rows regenerated at r = 0.35 when they exceed the cap.Rows still exceeding the cap after recovery are dropped.
  • The final RL parquet contains 19,236 rows before weighted sampling.
  • Targeted add-ons include multiple-choice examples discouraging erroneous abstention and structured-document examples increasing coverage of structure-dependent arXiv questions.
  • Weighted refill sampling targets an 86% answerable and 14% unanswerable training-stream mixture.The sampler draws from source-specific pools without replacement until a source is exhausted, then reshuffles and refills that source.

D Dataset Statistics and Quality Analysis

The dataset combines verified active-perception trajectories for answerable and unanswerable questions, while quality analysis shows strong evidence coverage and reasonably efficient localization.

  • Dataset Statistics: 17,913 SFT trajectories comprise 14,216 answerable and 3,697 unanswerable rows, while the RL parquet contains 19,236 rows.Before weighted sampling, RL includes 10,579 answerable and 8,657 unanswerable rows; training changes the effective ratio to 86%/14%.
  • Dataset Statistics: The resize ratio controls page-image resolution before tokenization, and the prompt budget includes both text and image tokens.RL is more conservative than SFT because the 24k prompt cap recovers some r = 0.5 examples at r = 0.35.
  • Evaluation Definitions: Coverage metrics are computed only for answerable rows with recoverable evidence annotations, excluding unanswerable and evidence-unannotated examples.The region-hit threshold is τ = 0.5.
  • Trajectory Quality: 95.33% of eligible answerable trajectories hit an evidence page, and 85.00% cover at least half of an evidence box.Evidence metrics exclude unanswerable rows and rows without evidence metadata; page and region hit rates also exclude zero-crop trajectories.
  • Trajectory Quality: 65.34% of crops are evidence-region hits, with mean best crop/evidence IoU of 52.37% and average crop-union area of 14.50% of a page.These metrics assess spatial precision and the amount of page area inspected.

E.1 Evaluation Datasets

The evaluation spans multi-page, extreme-length, and out-of-distribution visual benchmarks, using macro-averaged correctness, hallucination, cost, and latency metrics alongside judge calibration.

  • Evaluation Datasets: The study evaluates six datasets covering standard multi-page VQA, documents up to 468 pages, and high-resolution natural images, maps, and composite charts.The benchmarks include DUDE, MP-DocVQA, MMLongBench-Doc, LongDocURL, MME-RealWorld-Lite, and O3-Bench.
  • Metrics: Reported benchmark metrics are macro averages, with error rate defined as 1 −accuracy and hallucination rate measured on unanswerable questions.Hallucination rate counts substantive answers instead of abstentions.
  • Metrics: Sequence length includes all text and image tokens, including tool responses, while latency includes every model call and tool execution in a trajectory.These definitions make inference cost cover the complete tool-using interaction.
  • Judge Calibration: The selected legacy-v2 judge reduces false negatives while keeping false positives low, improving the compromise between the original legacy and single-call judges.The calibration uses a 150-example manually labeled stress set containing long answers, multi-target lists, unanswerable questions, and answerable refusals.
  • Evaluation Results: On uncapped long documents, the RL model remains strongest and drops 2.2 points at r = 0.7, versus 3.7 for SFT and 4.9 for base zoom.The comparison indicates greater robustness when all pages are retained.

F.3 Detailed Comparison with Related Methods

InSight-doc differs from retrieval-assisted and coarse-to-fine methods by acquiring region-level visual evidence end-to-end without an external page retriever.

  • Method Positioning: Retrieval-based methods reduce context by selecting pages, document images, or text chunks before answer generation, whereas InSight-doc begins with the full document view.Their central question is which evidence units to retrieve; InSight-doc dynamically zooms into sub-page regions during reasoning.
  • Method Positioning: InSight-doc is retriever-free and performs multi-step region-level acquisition, while VRAG-RL remains a visual RAG/search-agent pipeline.The distinction concerns both evidence granularity and whether acquisition is end-to-end or retrieval-assisted.
  • Comparison Caveat: Cross-paper scores are for positioning rather than controlled head-to-head comparison because methods differ in backbones, data, preprocessing, resolution, budgets, and evaluation code.A fully controlled comparison is unavailable for unreleased methods such as CogDoc and Doc-V⋆.
  • Comparison with Doc-V⋆: Doc-V⋆ also starts from a low-resolution overview, but it relies mostly on an external retriever to fetch high-resolution pages.Its overview and evidence-acquisition design make it close in spirit, despite the different retrieval setting.
  • Comparison with Doc-V⋆: 94.0–99.8% of Doc-V⋆ trajectories call its external retriever, whereas 81.6–99.3% of InSight-doc trajectories acquire region-level evidence themselves.Without the retriever, Doc-V⋆ accuracy drops from 39.8% to 34.9% on MMLongBench-Doc.
  • Controlled Comparison: In a controlled proxy comparison, InSight-doc achieves higher accuracy with fewer input tokens at all comparable retrieval budgets.The proxy uses ColQwen2.5 to retrieve K ∈{8, 16, 32} pages and the same Qwen3-VL-8B-Instruct answer-generation backbone.

F.5 Additional Performance Analysis

Additional analyses show that InSight-doc retains an accuracy advantage as documents lengthen while reducing latency and sequence length, and examples illustrate sequential visual evidence acquisition.

  • Long-Document Performance: Increasing page count from 47.7 to 71.0 pages for MMLongBench-Doc and from 89.0 to 106.5 pages for LongDocURL lowers accuracy and increases sequence length for both systems.InSight-doc nevertheless remains consistently stronger on the longest-document subset.
  • Long-Document Performance: +11.6, +6.4, and +6.4 accuracy points are the gains over Qwen3-VL-8B on the longest-document subset at r = 0.25, 0.35, and 0.5.The averages cover MMLongBench-Doc and LongDocURL.
  • Accuracy-Efficiency Tradeoff: At r = 0.35, InSight-doc reaches 59.1% versus 57.5% macro accuracy while reducing latency from 29.6s to 9.3s and sequence length from 111.7k to 33.4k tokens.This comparison uses the full evaluation set against Qwen3-VL-8B at r = 0.7.
  • Accuracy-Efficiency Tradeoff: On the longest-document subset, the corresponding comparison is 56.2% versus 53.2%, with latency reduced from 39.3s to 11.2s and sequence length from 136.8k to 42.4k tokens.The reported tradeoff becomes more favorable as page count increases.
  • Qualitative Analysis: Examples depict unanswerable-question handling, sequential zoom-in refinement, focus adjustment, exploration of candidate regions, and evidence gathering across pages.The examples include generalization to natural-image visual search despite document-only training.
Loading 2608.10628v1…