PaperlayerView exact revision on arXiv

Source-linked AI summary

DR Pixel: Page-Native Deep Research with Vision-Language Models

Tiansheng Hu, Yichuan Wang, James V. Talwar, Nicholas Tomlin, Chen Zhao, Sewon Min

arXiv:2611.03097v1cs.CVcs.CLcs.IR

TL;DR

Deep research over scientific papers often reduces documents to OCR text, although important evidence appears in figures, tables, and equations. DR PIXEL retrieves and reasons directly over rendered pages, and counterfactual credit reweighting trains its agent by weighting steps according to their contribution to the answer. Page-native DR outperforms OCR-based pipelines with fewer retrieval calls, and training improves performance across benchmarks.

  • Problem

    Deep-research systems commonly reduce complex scientific papers to OCR text, despite important evidence appearing in figures, tables, and equations.

    Source: §1 · p. 1

  • Method

    DR PIXEL retrieves and reasons over rendered page images, while counterfactual credit reweighting weights training steps by their contribution to the final answer.

    Source: §1 · p. 2

  • Results

    Page-native DR consistently outperforms OCR-based pipelines across tested models while using fewer tool calls, and training improves results across five benchmarks.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Directly retrieving and reasoning over visual document pages makes complex deep research more effective and efficient within the evaluated settings.

    Source: §6 · p. 10

    A two-tool text-and-image approach still separates visual content from its surrounding text, and information loss or corruption during parsing remains.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Complex documents such as scientific papers contain rich information in tables, figures, and equations; however, existing deep research (DR) agents typically reduce papers to OCR-extracted text. We instead ask whether vision-language models (VLMs) can enable page-native deep research: directly retrieving and reasoning over rendered document page images. We introduce DR Pixel, the first open-source end-to-end multimodal DR agent that uses rendered page images as its sole retrieval and reasoning unit, operating over a corpus of 1.06M pages. To evaluate this setting, we construct DRPixelBench, a benchmark on scientific papers requiring reasoning over text, figures, tables, and equations. Surprisingly, page-native DR already works well without training, outperforming OCR-based baselines across state-of-the-art models like GPT-5.4, Gemini-3.5-Flash, and Qwen3.6-27B, while requiring fewer tool calls. We next turn to DR training. A key challenge is that teacher trajectories for agentic supervised finetuning are noisy, mixing critical reasoning and action steps with redundant or failed attempts. To address this, we introduce counterfactual credit reweighting, a fine-grained credit assignment method that weights individual reasoning and action steps by their counterfactual contribution to the final answer. Our training improves retrieval recall by about 15 points and final accuracy by over 10 points on DRPixelBench, and likewise outperforms the untrained baseline on four existing benchmarks. On out-of-distribution ViDoRe-V3, DR Pixel comes within 1.8 points of an oracle that is given the gold pages directly. Our results suggest that the conventional OCR-then-retrieve pipeline is not only unnecessary but suboptimal: directly retrieving and reasoning over visual document pages makes complex DR both more effective and more efficient.

1 INTRODUCTION

Scientific papers contain important evidence in figures, tables, and equations, but deep-research systems typically reduce them to OCR text. DR PIXEL instead retrieves and reasons over rendered pages, and the paper introduces a benchmark and training approach for this setting.

1 INTRODUCTION

  • Deep-research systems often linearize papers into text, which can distort visual structure and allow OCR errors to propagate into retrieval and reasoning.The paper frames direct page retrieval and reasoning as an alternative to this text-based pipeline.Source: §1 · p. 1
  • DR PIXEL is an open-source page-native agent that iteratively retrieves and reasons directly over 1.06M rendered scientific-paper pages.Unlike OCR pipelines, it keeps retrieval and reasoning in visual space.Source: §1 · p. 2, Figure 1 · p. 2
  • DRPIXELBENCH contains 500 questions over 1M pages, combining long retrieval-and-reasoning chains with questions requiring interpretation of figures and tables.Each question is grounded in a gold page and paired with a directly verifiable answer.Source: §1 · p. 2
  • Page-native DR improves accuracy by 5.1–8.4 points across GPT-5.4, Gemini-3.5-Flash, and Qwen3.6-27B, while using 20%–30% fewer retrieval calls than OCR-based pipelines.These results are reported without targeted training.Source: §1 · p. 2
  • With Qwen3.6-27B, counterfactual credit reweighting improves accuracy by 12.0 points (30% relative) over the untrained model and closes the gap to GPT-5.4.The method weights reasoning and action steps by their contribution to the final answer, estimated by removing each step from the trajectory.Source: §1 · p. 2

2 RELATED WORK

Prior deep-research work often targets open-web search, while page-native document retrieval has largely focused on shallow retrieval. DR PIXEL instead addresses sustained search and reasoning over complex document collections, with step-level credit assignment for training.

Deep research over documents

  • Multimodal deep research often retrieves text and images through separate tools, although complex document questions may require interpreting both on the same page.The authors argue that separate interfaces are suboptimal for this setting.Source: §2 · p. 3

Page-native document retrieval

  • Page-native document retrieval has expanded to answer generation and iterative retrieval, but benchmarks such as ViDoSeek and SlideVQA often need only shallow search.On ViDoSeek and SlideVQA, one retrieval step already recalls the gold page in the top five for more than 90% of questions.Source: §2 · p. 3

Agent training

  • Outcome-based RL assigns one reward to a rollout and SFT imitates every teacher step, whereas this paper reweights agentic steps by counterfactual contribution to answer quality.Contribution is estimated by removing a step and measuring the resulting change in answer quality.Source: §2 · p. 3

3 DRPIXELBENCH

Existing benchmarks separate text-based deep research from multimodal document question answering, leaving a gap for questions that require both sustained retrieval and visual evidence. DRPIXELBENCH is designed to combine those demands with questions and answers that can be checked directly.

DRPIXELBENCH

  • Existing text-based deep-research benchmarks are text-only, while multimodal document QA often uses templated questions requiring only one or a few retrieval steps.The paper identifies this separation as the gap motivating a benchmark for complex document research.Source: §3 · pp. 3–4
  • DRPIXELBENCH is the only benchmark described as combining iterative search over a large corpus, multimodal evidence, non-templated questions, and verifiable short answers.The benchmark design targets four properties: agentic search, multimodal evidence, non-templated questions, and directly verifiable answers.Source: Table 1 · p. 4, §3 · p. 4
  • DRPIXELBENCH shares a corpus of 99,055 papers with AirQA, M3SciQA, and IRPAPERS.The corpus combines arXiv and open-access conference papers with papers from those three benchmarks.Source: §3 · p. 4
  • The benchmark contains 500 QA pairs built from scientific papers, with questions classified as Text or Visual according to whether answering requires running text or a table, figure, or equation.Question generation uses facts extracted from the full paper, followed by automatic checks and human review.Source: §3 · p. 4

4 DR PIXEL INFERENCE

DR PIXEL preserves the ReAct loop while replacing text retrieval with search over rendered paper pages that a vision-language agent reads directly. Without targeted training, it outperforms OCR-based pipelines across three models, with fewer retrieval calls; results also identify retrieval and reasoning gaps for Qwen3.6-27B.

4.1 INFERENCE METHOD

  • The agent follows a ReAct loop in which each turn produces a thought and either a search query or a final answer, using previous thoughts, actions, and search results as history.A search tool returns results from a fixed corpus; the trajectory ends when the agent issues a final answer.Source: §4.1 · pp. 4–5

4.2 RESULTS

  • DR PIXEL consistently outperforms both OCR-based settings across all three models, improving accuracy over the text-and-figure pipeline by 5.6, 5.1, and 5.2 points for GPT-5.4, Gemini-3.5-Flash, and Qwen3.6-27B, respectively.It reduces retrieval calls by 20% to 30%; all compared pipelines use no training.Source: Table 2 · p. 5, §4.2 · p. 6

4.1 INFERENCE METHOD

  • Text-based retrieval can lose or corrupt document information, while adding a separate image-search tool still separates visual content from its explanatory text and retains parsing loss.Tables and equations may be parsed incorrectly, figures may be reduced to captions or omitted, and downstream reasoning cannot recover information discarded during parsing.Source: §4.1 · p. 5
  • DR PIXEL replaces text retrieval with a VLM-based search over rendered page images, which are passed directly to a VLM agent for iterative reasoning and answering.The ReAct loop is preserved; pages are rendered and embedded offline, and the query is mapped into the same embedding space to retrieve top-k page images.Source: §4 · p. 4, §4.1 · p. 5

4.2 RESULTS

  • DR PIXEL achieves the highest GPT-5.4 accuracy on questions requiring tabular and figure evidence, and remains on par for questions grounded in text or equations.Figure 2 breaks accuracy down by evidence source across the two OCR-based settings and DR PIXEL.Source: Figure 2 · p. 6, §4.2 · p. 6
  • 11.8 points: Qwen3.6-27B trails GPT-5.4 in DR PIXEL accuracy, with a 9.4-point lower gold-page retrieval rate and larger reasoning gaps when gold pages arrive earlier.Qwen3.6-27B has duplicate queries in 5.3% of searches and at least one duplicate in 21.6% of trajectories; its accuracy gap is 5.1 points when the gold page comes from the latest search and 12.3 points when it was retrieved earlier.Source: §4.2 · p. 6, Table 3 · p. 6

5 DR PIXEL TRAINING

DR PIXEL training uses counterfactual credit reweighting to emphasize useful reasoning and search steps. The trained agent improves DRPIXELBENCH performance and transfers to external benchmarks, with ablations examining corpus size, resolution, and trajectory type.

5.1 TRAINING METHOD

  • Counterfactual credit reweighting scores reasoning and search steps by their contribution to gold-answer support, then assigns rank-based weights and masks low-value steps.Reasoning steps are also checked against the likelihood gain of their corresponding actions; qualifying steps can recover a median positive weight if previously masked for negative credit.Source: §5.1 · pp. 6–7, Figure 3 · p. 7

5.2 RESULTS

  • DR PIXELTrained raises DRPIXELBENCH accuracy from 40.1 to 52.1, slightly surpassing GPT-5.4 while retrieving gold pages more often with fewer searches.Results are averaged over three runs.Source: Table 4 · p. 8, §5.2 · p. 8
  • Vanilla SFT requires 57% more searches and exhausts the 20-call budget on 35.8% of questions, versus 9.8% for DR PIXELTrained.DR PIXELTrained also exhausts the budget less often than GPT-5.4 (17.8%) and the untrained backbone (19.2%).Source: §5.2 · p. 8
  • DR PIXELTrained improves accuracy over the untrained backbone by 9.6 to 14.4 points across IRPAPERS, M3SciQA, and AirQA.Its search effort decreases on IRPAPERS and AirQA but increases on M3SciQA.Source: §5.2 · p. 8, Table 5 · p. 8
  • On ViDoRe-V3, post-training improves both page- and text-based retrieval by 4.1 points and remains competitive with the Page Oracle.The two retrieval settings remain close, as only 1.7% of questions genuinely require visual content.Source: §5.2 · pp. 8–9, Table 6 · p. 9

5.3 ABLATION

  • As the corpus grows from 20,000 to 99,000 papers, DR PIXELTrained's page-hit rate drops 8.0 points, versus 12.5 for GPT-5.4.Its retrieval advantage over GPT-5.4 widens from 1.8 points to 6.3 points.Source: §5.3 · p. 9, Figure 4 · p. 9
  • Capping pages at 1.05 megapixels reduces input tokens by 30% to 48% across models while lowering accuracy by 1.0 to 3.8 points.Output tokens rise slightly, but account for less than 25% of all tokens; the cited accounting counts each token once in the final context.Source: §5.3 · pp. 9–10, Figure 5 · p. 9
  • Visual-trajectory fine-tuning reaches 50.2 accuracy versus 46.0 for Text fine-tuning, and transfers better across question types.It stays within 1.4 points of Text fine-tuning on Text questions, while Text fine-tuning trails by 6–11 points on Visual questions.Source: §5.3 · p. 10, Table 7 · p. 10, Figure 6 · p. 10

6 CONCLUSION

DR PIXEL replaces OCR-first research over complex documents with retrieval and reasoning over rendered pages, and reports advantages in performance, efficiency, and transfer.

6 CONCLUSION

  • DR PIXEL retrieves and reasons directly over rendered pages instead of beginning with OCR.The paper says this requires document rendering rather than format-specific parsing.Source: §6 · p. 10
  • Page-native deep research consistently outperforms OCR-based pipelines across tested models while using fewer tool calls, even without task-specific training.The benchmark contains 500 verifiable questions over 1.06M pages of scientific literature.Source: §6 · p. 10
  • Counterfactual credit reweighting brings a 27B model to GPT-5.4-level accuracy with lower search cost, and gains transfer to external benchmarks and non-scientific documents.The conclusion notes potential appeal for private, layout-rich financial or legal corpora where OCR and parsing pipelines are costly and brittle.Source: §6 · p. 10

AI USAGE STATEMENT

The appendix records AI usage and details benchmark construction, annotation, corpus composition, deployment, and retrieval-budget comparisons.

AI USAGE STATEMENT

  • Gemini-3.5-Flash generated and filtered benchmark questions, GPT-5.4 produced training trajectories, and GPT-5.4-mini judged answers; every benchmark question received human review.LLM assistants also supported literature search, writing polish, and coding; the authors attribute research ideas, experimental design, and analysis to themselves.Source: AI USAGE STATEMENT · p. 10

A.1 OCR PARSING DETAIL

  • The page-native corpus uses full-page images rendered at 150 dpi, while the text corpus uses 256-word chunks and the figure corpus uses cropped figure and chart images.All three corpora use the same PDFs; the page-native corpus skips parsing.Source: §A.1 · p. 18

A.2 DRPIXELBENCH STATISTICS BREAKDOWN

  • Every DRPIXELBENCH question is grounded to exactly one page, and its statistics group questions by evidence modality.Source: Table 8 · p. 18

A.3 DRPIXELBENCH VERIFICATION DETAIL

  • The drafted questions pass through four named filters, with Table 9 reporting the share remaining after each filter.Source: Table 9 · p. 19

A.4 DRPIXELBENCH CORPUS DETAIL

  • The corpus contains 99,055 documents and 1,062,153 pages, with 93.0% of documents appearing at a peer-reviewed venue.Source: Table 10 · p. 19

A.5 DRPIXELBENCH ANNOTATION

  • Annotators checked question validity, answer concision for reliable grading, and whether the gold page was correct and sufficient; they could edit or reject questions.About one in ten questions passed unchanged and about half were rejected, commonly for contradictory answers, insufficient single-page evidence, or overly open-ended answers.Source: §A.5 · p. 20

B.1 DEPLOYMENT AND ENGINEERING DETAIL

  • Deployment used identical prompts, tools, and search budgets across models; retrieved pages returned as images, and prior observations remained available across turns.The append-only prefix achieved a reported 96% multimodal cache hit rate.Source: §B.1 · pp. 20–21

B.2 COMPARISON ON RETRIEVAL BUDGET

  • Three retrieved pages contained 1.08× the text of five retrieved chunks on average, supporting the authors’ claim of a comparable text budget per search.Page text was obtained by OCR for this comparison.Source: §B.2 · p. 21, Table 13 · p. 21

C.1 DR PIXELTRAINED TRAINING DETAIL

This appendix details the training setup and data, then describes how training changes retrieval behavior relative to GPT-5.4 and the base model.

C.1 DR PIXELTRAINED TRAINING DETAIL

  • Counterfactual credits are computed offline against the frozen base policy, assigned separately to reasoning and action messages, and applied as per-message loss weights during ordinary SFT.Observation tokens are excluded from the loss in every training arm.Source: §C.1 · p. 21
  • The 2,500 training trajectories came from 1,164 visual seed questions, with six GPT-5.4 rollouts sampled per question and incorrect-answer trajectories discarded.Retained trajectories were limited to the three shortest per question; papers supporting training questions were disjoint from evaluation papers.Source: §C.1 · p. 21

C.2 DR PIXELTRAINED BEHAVIOR COMPARISON WITH GPT-5.4 AND DR PIXELBASE

  • Training reverses the retrieval gap to GPT-5.4, while the remaining reading deficit is confined to gold pages retrieved on earlier searches.Source: Table 14 · p. 22

C.3 DR PIXELTRAINED TRAINING ABLATION

The ablation adds training components incrementally and reports fewer searches alongside gradual final-accuracy benefits.

C.3 DR PIXELTRAINED TRAINING ABLATION

  • Each added component removes searches while gradually benefiting final accuracy.Source: Table 15 · p. 22, §C.3 · p. 22
  • The full DR PIXELTrained method combines a 1.5 final-answer weight, rank-converted per-step counterfactual credits, and cross-channel correction.Source: §C.3 · p. 22, Table 15 · p. 22
  • Searches per question fall from 14.0 under uniform weighting to 8.9 under the full method.The table caption identifies tool/q as searches per question.Source: §C.3 · p. 22, Table 15 · p. 22

C.4 OTHER VARIATION FOR Counterfactual Credit Reweighting

The ablations test how credit weights are normalized while holding credits, masks, and the training recipe fixed. Rescaling weights harms reading accuracy, changing the ranking pool does not help, and equalizing trajectory mass performs worst.

Overall comparison

  • Normalization ablations show that rescaling weights reduces reading accuracy, alternate ranking pools do not help, and equalizing trajectory mass performs worst.Table 16 keeps the credits, mask, and training recipe fixed across variants.Source: Table 16 · p. 23, §C.4 · p. 23

Weight magnitude

  • Matching mean weights to the uniform baseline lowers accuracy given the gold page from 68.7 to 63.1, while barely changing search counts.Masking non-positive-credit steps removes about a quarter of steps and leaves mean weight roughly a third below the uniform baseline.Source: §C.4 · p. 23

Rank support

  • Whole-training-set ranking leaks question difficulty into step weights, but within-trajectory ranking stays near the reference and within-question ranking is 2.3 points lower.Mean surviving trajectory weight ranges from 0.65 at the tenth percentile to 1.70 at the ninetieth.Source: §C.4 · p. 23

Trajectory contribution

  • Equalizing total weighted mass across trajectories is the weakest variant, five points below the reference, because lightly masked trajectories receive a large multiplier.Trajectory mass spans about a factor of six between the tenth and ninetieth percentiles.Source: §C.4 · p. 23

Interpretation

  • The ablations indicate that credit’s effect lies in which steps are masked and how masking is distributed, not numerical rescaling or rank support.Rescaling either has no effect or degrades reading, while changing the ranking pool has no effect on results.Source: §C.4 · p. 23

C.5 Counterfactual Credit Reweighting MECHANISM EXPLANATION

Counterfactual credit reduces search effort while preserving retrieval by preferentially retaining evidence-finding searches and pruning later continuation. Its masking differs across reasoning and search steps, with longer trajectories losing more search steps.

Stopping behavior

  • DR PIXELTrained uses 8.9 searches per question versus 14.0 for vanilla SFT at the same page hit rate, with stopping behavior driving the difference.By the fifth search, the trained model has finished a third of questions while vanilla SFT has finished almost none.Source: §C.5 · p. 24, Figure 8 · p. 24
  • Vanilla SFT stops with probability 4.4% per step, versus 10.2% for DR PIXELTrained, leaving vanilla’s expected 22.8-search trajectory beyond the 20-call budget.The trained model’s stopping probability is close to the untrained backbone’s 9.4% and GPT-5.4’s 8.9%.Source: §C.5 · p. 24, Figure 8 · p. 24

Channel and length effects

  • Credit masks 62.6% of reasoning steps but 24.5% of search steps; search masking rises from 0.2% on one-search trajectories to 40.2% at the budget.Reasoning masking stays between 51% and 68% across trajectory lengths; its small slope is attributed to cross-channel correction.Source: §C.5 · pp. 24–25

Evidence-finding search

  • The search that first retrieves a page cited by the final answer is masked only 2.5% of the time, while other searches are masked 15%–42%.Among surviving steps, the evidence-finding search has weight 1.25; its position is not supplied to the method.Source: Figure 7 · p. 24, §C.5 · p. 25

D.1 SUPPLEMENT MATERIAL FOR ABLATION

The supplement reports a resolution trade-off and describes prompts used for evaluation and benchmark construction. The prompts define system behavior and grading, extract modality-specific facts, and generate questions from abstract-level expectations.

D.1 SUPPLEMENT MATERIAL FOR ABLATION

  • Lowering page resolution cuts input tokens by 30%–48% for a 1–4 point accuracy drop, while remaining 2.7–7.4 points more accurate than the OCR-based Text setting.The figure plots accuracy against input and output tokens as page resolution varies.Source: Figure 9 · p. 25

E PROMPTS

  • The system prompt is shared across evaluated models with runtime-filled tools, observations, and a 20-call budget; the grading prompt uses three GPT-5.4-mini votes.Empty predictions are marked incorrect, while page-hit and document-hit metrics use exact identifier matches against gold annotations.Source: Figure 10 · p. 26, Figure 11 · p. 27
  • Atomic fact extraction uses image input for figures, OCR cell text for tables, LATEX source for equations, and chunks for body text.Caption and nearby-text flags identify facts unavailable from surrounding prose.Source: Figure 12 · p. 28
  • Question generation uses the abstract enriched with atomic facts, not the paper body, before later prompts produce a precise question, short gold answer, and researcher-style wording.This setup drives questions from what a reader might wonder rather than evidence already in view.Source: Figure 13 · p. 28

Related papers

Loading 2611.03097v1…