PaperlayerView exact revision on arXiv

Source-linked AI summary

Information-Gain Calibrated On-Policy Distillation for Deep Search Agents

Chenxi Li, Xueyang Feng, Songyang Liu, Chenxu Wang, Xiaoying Tang, Chaozhuo Li, Litian Zhang

arXiv:2610.20664v1cs.IR

TL;DR

In deep search, standard OPD scores student queries from pre-retrieval context alone, although query usefulness depends on the evidence retrieved. IG-OPD uses post-retrieval information gain to calibrate turn budgets and query preferences. Across six multimodal search benchmarks, it improves average accuracy over OPD for 2B and 4B students, and the 4B student surpasses its 8B teacher on MMSearch and BrowseComp-VL.

  • Problem

    Standard OPD supervises each student action from preceding context without accounting for subsequent environmental feedback, even though query usefulness depends on retrieved evidence.

    Source: §1 · pp. 1–2

  • Method

    IG-OPD uses post-retrieval information gain to recalibrate turn-supervision budgets by information criticality and query preferences by evidence utility.

    Source: §1 · p. 3, §4.3 · p. 5, §4.4 · p. 6

  • Results

    Across six multimodal search benchmarks, IG-OPD improves average accuracy over OPD by 3.4 and 1.7 percentage points for 2B and 4B students, respectively, and the 4B student surpasses its 8B teacher on MMSearch and BrowseComp-VL.

    Source: §1 · p. 3

  • Takeaways & Limitations

    IG-OPD’s evidence-aware calibration is reported to improve search accuracy without adding inference overhead.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) has recently emerged as a promising approach for transferring the capabilities of large agents to compact students. In deep search, where agents iteratively retrieve external evidence, a teacher's preference among plausible queries before retrieval may not predict which queries will return useful evidence. Standard OPD nevertheless supervises each student query from the preceding context alone, inducing a failure mode we term \textbf{\textit{evidence-blind supervision (EBS)}}. To understand this, we regroup the OPD objective by interaction turns and reveal two implicit priors: a \textit{budget prior} that allocates supervision by token volume, and a \textit{direction prior} that favors tokens preferred by the teacher, neither of which accounts for the evidence returned by the current query. Our empirical analysis shows that both priors contribute to EBS in practice: the first search turn accounts for 51.1\% of total absolute information gain but receives only 18.0\% of the supervision budget, while 43.7\% of search queries have teacher rewards that disagree in sign with post-retrieval information utility. To mitigate this, we propose \ours, which uses post-retrieval information gain (IG) to calibrate both priors: reallocating supervision toward information-critical turns while calibrating query preferences using the utility of retrieved evidence. Experiments on six multimodal search benchmarks show that \ours improves average accuracy by 3.4 and 1.7 percentage points for 2B and 4B students over standard OPD, enabling the 4B student to surpass its 8B teacher on MMSearch and BrowseComp-VL.

1 INTRODUCTION

Deep-search agents need external evidence, but standard OPD scores student queries from pre-retrieval context, creating evidence-blind supervision (EBS). IG-OPD uses post-retrieval information gain to recalibrate turn budgets and query preferences, improving student accuracy across multimodal search benchmarks.

1 INTRODUCTION

  • EBS occurs when a query’s teacher-based score ignores the evidence it retrieves, potentially penalizing a useful query that returns gold evidence.The paper’s example contrasts the student’s evidence-retrieving query with a teacher-preferred query that retrieves no useful evidence.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The first search turn accounts for 51.1% of total information criticality but receives only 18.0% of OPD’s token budget.The budget prior allocates supervision by token volume, while the direction prior favors teacher-preferred queries regardless of retrieved evidence utility.Source: §1 · p. 2
  • The paper identifies EBS as a failure mode in deep search where supervision assigned before retrieval can under-credit information-critical turns and penalize useful queries.Plausible queries can differ in wording, language, or retrieval path, while their usefulness depends on the evidence actually retrieved.Source: §1 · p. 2
  • Across six multimodal search benchmarks, IG-OPD improves average accuracy over OPD by 3.4 and 1.7 percentage points for 2B and 4B students, respectively.The 4B student also surpasses its 8B teacher on MMSearch and BrowseComp-VL, with zero inference overhead.Source: §1 · p. 3
  • IG-OPD uses post-retrieval information gain to recalibrate the two OPD priors: ITC reallocates turn budgets by information criticality, and IQC calibrates query preferences by evidence utility.The approach is presented as evidence-aware supervision using a lightweight post-retrieval signal.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work improves multimodal search with RL and refines OPD supervision for instability, uncertainty, and multi-turn errors. The paper characterizes these approaches as still evidence-blind in multi-hop search and positions IG-OPD as calibrating supervision using post-retrieval utility.

2 RELATED WORK

  • Multimodal search systems commonly use RL for multi-step retrieval and reasoning, but sparse outcome rewards complicate credit assignment and make training costly.Examples include GRPO-trained MMSearch-R1 and WebWatcher and OpenSearch-VL’s multi-hop data and safeguards against cascading tool errors.Source: §2 · p. 3
  • OPD research addresses training instability, indiscriminate imitation, and multi-turn compounding errors through reward adjustments, selective supervision, and adaptive guidance.Other approaches also combine distillation with RL objectives.Source: §2 · p. 3
  • The paper argues that these strategies remain evidence-blind in multi-hop search because they supervise actions without considering generated queries’ post-retrieval utility.IG-OPD uses information gain to calibrate turn budgets and update directions against realized evidence utility.Source: §2 · p. 3

3 PRELIMINARIES

The paper formulates search as a student-generated sequence of deliberation, tool calls, observations, and a final response. Standard OPD applies teacher supervision to student-generated tokens, but its turn-level objective does not consult the evidence returned by each turn.

3.1 PROBLEM FORMULATION: MULTI-TURN SEARCH AGENTS

  • A multi-turn search rollout alternates intermediate deliberation, tool calls, and returned evidence before a final response synthesizes the collected evidence.The formulation also allows an optional image alongside the textual question.Source: §3.1 · p. 3
  • Each intermediate turn contains a chain of thought, a tool call, and its resulting evidence; the final turn produces a response from the collected evidence.Training losses are computed only on tokens generated by the student policy, not externally produced observations.Source: §3.1 · p. 3

3.2 ON-POLICY DISTILLATION

  • OPD trains a student on its own rollouts using a frozen teacher’s sampled-token log-likelihood ratio objective.The objective is averaged over student-generated tokens.Source: §3.2 · pp. 3–4
  • Regrouping OPD by interaction turn reveals implicit supervision priors, while the per-token distillation signal depends on the prefix and not the returned observation.This omission of post-retrieval evidence gives rise to EBS.Source: §3.2 · p. 4

4 INFORMATION-GAIN CALIBRATED ON-POLICY DISTILLATION

IG-OPD addresses evidence-blind supervision by using post-retrieval information gain to recalibrate OPD’s turn budgets and query preferences. The section diagnoses mismatches between standard OPD supervision and evidence utility, then presents ITC, IQC, and their combined training objective.

4.1 DIAGNOSING EVIDENCE-BLIND SUPERVISION WITH INFORMATION GAIN

  • 51.1% of total turn information criticality comes from the first search turn, yet it receives 18.0% of the budget; 43.7% of search turns have reward and gain signs that disagree.The final response receives 47.6% of the budget despite contributing zero information gain; 19.8% of queries retrieve useful evidence but are penalized, while 23.9% retrieve misleading evidence but are rewarded.Source: Figure 2 · p. 4, §4.1 · pp. 4–5

4.2 INFORMATION-GUIDED TURN-CREDIT CALIBRATION

  • ITC replaces token-volume weighting with a per-turn budget shaped by absolute information gain, while retaining a baseline share for every turn.The tanh limits the influence of extreme gains; setting λ = 0 gives equal per-turn budgets, and the default is λ = 0.1.Source: §4.2 · p. 5

4.3 INFORMATION-GUIDED QUERY-PREFERENCE CALIBRATION

  • IQC applies signed post-retrieval gain to action-critical query tokens, increasing their OPD signal for useful evidence and decreasing it for harmful evidence.The correction is divided evenly across tokens in the tool call, avoiding length-induced token bias.Source: §4.3 · p. 5
  • The IQC correction can be interpreted as tilting the teacher’s trajectory preferences with a confidence pseudo-likelihood of the gold answer.The practical objective uses ITC-calibrated budgets, a length-normalized pseudo-likelihood, and macro-action-level correction.Source: §4.3 · p. 6

4.4 OVERALL OBJECTIVE

  • IG-OPD treats calibrated advantages as fixed token-level supervision signals and optimizes the student with a policy-gradient surrogate.Computing the turn gain and query utility requires one additional lightweight teacher scoring pass per turn, but no additional rollouts.Source: §4.4 · p. 6

5 EXPERIMENT

Across six multimodal search benchmarks, IG-OPD improves accuracy over OPD for both student sizes, and the 4B student surpasses its 8B teacher on two benchmarks. Ablations and diagnostics show how its calibrations redistribute supervision, improve evidence retrieval, and depend on search position and training stage.

5.1 EXPERIMENT SETUP

  • Evaluation covers six multimodal web-search benchmarks spanning single-hop lookup, multi-hop retrieval, and long-horizon search, with accuracy judged against reference answers by GPT-4o.The teacher is OpenSearch-VL-8B; Qwen3-VL-2B/4B students are cold-started on 10K trajectories before OPD training on 4K samples, and training and evaluation use a live web environment.Source: §5.1 · pp. 6–7

5.2 CAN IG-OPD TRANSFER DEEP-SEARCH CAPABILITY TO COMPACT STUDENTS?

  • 3.4 points at 2B (49.4 vs. 46.0) and 1.7 points at 4B (55.0 vs. 53.3) are IG-OPD’s average-accuracy gains over OPD; the 4B student surpasses its 8B teacher on MMSearch and BrowseComp-VL.IG-OPD ranks first on five of six benchmarks in both student groups.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 WHICH CALIBRATION MATTERS?

  • On MMSearch at 2B, combining ITC and IQC reaches 62.0, while uniform budgeting reaches 58.3 and IQC alone reaches 56.7.Information-magnitude budgeting adds 1.7 points over uniform budgeting; IQC added after ITC contributes 2.0 points, supporting complementary roles for budget and query-direction calibration.Source: §5.3 · p. 8, Table 2 · p. 8
  • On MMSearch at 2B, accuracy rises to 62.0 at αmax = 10, then falls to 59.7 at αmax = 15; λ values from 0.1 to 0.3 outperform the uniform budget.At λ = 0.3, the most-to-least credited turn ratio is bounded by 1.3; at αmax = 10, IQC correction is roughly half of the OPD signal.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 DOES IG-OPD RESOLVE EVIDENCE-BLIND SUPERVISION?

  • ITC assigns 80.9% of supervision to four search groups, each receiving 19.5%–20.8%, while the response turn retains 19.1%.Under the token-volume prior, the response turn absorbs 47.6% and the third search turn receives 7.2%; ITC orders search groups by realized information magnitude, with Search 1 highest.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
  • By step 40, IG-OPD raises gold-evidence hit rates from 12.9% to 16.9% on the first search and from 34.1% to 39.1% on any search, while searches per rollout stay essentially unchanged.Training time per step rises from 397 to 443 seconds (+11.6%) and peak GPU memory from 68.1 to 75.3 GB (+10.6%); inference has no added overhead.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 IS POST-RETRIEVAL IG RELIABLE, AND WHY RAMP IT?

  • IG agreement with final-answer correctness is 82.2% on the first search and 70.0% on later searches; positive first-search IG predicts success at 87.5%, while negative later-search IG predicts failure at 85.7%.The top 30% largest gains raise both rates to about 90%; a cosine ramp avoids early search suppression, preserving 5.78 turns versus 5.29 without ramping over the first 40 training steps.Source: §5.5 · p. 9, Table 3 · p. 9

6 CONCLUSION

IG-OPD addresses evidence-blind supervision in deep search with two information-gain calibrations. Across six OOD multimodal search benchmarks it outperforms existing distillation and reinforcement-learning baselines, and its 4B student can surpass an 8B teacher.

6 CONCLUSION

  • IG-OPD addresses evidence-blind supervision with ITC, which reallocates supervision by localized information gain, and IQC, which aligns query updates with retrieved-evidence utility.The framework is presented as a response to structural mismatches in traditional OPD.Source: §6 · pp. 9–10
  • Across six OOD multimodal search benchmarks, IG-OPD significantly outperforms existing distillation and reinforcement-learning baselines; its 4B student can surpass its 8B teacher without inference-time overhead.The paper identifies MMSearch and BrowseComp-VL as benchmarks where the 4B student surpasses the teacher.Source: §6 · p. 10
  • The authors identify extending post-retrieval evidence calibration to other environments as a promising direction for scaling autonomous web agents.Source: §6 · p. 10

AI USE STATEMENT

The authors used generative AI tools for literature discovery, development, analysis, figure and table preparation, and language editing, while retaining responsibility for the manuscript.

AI USE STATEMENT

  • Generative AI supported literature discovery, code development and debugging, exploratory analysis, figure and table preparation, and language editing.Source: AI USE STATEMENT · p. 10
  • The authors independently verified the relevant literature, implementations, numerical results, manuscript content, and accompanying artifacts.Source: AI USE STATEMENT · p. 10
  • The authors state that they made the methodological choices, scientific claims, and conclusions and take full responsibility for the final manuscript and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper directs readers to appendices for algorithm, environment, training, and data-construction details, and to the main benchmark section for evaluation protocols.

REPRODUCIBILITY STATEMENT

A METHOD DETAILS AND DERIVATIONS

The appendix derives OPD's turn-wise formulation and describes how IG-OPD calibrates supervision budgets and query updates. It also gives the gradient interpretation, a power-posterior view, and the complete algorithm.

A.1 TURN-WISE DECOMPOSITION OF OPD

  • OPD's sampled-token objective uses student-generated tokens to estimate teacher–student log-probability gaps, then regroups the objective by turns into token-volume budget and teacher-preference direction priors.Tool observations enter later prefixes as context but do not receive token loss.Source: §A.1 · p. 14, N , ¯DOPD · p. 14

N , ¯DOPD

  • Because OPD fixes its budget and query direction before retrieval evidence arrives, IG-OPD substitutes an information-based turn budget and adds a signed post-retrieval correction to search-action tokens.When λ = 0, the normalized budget assigns every turn the same aggregate OPD budget, regardless of generated-token count.Source: N , ¯DOPD · p. 15

A.2 POLICY-GRADIENT SURROGATE AND GRADIENT FLOW

  • The policy-gradient surrogate applies the calibrated turn budget across generated tokens and the signed evidence correction only to search-decision tokens; clipping bounds each token's contribution.Retrieval, reference-answer scoring, and budget computation remain outside the autograd graph.Source: §A.2 · pp. 15–16

A.3 POWER-POSTERIOR INTERPRETATION

  • The power-posterior interpretation keeps the teacher trajectory distribution as a prior and tilts it by post-retrieval answer potential, with the evidence contribution increasing as the cosine schedule rises.At αk = 0, the target is the teacher trajectory distribution.Source: §A.3 · pp. 16–17

A.4 COMPLETE IG-OPD ALGORITHM

  • The algorithm samples tool-interleaved rollouts, scores generated tokens, computes information gains and normalized turn budgets, masks search-action tokens, and updates the student with calibrated scores.The frozen teacher scores reference-answer tokens under the initial context and each post-observation history; IQC masks select parsed tool-name and supported argument-value spans.Source: §A.4 · p. 17

B.1 AGENT ENVIRONMENT AND TEACHER TRAINING

The agent operates through multi-turn tool calls and observations, combining web retrieval with visual and document-processing tools. Before distillation, the teacher is trained with GRPO rewards that cover answer correctness, search quality, execution, and efficiency.

B.1 AGENT ENVIRONMENT AND TEACHER TRAINING

  • The shared environment supports action–observation interaction, with tool results appended to the trajectory and image-processing tools able to operate on stable identifiers for original or transformed images.Trajectories allow at most one tool per turn and up to 20 turns.Source: §B.1 · p. 18, Table 4 · p. 18
  • Text retrieval returns query-focused evidence summaries, while reverse visual lookup can identify entities before factual text searches.Text retrieval combines Serper results, Jina page extraction, and query-conditioned summarization; observations retain source URLs.Source: §B.1 · p. 18
  • The visual stack extracts structured document content and supports local image transformations, including cropping, perspective correction, sharpening, and super-resolution.Document parsing returns recognized text, Markdown, reading order, and typed layout blocks.Source: §B.1 · pp. 18–19
  • Teacher GRPO jointly rewards final-answer correctness, evidence relevance and query progression, valid interaction formatting, and search efficiency.The efficiency factor discounts redundant retrieval after the fifth search without making search count the dominant objective.Source: §B.1 · p. 19
  • The teacher is trained with groups of eight trajectories per prompt, and its update-60 checkpoint is frozen for subsequent OPD and IG-OPD training.Teacher rollouts use temperature 0.7 and top-p = 1.Source: §B.1 · p. 19

B.2 OPD DATA CONSTRUCTION

The OPD corpus combines search-intensive multimodal questions with cached reference answers and reusable initial visual-search results. The section also specifies shared training configurations, agent instructions, answer judging, and tool interfaces.

B.2 OPD DATA CONSTRUCTION

  • The 3,840-prompt corpus is sampled without replacement at a fixed 70:15:15 ratio from Search-VL-RL-8K, VDR-Bench-2K, and the FVQA training split.The sources respectively broaden open-domain visual search, document-centric perception, and fact-oriented retrieval coverage.Source: §B.2 · pp. 19–20, Table 5 · p. 20
  • Reference answers support outcome-reward baselines and frozen-teacher probes, while cached initial visual-search responses stabilize paired runs without fixing later student-generated queries.The dataset builder validates answer-cache coverage and creates a deterministic, identifier-verified sample.Source: §B.2 · p. 20

B.3 TRAINING CONFIGURATIONS

  • OPD and IG-OPD share the student initialization, frozen teacher, rollout engine, optimizer, and distributed configuration, while training optimizes only student-generated action tokens.The 2B and 4B runs start from corresponding update-154 SFT checkpoints.Source: §B.3 · pp. 20–21, Table 6 · p. 21

C.1 SYSTEM PROMPT

  • The system prompt directs agents to inspect and enhance images with tools, chain operations when needed, and use text search to verify facts not visible in the image.Each action is preceded by a reasoning block, and the agent waits for the observation before choosing its next action.Source: §C.1 · p. 22

C.2 JUDGE PROMPT

  • A GPT-4o judge compares each complete report with the reference answer and returns a binary correctness decision, accepting equivalent entities, units, dates, and wording.Outputs that miss the required format are resubmitted with a formatting reminder.Source: §C.2 · p. 23

C.3 TOOL SCHEME

  • The tool scheme exposes document parsing, visual search, text search, and image operations through JSON function calls, with visual tools returning image identifiers and retrieval tools returning text observations.The web-search entry is a compatibility alias for text search, and the rollout parser canonicalizes query→q and lang→hl.Source: §C.3 · pp. 23–25

D CASE STUDY

Four successful 4B IG-OPD trajectories illustrate multimodal search, image processing, and retrieval chains across benchmark tasks. The paper also describes the evaluation as read-only and limited to public information and benchmark images.

D CASE STUDY

  • Four successful 4B IG-OPD trajectories cover visual identification, document understanding, image enhancement, and cross-entity multi-hop retrieval.The examples include identifying NASA’s VIPER and verifying its cancellation date, and identifying Myeongdong Night Market through cropping, layout parsing, visual search, and textual verification.Source: D CASE STUDY · pp. 26–27, Figure 6 · p. 26, Figure 7 · p. 26

E ETHICS STATEMENT

  • The experimental agent operates read-only, retrieving public information and processing benchmark images without side effects on external systems or collecting private personal data.The authors report preserving source URLs for traceability and following applicable dataset and service terms.Source: E ETHICS STATEMENT · p. 28

Related papers

Loading 2610.20664v1…