PRInTS: Reward Modeling for Long-Horizon Information Seeking

• Existing PRMs assess isolated, short reasoning units with coarse feedback, although information-seeking step quality also depends on tool interactions and their interpretation.Relevant dimensions include tool-output interpretation, tool-call informativeness, and the next-action plan.Source: §1 · p. 1, Figure 1 · p. 2
• Growing tool responses make trajectory context difficult to manage, motivating compact summaries instead of evaluation over the full history.Accumulated reasoning, tool calls, and tool outputs can produce noisy evaluations.Source: §1 · p. 2
• PRInTS scores complete reasoning-plus-tool steps across multiple quality dimensions, using information-gain estimation to provide dense step-level guidance.At test time, it evaluates candidate next steps and selects the one expected to yield the greatest information gain.Source: §1 · p. 2
• The training pipeline builds preference step pairs and information-gain scores from Monte Carlo rollouts, then trains scoring and summarization with complementary objectives.Scoring uses reinforcement learning rewards for information-gain estimation and pairwise preferences; summarization is trained with supervised fine-tuning.Source: §1 · pp. 2–3
• PRInTS improves test-time information seeking across Qwen3-32B, DeepResearch-30B-A3B, and Gemini-2.5-Flash, with absolute average accuracy gains of 9.3%, 3.9%, and 4.0%, respectively.On GAIA, PRInTS raises DeepResearch-30B-A3B from 61.9% to 64.4%, surpassing DeepSeek-V3.1-671B at 63.1%.Source: §1 · p. 3

• Recent information-seeking agents improve trajectory quality through training, but can require substantial supervision and access to model weights, limiting generalization when the underlying model changes.PRInTS is presented as a test-time guidance approach that is orthogonal to agent fine-tuning.Source: §2 · p. 3
• Outcome reward models assess complete trajectories, whereas process reward models provide finer-grained evaluation of individual steps.Generative PRMs can also produce justifications for step scores and have been applied across several domains.Source: §2 · p. 3
• PRInTS jointly evaluates reasoning with tool interactions across multiple information-gain dimensions and uses compact trajectory summaries to manage growing context.Source: §2 · p. 3

3 Process Reward via Information Gain Score and Trajectory Summarization

• PRInTS combines candidate-step scoring with recursive trajectory summarization, using the compact summary and latest tool response to guide selection and preserve information for later evaluations.The same model is trained in both roles.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Preliminaries

• An information-seeking trajectory interleaves reasoning, tool calls, and tool responses; the agent conditions each next reasoning step and action on the query and accumulated trajectory.The interaction repeats until the agent submits an answer.Source: §3.1 · p. 4

3.2 Step-level Information Gain Estimation

• Step information gain is estimated from the change in mean answer accuracy before and after a reasoning-and-tool step, using Monte Carlo rollouts to measure its contribution to task success.Preference pairs are labeled by comparing the estimated scores; the higher-scoring step becomes the winner and seeds the next pair.Source: §3.2 · pp. 4–5
• PRInTS learns to predict information-gain scores and pairwise preferences with GRPO, combining score-accuracy and comparison rewards using weights that reduce the influence of noisy pairs.Pairs with larger score margins receive higher weights; smaller-margin pairs receive lower weights.Source: §3.2 · pp. 5–6

3.3 Info-seeking Summary Generation

• For each step, a recursively updated summary incorporates the query, previous summary, latest tool response, and current step to retain findings and plan development in compressed form.PRInTS is trained with supervised fine-tuning to imitate these annotated summaries.Source: §3.3 · p. 6

4 Experiments

• Experiments use three agents, three benchmarks, GPT-5 answer judging, and Avg@3; PRInTS selects the highest-scoring of four candidate steps at test time.Baselines include the base agent, reasoning heuristics, and existing PRMs.Source: §4 · pp. 6–7

4.1 Results and Discussion

• 9.3% absolute average accuracy gain over Qwen3-32B, versus 1.5% for StepWiser, with the same annotations.Verbal-progress and Web-Shepherd add only 0.2% and 0.5%, respectively; PRInTS uses multi-factor analyses and dense comparative scoring.Source: §4.1 · p. 7, Table 1 · p. 7
• 3.9% absolute average accuracy gain over Tongyi DeepResearch-30B-A3B; on GAIA, accuracy rises from 61.9% to 64.4%.PRInTS also improves challenging GAIA Level 3 and WebWalkerQA Hard subsets; the augmented agent reaches competitive performance with OpenAI DeepResearch (67.4%) and surpasses DeepSeek-V3.1-671B (63.1%) on GAIA.Source: §4.1 · pp. 7–8, Table 2 · p. 7
• 4.0% absolute average accuracy gain on Gemini-2.5-Flash, versus 1.5% for the second-best method; the GAIA Level 3 gain is 5.5%.The reported gains support generalization to a closed-source frontier model.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 Analysis and Ablations

• 7.7% absolute average accuracy improvement over full raw history when scoring with trajectory summaries.Summarization performs best or second-best across FRAMES and GAIA Levels 1–2; extending raw history does not improve performance, and H−2: outperforms H−1:, H−4:, and Ht.Source: §4.2 · p. 8, Table 4 · p. 8
• 2.0% and 3.1% absolute average accuracy gains from combining score and comparison rewards over either reward alone, with adaptive weighting adding 1.0%.The adaptive weight reduces the influence of small-margin preference pairs that may reflect annotation noise.Source: §4.2 · p. 9, Table 5 · p. 9
• 2.5%, 3.8%, and 8.9% absolute accuracy gains at n = 2, 4, and 8, respectively, on GAIA Level 2 with Qwen3-32B.Performance declines at n = 16; the authors attribute this to selecting superficially informative exploratory steps over candidates that correctly finish the task.Source: §4.2 · p. 9, Figure 4 · p. 9

5 Conclusion

• PRInTS combines information-gain scoring and recursive trajectory summarization, jointly trained to guide steps under accumulating context.The paper reports consistent gains across three agents and says the approach also improves frontier information-seeking agents.Source: §5 · p. 9

Limitations

• The underlying QA samples focus on English queries, leaving multilingual coverage as a direction for expanding PRInTS’s versatility.Using PRInTS as a dense reward signal to train agents with reinforcement learning is also left for future work because of significant computational demands.Source: Limitations · p. 10

Ethical Considerations

• Finer-grained evaluation of intermediate reasoning and tool interactions may reduce erroneous or inefficient information-seeking behavior.The authors identify scientific research and knowledge discovery as possible beneficiaries, and report no additional ethical implications beyond those of agentic research systems generally.Source: Ethical Considerations · p. 10

Tool Use

• Inspect-Eval supports ReAct-style multi-turn reasoning and tool interactions, including web search, browsing, and code execution.Agents use these tools to search and synthesize external information.Source: A Details of Experimental Setups · p. 13

Evaluation Benchmarks

• The evaluation uses GAIA’s text-only validation subset, 247 English WebWalkerQA questions, and 300 randomly selected FRAMES samples.GAIA spans three difficulty levels; WebWalkerQA covers three levels of web-based reasoning.Source: A Details of Experimental Setups · p. 13

Baselines

• Web-Shepherd evaluates checklist items with coarse Yes, No, or In progress labels, while GenPRM verifies trajectory-step correctness with a binary verdict.StepWiser uses binary effective/ineffective labels, with positive labels assigned when the current-to-previous mean-accuracy ratio exceeds 0.7.Source: A Details of Experimental Setups · p. 13
• Confidence scores use negative average log-probability over top-10 tokens, Relevance uses Jaccard similarity with past steps, and Verbal progress rates proximity to task completion from 1 to 5.Higher confidence scores indicate lower uncertainty; higher Relevance indicates greater contextual coherence.Source: A Details of Experimental Setups · p. 13

Data Annotations

• Figure 5’s annotations span the information-gain score range, with winning and losing steps showing clear separation.The authors say the balanced distribution covers harmful through highly effective steps and supports dense comparative scoring.Source: Figure 5 · p. 14, A Details of Experimental Setups · p. 14

Train Configurations and Data Annotations

• Training alternates four SFT summarization epochs with four GRPO scoring epochs, while annotations use 4,344 questions and retain 2,294 preference pairs.The reported training run takes 16 hours on one node with eight RTX 6000 Ada GPUs; annotation estimates information gain from eight rollouts per step and discards steps with 0% or 100% success.Source: A Details of Experimental Setups · p. 14

Ablation study of summarizer

• The summarizer ablation reports better performance when PRInTS summarizes and scores than when a separately trained PRM scores with Qwen3-32B summaries.The caption describes the two abilities as complementary.Source: Table 6 · p. 14

Frontier Model Performance

• The GAIA comparison uses reported reference scores of 67.4% for OpenAI DeepResearch and 63.1% for DeepSeek-V3.1-671B.These figures are reference points for evaluating PRInTS integrated with DeepResearch-30B-A3B.Source: A Details of Experimental Setups · p. 14

PRInTS as Summarizer

• PRInTS jointly trained as scorer and summarizer outperforms a separated design pairing a scoring-only PRM with Qwen3-32B summaries.The authors hypothesize positive transfer because both abilities use the same input; their alternating SFT-GRPO schedule integrates them.Source: B Additional Experiments · p. 14

Dataset Scaling

• 29.5 accuracy on GAIA Level 2 is achieved with 50% (∼1k samples) of annotation data using a 4B model, and the scaling curve saturates beyond 100%.The authors describe ∼2k preference pairs as a data-efficient point.Source: B Additional Experiments · p. 15, Figure 6 · p. 15

Stability of Information Gain Estimation

• At M = 4, 8, and 16, preference flip rates fall from 3.62% to 1.81%, while variance drops from 0.025 to 0.004 as rollout count increases.MAE relative to the M = 32 reference falls from 0.097 to 0.037.Source: B Additional Experiments · p. 15

Impact of Annotation Policy Model

• Qwen3-32B and Qwen3-14B yield consistent average information gain scores, but the 32B annotator produces more preference pairs: 240 versus 169.The pipeline terminates when a step reaches 100% or 0% success, so a sufficiently capable annotator is needed to maximize learning signal.Source: B Additional Experiments · p. 15

Qualitative Analysis of Failure Cases

• PRInTS can penalize correct heuristic answers lacking definitive textual evidence, prompting continued search that prolongs trajectories and raises context-overflow or step-limit-timeout risk.The authors describe this as a trade-off between strict factual grounding and trajectory efficiency.Source: B Additional Experiments · p. 15