PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents via Retrieval Grounding

Hyunho Kook, Junhyuk So, Tianyu Fu, Haizhong Zheng, Beidi Chen

arXiv:2608.24024v1cs.AI

TL;DR

Confidence-based voting transfers poorly to multi-turn search agents because retrieved documents inflate copied-token logprobs and flatten scores within questions. The paper proposes Retrieval-Grounded Voting, which scores answer-document overlap outside the contaminated context, and finds consistent gains across four benchmarks and five LLMs. RGV can improve accuracy by up to +5.4% over confidence-based voting, with important limitations when retrieval is weak or grounding is topically relevant but factually wrong.

  • Problem

    Confidence-based voting transfers poorly from single-turn reasoning to search agents because retrieved documents contaminate the token-logprob signal used for weighting.

  • Method

    Retrieval-Grounded Voting weights each rollout by lexical overlap between its final answer prose and the documents it retrieved, using an external retrieval-log signal.

  • Results

    Across four search-agent benchmarks and five LLMs, RGV outperforms confidence-based voting by up to +5.4% accuracy and +35% on minority-correct questions.

  • Takeaways & Limitations

    When retrieved evidence contaminates model context, environment-derived aggregation signals offer a more reliable voting foundation than internal model signals.

  • Takeaways & Limitations

    RGV inherits retrieval quality and can reward topically relevant but factually incorrect documents; short answers can weaken its grounding signal.

Abstract

from arXiv · show

Confidence-based voting aggregates parallel LLM rollouts by weighting each with internal signals such as token log probabilities, and has been actively studied for single-turn reasoning. However, modern LLMs increasingly act as multi-turn search agents that retrieve and condition on external documents. In this paper, we show that confidence-based voting transfers poorly to this multi-turn setting, and identify the underlying failure reason as copy inflation: when retrieved documents are appended to an agent's context, tokens copied from those documents receive systematically inflated log probabilities. This flattens confidence scores within each question and weakens the resulting weighted vote. To address this issue, we propose Retrieval-Grounded Voting (RGV), which scores each rollout by the lexical overlap between its final answer and the documents it retrieved. By computing the signal outside the contaminated context, RGV sidesteps both token log probabilities and additional LLM calls. Across four search-agent benchmarks and five LLMs, RGV consistently outperforms confidence-based voting, with gains of up to +5.4% accuracy and +35% on minority-correct questions, where the correct answer appears in only 1-2 of 8 rollouts.

1 Introduction

Confidence-based voting transfers poorly from single-turn reasoning to multi-turn search agents that condition on retrieved documents. The paper diagnoses this gap and proposes Retrieval-Grounded Voting, reporting gains across benchmarks and models.

  • Motivation: Confidence-based voting is understudied and transfers poorly when search agents condition on retrieved documents.Prior voters were designed for single-turn settings, while modern agents append retrieved evidence to their context.
  • Contributions: The paper identifies copy-inflation as the mechanism that degrades logprob-based confidence in search agents.Retrieved documents inflate the logprobs of copied tokens, compressing confidence scores within each question.
  • Contributions: RGV weights each rollout by lexical overlap between its answer prose and retrieved documents, using a signal outside the contaminated context.The method requires no logprobs, fine-tuning, or extra LLM calls.
  • Results: +5.4% accuracy and +35% on minority-correct questions are the reported maximum gains over logprob-based confidence voting.The evaluation spans four search-agent benchmarks and five LLMs.
  • Results: At eight rollouts, RGV matches DeepConf while halving the rollout budget at equal accuracy.The gains are robust to overlap-metric choice and add negligible CPU cost.

2 Background

The paper studies ReAct-style search agents that interleave reasoning with tool calls and produce answers from retrieved documents. Parallel rollouts are combined by weighted majority voting, with DeepConf using token-level confidence and RGV using retrieval grounding.

  • Multi-turn search agents: Multi-turn search agents interleave model reasoning with search or page-visit tool calls that return text documents.Retrieved snippets are appended to the context used at later turns.
  • Rollouts: A rollout is one trajectory containing tool calls, their responses, emitted thoughts, and a final predicted answer.The record distinguishes answer prose from the documents fetched during the trajectory.
  • Voting: Weighted majority voting selects the answer whose rollouts carry the largest total nonnegative weight.Simple majority uses unit weights, while other methods assign rollout-specific scores.
  • Voting: DeepConf reduces token-level logprob confidences to one rollout weight through sliding-window aggregation.The headline configuration uses the Lowest-Group reduction with window size W=1024.
  • Voting: RGV replaces the confidence weight with a direct grounding check between answer prose and retrieved documents.Figure 2 contrasts this retrieval-based signal with copy-inflated DeepConf scores.

3 Motivation

Retrieved documents inflate the logprobs of copied tokens, causing DeepConf scores to lose within-question variation and stop tracking correctness. Because copying is pervasive, the resulting weighted vote can collapse toward simple majority.

  • Copy-inflation: Copy-inflation makes copied-token logprob reflect the ease of copying rather than whether the copied content answers the question.Confidence voting cannot distinguish copying the right entity from copying an irrelevant one.
  • Score flattening: As copy fraction grows, DeepConf’s within-question score-variance share shrinks from 0.39 to 0.20.Same-question rollouts receive nearly identical scores, so the weighting tiebreaker disappears.
  • Prevalence: Median rollouts copy 93% of content tokens, and 85% of questions have mean copy fraction ≥0.90.The mechanism therefore applies broadly across the benchmark rather than only to a corner case.
  • Consequences: DeepConf gives impossible questions 87% of its peak score and has 54.2% overlap between correct and wrong rollout scores.The pattern replicates across every tested benchmark and model.
  • Design principle: Robust voting requires a signal measured outside the contaminated context and independent of the model’s hidden state.This principle motivates the retrieval-log signal used by RGV.

4 From principle to method: Retrieval-Grounded Voting

RGV addresses copy-inflation by scoring how lexically anchored each answer is in the documents retrieved by its rollout. Its design uses a best-document match and answer-side normalization while reading the document signal from the retrieval log.

  • RGV score: RGV scores each rollout by the maximum fraction of answer-prose tokens anchored in any retrieved document.The score operates on token sets derived from answer prose and retrieved documents.
  • Design choices: RGV’s overlap design choices matter only at the margin, while the signal source carries the bulk of the gain.Alternative overlap functions yield similar headline numbers.
  • Design choices: RGV uses the single best-matching document rather than the union of documents to avoid diluting a strong supporting match.This is the method’s max-over-docs design choice.
  • Design choices: Answer-side normalization avoids penalizing a well-anchored answer when its supporting retrieved document is long.The method divides by answer-prose tokens rather than the prose-document union.
  • External signal: The retrieval log supplies the document-side signal independently of the model’s hidden state.RGV therefore avoids token logprobs and cannot inflate its weight through verbosity or self-assuredness.

5 Experimental setup

The evaluation covers four multi-turn search-agent benchmarks, five LLMs, and eight parallel rollouts per question, using a common judging and weighted-voting protocol.

  • Benchmarks: The study evaluates BrowseComp-Plus, BrowseComp, GAIA, and FRAMES, covering fixed-corpus retrieval, open-web browsing, general-assistant tasks, and multi-hop Wikipedia synthesis.
  • Models: Five LLMs are tested across the four benchmarks: gpt-oss-120b, MiniMax-M2.7, GLM-5.1, Kimi-K2.5, and Tongyi-DeepResearch.
  • Rollout protocol: Each question receives N=8 parallel rollouts, with benchmark-specific search and document-fetching tools.
  • Judge: A single Qwen3-32B judge evaluates final answers using only the question, gold answer, and rollout answer.
  • Voting protocol: Simple-majority, DeepConf, and RGV supply per-rollout weights to a common weighted-majority vote, while single and oracle are non-voting references.

6 Results

Across the benchmark suite, RGV is best in all 20 benchmark×model cells and improves most when voting is difficult, while grounding analyses explain its signal and identify remaining failure cases.

  • Headline results: RGV is best in all 20 benchmark×model cells, with gains largest at low single-rollout accuracy and on noisy open-web retrieval.
  • Validation: RGV separates judge-correct from judge-wrong rollouts better than DeepConf, with ROC AUC 0.908 versus 0.837.RGV scores also correlate with gold recall at r=0.57, compared with 0.36 for DeepConf.
  • Grounding mechanism: Rare-entity tokens carry more discriminative grounding signal than digits and common content words contribute little to the correct–wrong score gap.Digit tokens produce a 0.145 gap, capitalised entity names 0.092, and common content words 0.010.
  • Grounding mechanism: The correct–wrong mean gap is 0.080, falling to 0.025 after across-question shuffling and 0.002 under a length-matched random-token null.A within-question document shuffle preserves the 0.080 gap.
  • Difficulty: In the 1−2/8 minority-correct bucket, RGV scores 84.4% versus DeepConf’s 49.2%, a +35.2% gap.These are questions where one or two of eight rollouts are correct.
  • Scaling: RGV remains above DeepConf as rollout budgets vary, and reaches DeepConf’s N=8 accuracy with N=4 rollouts.RGV at N=4 achieves 65.9%, matching DeepConf at N=8 with 65.7%.
  • Cost: RGV uses no extra LLM call or logprobs and costs approximately 0.3 ms per rollout on one CPU thread.Its cost–accuracy advantage combines with matching DeepConf accuracy at half the rollout budget.
  • Design choices: The max-over-docs reduction consistently leads, while all four reductions beat simple majority by +7 to +9%.Max adds a further +0.6 to +1.8% by focusing on the most relevant document.

7 Related work

Prior work uses confidence proxies, trajectory aggregation, recalibration, and grounding metrics, but these approaches leave distinct costs or unresolved signal-transfer issues for search agents.

  • Confidence-based voters primarily target single-turn reasoning and provide no diagnosis for their weaker transfer to search agents.
  • RL-based recalibration and verbalised-confidence methods target self-reported confidence rather than the token-logprob signal studied here.
  • Trajectory-aware aggregators score or compress rollouts with additional model inference, increasing the cost beyond parallel sampling.
  • RGV’s lexical-overlap signal draws on established extractive-coverage and grounding metrics from summarisation and retrieval-augmented evaluation.

8 Conclusion

The paper identifies copy-inflation as the mechanism degrading token-level confidence in search agents and proposes RGV as an external, retrieval-derived voting signal. Across the evaluated benchmarks and models, RGV outperforms simpler voting methods at minimal cost, while the diagnosis may apply to other mechanisms consuming token confidence.

  • Copy-inflation occurs when retrieved documents inflate token-level confidence, contaminating logprob-based aggregation in search agents.
  • RGV reads voting evidence from environment-derived retrieval records rather than the contaminated model context.
  • Across four benchmarks and five LLMs, RGV outperforms simple majority and confidence-based voting while adding minimal cost.
  • The same contaminated token-confidence signal may affect early stopping, routing, abstention, confidence-shaped rewards, and hallucination detection.

Limitations

RGV is bounded by retrieval quality and by the lexical nature of its grounding score, with weaker performance for short answers and limited scope beyond retrieval-centered English search agents. The evaluation also reports sampling and implementation boundaries relevant to interpreting its results.

  • Retrieval dependence: RGV cannot exceed the quality of the documents returned by retrieval, and offers little benefit when retrieval contributes little to the task.
  • Grounding limitations: A wrong rollout can outscore a correct one when it echoes topically relevant but factually incorrect evidence, because RGV performs no entailment check.
  • Grounding limitations: RGV selected the correct answer on 590 of 616 questions with an available correct rollout, leaving 26 failures.
  • Answer-length dependence: The signal weakens for very short answer prose, where normalization reduces grounding to answer containment and the vote merely matches DeepConf.
  • Scope: The study focuses on English multi-turn search agents; non-retrieval tools and reasoning-intensive tasks require different grounding signals or show smaller gains.
  • Robustness: An adversarially poisoned corpus could reward rollouts copying from it, although injecting up to 16 irrelevant documents leaves voting unchanged.
  • Evaluation: Headline accuracies use 150-question samples, while three-fold variability estimates are stability indicators rather than formal hypothesis tests.
  • Alternative signals: Verbalised confidence requires an additional deterministic follow-up call and is evaluated separately from the main-text signal comparison.

D Cross-dataset evidence for copy-inflation

Cross-dataset diagnostics show that copy-inflation is consistent across the evaluated benchmark–model configurations and model families, including a reasoning-tuned model. Impossible questions nevertheless receive high DeepConf scores in every configuration, indicating broad miscalibration rather than a single-setting artifact.

  • Cross-dataset replication: Copy-inflation diagnostics were replicated across all twenty benchmark×model configurations rather than only the deep-dive setting.
  • Cross-dataset replication: 82–97% of peak DeepConf scores went to impossible questions on every benchmark–model combination.
  • Cross-dataset replication: The replicated pattern supports copy-inflation as a general property of logprob-based confidence in multi-turn search agents.
  • Model-family analysis: Copy-inflation appeared in every tested model family, including a reasoning-tuned DeepSeek model.
  • Model-family analysis: The DeepSeek cell had the smallest copy-vs-non-copy gap, +0.04 nats, while copied tokens still comprised 89.7% of its output.

E Robustness to alternative lexical-overlap variants

RGV remains effective across alternative lexical-overlap scores, indicating that its gains primarily come from reading the retrieval log rather than from one particular metric.

  • Alternative lexical-overlap variants: The evaluated variants include Jaccard, ROUGE-2, ROUGE-L, BM25, and TF-IDF.These scores differ in symmetry, n-gram granularity, corpus scoring, and vector representation.
  • Semantic comparison: A frozen dense encoder replaces lexical overlap in a semantic-scoring variant while keeping the max-over-docs reduction and other settings fixed.The encoder is intfloat/e5-base-v2 with 109M parameters and no fine-tuning.
  • Semantic comparison: The embedding voter beats simple majority and DeepConf on all three degraded-retrieval cells but does not reach the lexical score.It requires roughly five orders of magnitude more compute, versus approximately 0.3 ms per rollout for the lexical score.
  • Interpretation: Two scorers with otherwise unrelated constructions land on the same side of the baselines, supporting the signal-source account of RGV’s gains.The paper treats metric choice as secondary to reading overlap outside the contaminated context.
  • Alternative lexical-overlap variants: RGV’s advantage persists across lexical-overlap variants, while the highest value in each row is bolded in Table 7.All variants use the same max-over-docs reduction; only the per-rollout overlap score changes.

G Robustness of the copy-inflation gap

The copy-inflation gap is robust across token-selection rules: copied tokens receive higher log probabilities, and the effect is not tied to one measurement convention.

  • Measurement robustness: +0.50 nats is the headline per-token mean logprob gap between copy and non-copy content tokens.The gap is +0.61 on judge-correct rollouts and +0.38 on judge-wrong rollouts.
  • Measurement procedure: The analysis constructs retrieved-document text and a normalized generated-token stream for each rollout before classifying copy tokens by substring overlap.Tokens are lower-cased, stripped of subword prefixes, restricted to alphanumeric characters, and filtered below two characters.
  • Measurement procedure: The aggregation pools kept tokens across rollouts using a token-subset predicate and optional token weight.The main-text headline is reproduced by the all-alphanumeric baseline.
  • Alternative token subsets: Alternative rules remove stopwords, apply IDF weighting, or select digits and capitalized-leading tokens.The IDF rule uses sample IDF over N=307 documents-as-rollouts.
  • Measurement robustness: All five token-subset rules produce significantly positive copy-inflation gaps, with the largest gap under IDF weighting.The result is consistent with rare entity-like tokens being especially associated with copied retrieval content.

H Degraded retrieval: weaker retriever and corpus mismatch

RGV remains useful under weaker retrieval, but corpus mismatch limits every aggregation rule; the paper identifies copy fraction and oracle headroom as regime indicators.

  • Weaker retriever: +5.3/+3.3 over simple majority and DeepConf, and +2.6/+2.0 on Tongyi-DeepResearch, are the grounded-vote margins on the two BM25 rows.The corresponding single-rollout accuracies decline from 47.3→31.4 and 58.9→48.8 after retrieval degradation.
  • Weaker retriever: On HotpotQA, all four aggregation rules fall within a 2.6-point band when single accuracy is 72.7 and the oracle headroom is 8 points.This cell has the lowest measured copy fraction, 78.4%, and DeepConf has rollout-level AUC 0.82.
  • Corpus mismatch: Under FRAMES corpus mismatch, single-rollout accuracy drops 78.7→59.6, while RGV at 60.0 is the only aggregation rule above single-rollout accuracy.Simple majority reaches 58.7 and DeepConf 58.0 as correlated retrieval errors undermine voting.
  • Regime diagnosis: Degradation is described as predictable from a cell’s copy fraction and remaining headroom to the oracle.These quantities indicate which aggregation regime applies before selecting a voting rule.
  • Internal repair: Masking copied tokens restores score spread but not correctness: AUC falls in 9/10 cells, while RGV retains 1.1–3.8× more within-question variance share in 10/10.The restored confidence spread does not systematically improve voting accuracy.
  • Internal repair: The contamination resists internal repair, so token-confidence consumers such as early stopping, routing, abstention, and reward shaping inherit the problem.The paper states that no simple adjustment recovers a usable weight.
  • Prompt sensitivity: In a prompt-format arm, the grounding instruction leaves mean voted weight unchanged at 0.091→0.091 and rollout-level AUC nearly unchanged at 0.840→0.844.A common instruction shifts all rollouts equally and does not alter their within-question ordering.
  • Selection versus voting: RGV beats argmax-DeepConf at every rollout count and beats simple majority at N=8, but voting exceeds selection by +4.7 points at N=8.Selection remains exposed to well-grounded-but-wrong rollouts, whereas answer-cluster voting averages that risk.
Loading 2608.24024v1…