PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay

Prateek Chhikara

arXiv:2608.26846v1cs.CLcs.AI

TL;DR

Interactive agents need confidence signals to decide whether to commit, retrieve, or defer, but confidence is often evaluated without its trajectory-level action consequences. The paper introduces matched trajectory replay to compare raw and calibrated confidence under identical answer states, evidence, budgets, and costs. Calibration improves committed-answer accuracy but shifts coverage, overall accuracy, and retrieval use unevenly, so commitment-risk calibration does not by itself determine whether another retrieval is worthwhile.

  • Problem

    Confidence is commonly evaluated in isolation, leaving the trajectory-level consequences of confidence-triggered commitment, retrieval, and deferral insufficiently measured.

  • Method

    Matched trajectory replay replays raw and post-hoc isotonic-calibrated confidence over identical stored answer and evidence states, budgets, thresholds, and action costs.

  • Results

    Across all six model–dataset pairs, calibration raises committed-answer accuracy by up to 41 percentage points, while overall accuracy rises by up to 15pp on HotpotQA and falls by up to 17pp on MuSiQue.

  • Takeaways & Limitations

    Calibration should make commitment risk interpretable, while retrieval should use a separate state-aware value-of-information or utility estimate.

  • Takeaways & Limitations

    Controlled benchmark-grounded replay fixes future evidence and queries, so it cannot capture how alternative actions would change live retrieval or subsequent model responses.

Abstract

from arXiv · show

Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost.

1. Introduction

The paper frames confidence as a monitoring signal whose thresholded use controls commitment, retrieval, or deferral, then proposes matched replay to evaluate those consequences. Calibration can improve commitment-risk interpretation, but its effects on coverage, accuracy, and retrieval cost are not uniformly beneficial.

  • Motivation: Calibration makes confidence interpretable as empirical current-answer correctness but does not estimate the value of another retrieval.A calibrated score also does not improve ranking because monotone maps preserve score order except for ties.
  • Approach: Matched trajectory replay changes only the confidence supplied to the controller while holding answers, evidence, thresholds, budgets, and costs fixed.The protocol fits calibration on a separate split, freezes it, and replays saved states under raw and calibrated policies.
  • Findings: At the same numerical threshold, calibration increases committed-answer accuracy across all six model–dataset pairs by up to 41 percentage points.The comparison uses Mistral, GPT, and Qwen models on HotpotQA and MuSiQue.
  • Findings: Overall accuracy improves by up to 15pp on HotpotQA but falls by up to 17pp on MuSiQue as coverage decreases and retrieval cost increases.Thus, greater selectivity does not imply greater system-level utility without explicit action costs.
  • Findings: A calibration map fitted before retrieval improves held-out calibration through depths one and two but is worse than raw confidence at depth three for all three models.The result warns that calibration validated at one trajectory state may not remain reliable as evidence changes the interaction.

2. Related Work

Related work treats calibration, selective prediction, metacognitive control, and adaptive retrieval as largely separate problems. The paper connects them by comparing confidence-to-action mappings on matched answer and evidence trajectories.

  • Confidence estimation and calibration: Calibration methods map model scores to probabilities whose empirical correctness can be inspected with reliability diagrams or calibration-error measures.Language-model confidence may come from verbalized reports, sampling variation, or self-evaluation signals.
  • Selective prediction and metacognitive control: Selective prediction formalizes risk–coverage trade-offs by allowing individual predictions to be accepted or rejected.Selective question answering and refusal-aware methods extend this framework to open-ended answers.
  • Adaptive retrieval and routing: Adaptive retrieval and routing methods trigger search or allocate computation using uncertainty, complexity, reflection, token probabilities, or learned signals.Calibration-oriented RAG changes retrieved documents to improve decision reliability and is complementary to this work’s intervention.
  • Paper’s position: Matched trajectory replay asks how actions change when controllers receive different confidence scores despite identical candidate answers and retrieval opportunities.This connects static confidence evaluation with interactive control.

3. Matched Trajectory Replay: Evaluation Protocol

The protocol constructs one deterministic evidence path per question, collects every depth-specific answer and confidence state, and enables paired replay under identical trajectories. Fixed evidence plans make controller differences attributable to the confidence-to-action mapping.

  • Trajectory construction: Each state contains the question, cumulative evidence, and remaining retrieval budget, while the model emits an answer and verbalized confidence at depths zero through three.Offline replay exposes every policy to the same answer and evidence sequence at each possible depth.
  • Matched evidence trajectories: Fixed retrieval plans interleave answer-supporting passages with ranked distractors and reveal complete slices cumulatively across three retrieval depths.Controllers can change only when the stored trajectory terminates, not future evidence or model responses.
  • Trace collection: The study records question, split, evidence plan, depth-specific prompts and responses, verbalized confidence, and correctness for each deterministic trace.Traces are generated at temperature zero over three retrieval rounds and judged with a fixed structured-output LLM judge.

4. Replay and Evaluation

Replay freezes the collected trajectories and compares raw and isotonic-calibrated confidence under common thresholds and budgets. The evaluation measures controller actions alongside answer quality, coverage, and retrieval cost.

  • Calibration replay: The calibration map is fitted on labeled depth-zero calibration records, frozen before final-test replay, and applied as κ_t = ϕ_M(v_t).No final-test trace contributes to fitting, and isotonic regression preserves score order apart from possible ties.
  • Controllers: Both gated controllers use thresholds 0.3 and 0.7 and allow at most three retrieval actions.They commit at confidence at least 0.7, retrieve below that threshold while budget remains, then abstain or escalate after exhaustion.
  • Threshold interpretation: The 0.7 commitment threshold represents a nominal 70% correctness requirement for calibrated confidence but only a heuristic for raw confidence.The fixed-number comparison is not matched on risk or coverage.
  • Baselines and metrics: Fixed-depth and adaptive baselines always commit with 100% coverage, while gated controllers trade coverage and retrieval cost for selective answering.Overall accuracy must therefore be interpreted jointly with committed accuracy, coverage, and cost.

5. Results

At a fixed threshold, calibration makes committed answers more accurate but shifts risk–coverage and OA–cost operating points through lower coverage and greater retrieval use. Its held-out calibration transfers through retrieval depth two but reverses at depth three, while retrieval benefits on average do not show that confidence identifies helpful episodes.

  • Risk–coverage: Table 1 requires OA, committed accuracy, coverage, and cost to be interpreted together because higher committed accuracy can accompany substantially lower overall accuracy.Fixed-depth and adaptive baselines always commit, whereas gated controllers select a subset of answers.
  • Fixed-threshold results: Calibration raises committed-answer accuracy by 15.8–41.0pp across all six model–dataset pairs, while reducing coverage by 4.9–92.4pp and adding 0.98–2.66 retrieval actions.On HotpotQA, OA rises 4.2–15.0pp; on MuSiQue, it falls 2.9–16.7pp.
  • Risk–coverage: At τ = 0.7, Calibrated-Gated generally commits fewer answers with lower selective risk than Raw-Gated, selecting different operating points despite the shared threshold.The shift is largest for Mistral, intermediate for GPT-OSS, and smallest for Qwen.
  • Calibration across depths: Calibration improves held-out ECE through retrieval depths one and two, but calibrated ECE exceeds raw ECE at depth three for all three models.The depth-zero reduction is 25.5–44.2pp to 2.0–2.5pp, while depth-three degradation is 1.8–7.6pp.
  • Accuracy–cost trade-off: Calibration does not uniformly improve OA–cost trade-offs: Mistral’s OA is unchanged at higher cost, GPT-OSS gains slightly, and Qwen gains substantially without surpassing the adaptive proxy.The sweeps begin from a more selective zero-budget policy and use more retrieval as the budget grows.
  • Retrieval value: Additional retrieval often improves answers on average, but confidence estimates current-answer correctness rather than the expected benefit of the next evidence step.Therefore, aggregate helpful transitions do not establish confidence-based episode-level retrieval control.

6. Discussion and Conclusion

Matched trajectory replay shows that calibration changes controller decisions rather than stored answers, improving committed-answer accuracy by selecting a more selective operating point. Its effects on overall accuracy and retrieval cost vary, while depth-zero calibration transfers only through the first two retrieval depths.

  • Calibration changes decisions made by a fixed threshold, not the stored answers.
  • Calibration improves committed-answer accuracy by moving the controller to a more selective operating point.
  • Overall accuracy and retrieval cost vary across datasets and models after calibration.
  • Depth-zero calibration improves held-out calibration through the first two retrieval depths but overcorrects high-confidence states at depth three.
  • Calibration should make commitment risk interpretable, while retrieval control requires a separate expected-information-gain or utility estimate.

7. Limitations

The controlled replay improves attribution by fixing evidence paths and future queries, but this limits external validity and may overstate retriever reliability. The study also has narrow empirical scope, depth-transfer assumptions, and evaluation choices that prevent its cost curves from representing end-to-end utility.

  • Controlled replay fixes future evidence and queries, so it cannot capture how actions change later retrieval or model responses in live agents.
  • Benchmark support and ranked distractors can overstate retriever reliability.
  • The empirical scope covers two multi-hop QA datasets, three models, one deterministic response collection per model, and deterministic source-order splits.
  • A depth-zero calibration map is assumed to remain valid after evidence acquisition, but this assumption fails at depth three.
  • LLM-judged correctness lacks human validation, and pricing only retrieval actions means OA–cost curves are not end-to-end utility curves.

A. Experimental Protocol

The experimental protocol elicits an answer and verbal confidence at each evidence state using fixed prompts and structured JSON outputs. Models are instructed to use visible evidence as data, answer concisely, and express uncertainty through a numeric confidence estimate.

  • The appendix documents answer and confidence elicitation, correctness judging, fixed evidence paths, model checkpoints, datasets, splits, and calibration diagnostics.
  • At each state, the model answers using parametric knowledge and currently available evidence, with prompts differing by retrieval depth.
  • Models must treat visible evidence as data rather than instructions, combine necessary facts internally, and return the shortest direct answer.
  • Confidence is a 0–100 estimate of the probability that the returned answer will be judged correct given the information available now.
  • The required output is one JSON object containing a non-empty short answer and numeric confidence, with no additional fields or commentary.

A.2. Fixed evidence paths

The fixed-evidence protocol constructs deterministic passage paths by ranking distractors and interleaving them with source-ordered supporting passages, then reveals evidence in complete slices at successive retrieval depths. Correctness is labeled by a fixed judge using the question, reference answers, and candidate answer.

  • Evidence-path construction: Supporting passages are identified from dataset annotations and ordered by their source index, while remaining candidate passages are treated as distractors.HotpotQA uses supporting-fact titles; MuSiQue uses each paragraph’s is_supporting flag.
  • Evidence-path construction: Distractors are ranked once per question using normalized question-token overlap and cosine similarity between question and passage text.Cosine similarity uses BAAI/bge-m3 embeddings, with min–max normalization over that question’s distractors.
  • Evidence-path construction: The primary plan interleaves source-ordered supports with retained ranked distractors, omitting elements when either list is exhausted.The passages define the support and distractor sequences used to form P(q).
  • Evidence-path construction: Each zero-based passage is assigned to one of three slices, and retrieval reveals the next entire slice at depths one through three.All slices are computed before model collection, and the model is queried at every depth on the fixed plan.
  • Correctness labeling: Correctness labels come from a temperature-zero Gemini judge that returns a strict JSON boolean after comparing the candidate answer with reference answers.Unparseable judge responses trigger retries before storing the parsed correct field.

A.4. Models, datasets, and splits

The evaluation compares three specified model checkpoints under common prompts, trajectories, controller settings, and judging, using deterministic calibration–test splits and fixed inference limits. Held-out replay and transition analyses reserve test states for evaluation while fitting calibration on pooled records.

  • Models and controls: Mistral Small, GPT-OSS 120B, and Qwen3 235B-A22B are evaluated under common prompts, trajectories, controller settings, and a fixed correctness judge.A separate isotonic calibration map is fit for each model, without fine-tuning the base models.
  • Datasets and splits: Examples are deterministically split into calibration and final-test sets, with isotonic maps fit once on pooled HotpotQA and MuSiQue calibration records.The maps are not dataset-specific, and the final-test split is reserved for replay results.
  • Inference settings: Inference uses temperature zero and caps generation at 4096 output tokens, alongside provider-reported context limits and model descriptions.The reported context limits describe the checkpoints or hosted identifiers rather than the experiment’s generation cap.
  • Transition analysis: The adjacent retrieval-transition analysis contains 7,367 usable transitions per model after excluding four duplicate depth-three state pairs.The controlled set has 2,457 traces with up to three adjacent transitions; four empty third slices are excluded.

B.1. Why transfer can fail at depth three

A depth-zero calibration map can become too conservative for later retrieval states, especially in the highest raw-confidence bin. Figure 5 visualizes the depth-zero confidence–correctness relationship on records used to fit each model’s isotonic map.

  • Depth-three transfer: 2,293 of 2,457 Mistral depth-three states occupy the 0.9–1.0 raw-confidence bin, where accuracy is 0.789 but calibrated confidence is 0.557.The mean raw confidence in this bin is 0.930.
  • Depth-three transfer: 1,713 of 2,457 GPT depth-three states occupy the 0.9–1.0 raw-confidence bin, where accuracy is 0.934 but calibrated confidence is 0.884.The mean raw confidence in this bin is 0.930.
  • Depth-three transfer: 2,018 of 2,457 Qwen depth-three states occupy the 0.9–1.0 raw-confidence bin, where accuracy is 0.871 but calibrated confidence is 0.730.The mean raw confidence in this bin is 0.988.
  • Depth-zero calibration: Figure 5 compares verbalized depth-zero confidence with empirical correctness on calibration records for each answer model, with hatching marking the calibration gap.The diagrams show the partition used to fit each model’s isotonic map.
Loading 2608.26846v1…