Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents

1 INTRODUCTION

• Citation support, factual correctness, and report utility do not establish that adaptive search exposed a representative view of the candidate pool.The paper frames selection bias as a distinct reliability problem.Source: §1 · pp. 1–2
• Early supporting evidence can steer later queries, document choices, and stopping, so a report may faithfully summarize what was read yet differ from the available evidence’s direction.Report inspection alone cannot determine whether the shift arose from exposure, selection, or stopping.Source: §1 · p. 1
• CESS estimates the candidate-pool target by combining predictions for every candidate with logged probabilities of selection and of reaching each search round.Shrinkage stabilizes short searches, and an interval replaces a point estimate when part of the pool cannot be sampled.Source: §1 · p. 2
• A pool-target estimate that recovers the same target under every search policy cannot also identify the effect of replacing that policy; paired interventions measure selection and stopping effects.Figure 1 contrasts CESS’s pool estimate with intervention-based measurement of opened evidence.Source: §1 · p. 2, Figure 1 · p. 2
• In the public Open Deep Research agent, CESS reduces mean absolute error against the candidate-pool target by 60.1% and ranking sensitivity by 87.2% relative to the Opened Mean.The paper also distinguishes this target correction from measuring search-policy effects.Source: §1 · p. 3

2.1 CANDIDATE-POOL TARGET AND SEARCH PROCESS

• The candidate pool is prespecified, and its target is the equally weighted average evidence direction; this is an audit target for that pool, not a claim about the open web or ultimate truth.The framework also permits prespecified quality or source-deduplication weights.Source: §2.1 · p. 3
• Early evidence changes later queries, selection probabilities, and stopping through the evolving search history.The sequential model tracks history, queries, ranking condition, exposed documents, selected documents, and stopping decisions.Source: §2.1 · p. 3

2.2 HOW SEARCH POLICIES CHANGE CONCLUSIONS

• CESS targets the fixed pool conclusion, whereas paired interventions measure how replacing selection or stopping policies changes the average conclusion formed from opened documents.The paper distinguishes the pool target, a typical policy execution’s Opened Mean, and the change in that mean between policies.Source: §2.2 · p. 3
• When two rankings use the same pool, CESS should correct both runs toward the same pool target, so their corrected gap cannot measure the ranking policy’s effect on opened evidence.Using that corrected gap as a policy-effect estimate would erase the effect by construction.Source: §2.2 · pp. 3–4

3 CAUSAL EVIDENCE SELECTION CORRECTION

• CESS predicts each candidate's evidence score before search, then combines the full-pool prediction with probability-corrected errors from opened documents.The prediction model covers documents the agent never opens; the correction uses logged probabilities of selection and reaching each round.Source: §3 · p. 4

3.1 SEQUENTIAL CORRECTION

• Under logged sequential-design probabilities and positive selection and continuation probabilities, the raw correction is unbiased for the candidate-pool target, even if the outcome model is misspecified.Inverse selection weighting represents pool residuals, while inverse continuation weighting restores contributions from rounds lost to stopping.Source: §3.1 · p. 4

3.2 WHY CORRECTION IS NOT POLICY ATTRIBUTION

• A pool-corrected contrast cannot identify a nonzero opened-evidence policy effect; attribution requires interventions or a full off-policy estimator covering the trajectory.When selection changes future queries, selection and continuation probabilities alone omit part of the trajectory likelihood ratio.Source: §3.2 · p. 5

3.3 FINITE-BUDGET SHRINKAGE

• Shrinkage retains a tunable share of the probability correction, trading prediction error against correction variance; the MS2, public-agent, and PERSPECTRUM settings use different specified anchors and coefficients.At λ = 0 the estimator uses the full-pool prediction, and at λ = 1 it uses the unshrunk sequential correction.Source: §3.3 · p. 5

3.4 LIMITED SUPPORT: AN INTERVAL INSTEAD OF A POINT ESTIMATE

• When some candidates are inaccessible, CESS reports an interval rather than a point estimate; its sharp width is ρ(U − L), or 2ρ for scores in [−1, 1].If inaccessible target mass is unknown, ρ can be varied in sensitivity analysis.Source: §3.4 · p. 5

4 EXPERIMENTS

• The experiments assess target recovery, finite-budget accuracy and ranking stability, and whether paired intervention effects differ from CESS contrasts.They use 200 MS2 questions per query model for the target-recovery analyses and include a separate paired selection–stopping intervention.Source: §4 · pp. 5–6

4.1 TARGET RECOVERY AND THE ACCURACY–STABILITY TRADE-OFF

• CESS reduces ranking sensitivity from 0.7478 to 0.0014 in simulation and from 0.0415 to 0.0185 for Qwen and 0.0391 to 0.0076 for OLMo on MS2.The MS2 reductions are 55.4% and 80.6%, respectively; the candidate evidence is held fixed across rankings.Source: §4.1 · p. 6, Figure 2 · p. 6
• 9.2% lower MAE, 39.4% lower ranking sensitivity, and 14.2% lower worst-ranking MAE: CESS outperforms Opened Mean on MS2.It also reduces those metrics by 16.4%, 43.6%, and 21.6%, respectively, relative to Sequential Doubly Robust.Source: §4.1 · p. 7, Table 1 · p. 7
• At matched MAE, 10 of 28 comparisons favor lower ranking sensitivity for CESS, 18 intervals include zero, and none favors higher sensitivity.The gains concentrate on PERSPECTRUM, where ranking more strongly changes the evidence opened.Source: §4.1 · p. 7, Table 2 · p. 7
• Task-wise CESS reduces MAE most when only one or two documents can be opened; under adaptive stopping, gains over Opened Mean are 11.9% for Qwen and 15.6% for OLMo.At four rounds, the gains are 9.1% and 15.8%, respectively.Source: §4.1 · pp. 7–8, Figure 3 · p. 8
• The public-agent transfer preserves the planner and synthesis process while replacing the search interface; by round four, 63.9% of paired branches generate different queries and 75.0% select different document sequences.The 216 trajectories cover 36 PERSPECTRUM tasks, two rankings, and three paired seeds.Source: §4.1 · p. 8
• 60.1% lower MAE and 87.2% lower ranking sensitivity: CESS improves on Opened Mean in the public-agent evaluation.It also reduces direction errors by 18.1 percentage points; results use the primary K = 2 budget.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 WHY POLICY ATTRIBUTION REQUIRES INTERVENTION

• The 4,800-trajectory paired intervention experiment finds only modest rank correlations, 0.269 and 0.236, between CESS contrasts and directly measured policy effects.Direct selection effects have ICCs of 0.679 for OLMo and 0.685 for Qwen; pool-target estimation and policy attribution are distinct outputs.Source: §4.2 · pp. 8–9, Table 4 · p. 9

5 RELATED WORK

• Report-quality, citation, factuality, coverage, and logical-support evaluations do not estimate a common candidate-pool conclusion from one adaptively selected trajectory.Accurate citations can coexist with an unrepresentative evidence sample.Source: §5 · p. 9
• Inverse-probability, doubly robust, and active-testing methods address selective observation, but Deep Research adds sequential queries and stopping that respond to earlier evidence.The paper distinguishes estimating the candidate-pool conclusion from measuring what a search policy changes.Source: §5 · p. 9
• CESS adapts probability correction to logged search, stabilizes it for short budgets, and reports bounds when some documents have zero selection probability; interventions address policy attribution.Source: §5 · p. 9

6 CONCLUSION

• In the public Open Deep Research agent, CESS reduces target error by 60.1% and ranking sensitivity by 87.2% relative to Opened Mean.At matched target error, 10 of 28 intervals favor lower ranking sensitivity, and none favors higher sensitivity; paired interventions measure what the search policy changed.Source: §6 · p. 9

A.1 CONSTRUCTION OF DOCUMENT EVIDENCE SCORES

• MS2 evidence scores encode reported outcome direction rather than effect magnitude or clinical benefit.A zero score can reflect equal increase and decrease probabilities or a high probability of no significant difference.Source: §A.1 · p. 14

A.2 OBSERVATION UNITS AND CESS CONFIGURATIONS

• A trajectory contributes one observation per selection round, so repeated selection of a document counts repeatedly and the number of observations may exceed distinct documents.The primary MS2 configuration uses one shrinkage coefficient per query model; Task-wise CESS predicts a coefficient per task before observing its document scores.Source: §A.2 · p. 14

A.3 IDENTIFICATION AND SUPPORT DIAGNOSTICS

• Removing selection weights introduces bias when selection favors evidence-related documents, while removing continuation weights matters when stopping depends on search history.With deterministic choice, only 3.6% of candidates are reachable on average, so a point estimate cannot recover the full-pool target; CESS instead reports an interval.Source: §A.3 · p. 15
• Under extremely limited overlap, tuning selects λ = 0 and falls back to the full-pool prediction rather than amplifying unstable inverse-probability weights.With probabilities estimated from visible ranking features, MAE is 0.0570, close to 0.0576 with exact probabilities and the same linear predictor.Source: §A.3 · pp. 15–16

A.4 CONTROLLED RANKING EXPERIMENT

• The unshrunk, unclipped sequential correction reduces the conclusion gap between rankings from 0.7478 for Opened Mean to 0.0014, a 99.8% reduction.The controlled test uses known selection and continuation probabilities.Source: §A.4 · p. 16

A.5 ADDITIONAL IDENTIFICATION DIAGNOSTICS

• With λ = 0.01, removing selection or stopping probabilities changes MAE by only about 10^-4 because little of the correction enters the final estimate.The unshrunk correction is tested separately to assess whether the weights remove bias.Source: §A.5 · p. 16

B.1 DESIGN UNBIASEDNESS AND POLICY INVARIANCE

• Inverse-continuation weighting gives each potential round an expected weighted residual equal to the pool-wide mean residual, including rounds censored by stopping.Averaging across rounds and adding the pool prediction yields the candidate-pool target in expectation.Source: §B.1 · p. 16
• The unbiasedness statement applies to the raw estimator; clipping, estimated probabilities, or an incorrect continuation model can introduce finite-sample policy dependence.Source: §B.1 · p. 16

B.2 INVARIANCE–ATTRIBUTION INCOMPATIBILITY

• Corrected estimates for two policies have zero expected contrast, so that contrast equals the opened-evidence effect only when the effect itself is zero.Source: §B.2 · p. 16
• The support interval is sharp: without further restrictions, unreachable documents can attain either endpoint of the outcome range.Source: §B.2 · p. 16

• The paired design crosses agent versus uniform document choice with agent-controlled versus fixed-length search, repeating each condition three times.It yields 4,800 trajectories across 1,200 task–model–replicate blocks; fixed-horizon runs execute ten rounds.Source: C PAIRED INTERVENTIONS ON LLM SEARCH · p. 17
• Stopping effects have low repeatability across runs, unlike selection effects, supporting continuous intervention estimates with uncertainty rather than single mechanism labels.Stopping-effect ICCs are 0.099 and 0.102, while selection-effect ICCs are 0.679 and 0.685 for OLMo and Qwen.Source: C PAIRED INTERVENTIONS ON LLM SEARCH · p. 17
• The selection-policy effect correlates only weakly with CESS contrasts, and correlations approach zero when later queries are held fixed.The correlations are 0.269 and 0.236 for OLMo and Qwen; under common-query replay, they are −0.027 and 0.086.Source: C PAIRED INTERVENTIONS ON LLM SEARCH · p. 17

• The evaluation preserves the public agent’s adaptive query generation, evidence use, and synthesis while replacing only its search tool with fixed candidate documents.Source: D TRANSFER TO A PUBLIC DEEP RESEARCH… · p. 17
• The paired test uses 36 tasks, two rankings, and three seeds, with four searches per trajectory and identical pre-intervention state and Gumbel draws within each pair.Source: D TRANSFER TO A PUBLIC DEEP RESEARCH… · p. 17
• At K = 2, CESS lowers MAE from 0.5727 to 0.2287, ranking sensitivity from 0.5849 to 0.0746, and direction error from 0.4907 to 0.3102 versus Opened Mean.Source: D TRANSFER TO A PUBLIC DEEP RESEARCH… · p. 17
• At K = 4, CESS achieves 0.0461 lower MAE than outcome-regression shrinkage at matched sensitivity, while the K = 2 MAE difference is inconclusive.The K = 2 matched-MAE sensitivity difference is 0.0100 with a 95% interval of [−0.1780, 0.0796].Source: D TRANSFER TO A PUBLIC DEEP RESEARCH… · p. 18
• By round four, 63.9% of paired trajectories have different queries and 75.0% have different document sequences, so the ranking test includes adaptive divergence.Source: D TRANSFER TO A PUBLIC DEEP RESEARCH… · p. 18

• The evaluated CESS variant moves 20% from the Opened Mean toward sequential correction, with its coefficient tuned on separate questions and fixed across 36 topics.The evaluation uses non-medical PERSPECTRUM topics and up to five search rounds.Source: E FIVE-ROUND PERSPECTRUM EVALUATION · p. 18
• 10.2% lower MAE, from 0.4734 to 0.4250, accompanies reductions in RMSE, within-question bias, and document-order sensitivity.The paired MAE difference is −0.0484 (95% CI: [−0.0569, −0.0395]); wrong-direction conclusions rise by 0.23 percentage points with an interval including zero.Source: E FIVE-ROUND PERSPECTRUM EVALUATION · p. 18
• Across query models and search controllers, MAE falls by 9.7% to 10.7%.Source: E FIVE-ROUND PERSPECTRUM EVALUATION · p. 18

F TEN-ROUND PERSPECTRUM EVALUATION

• The evaluation tunes on 267 questions and tests on 36 with no question or document shared; every candidate has positive selection probability under the specified controller.The LLM generates queries from summaries of documents read, while the controller uses relevance, visible evidence direction, repetition penalties, and ranking order.Source: F TEN-ROUND PERSPECTRUM EVALUATION · pp. 18–19
• Selection weights contribute to sensitivity reduction under both tested clipping orders, while the MAE effect depends on configuration.The four calculations use the same runs, initial prediction, shrinkage coefficient, and stopping correction; the unshrunk correction falls outside [−1, 1] in 13.62% of runs.Source: F TEN-ROUND PERSPECTRUM EVALUATION · p. 19, Table 9 · p. 20
• MAE falls from 0.4035 after one round to 0.3103 after ten, with CESS improving over Opened Mean at every reported round.Setting the stopping probability to 1 gives MAE 0.3146 versus 0.3103 for full CESS; the paired 95% CI for the difference includes zero.Source: F TEN-ROUND PERSPECTRUM EVALUATION · p. 20, Table 10 · p. 20
• Weighted estimates have effective sample size 4.35 at the median and 1.61 at the fifth percentile, and the ten-round runs do not assess confidence in individual report conclusions.They lack report-level probabilities or confidence intervals calibrated for adaptive search; the reported intervals describe average comparisons across questions.Source: F TEN-ROUND PERSPECTRUM EVALUATION · p. 20

G ACCURACY, RANKING STABILITY, AND MODEL PERTURBATIONS

• Across 28 dataset–model–budget–prediction settings, simpler methods are compared with CESS at matched error and matched ranking sensitivity using interpolation without extrapolation.The table’s counts are descriptive and are not adjusted for multiple comparisons.Source: G ACCURACY, RANKING STABILITY, AND… · p. 20, Table 11 · p. 21
• In controlled simulations with 8,000 evaluation questions, accuracy declines gradually as the outcome predictor weakens, while visible-feature probability estimates closely match exact probabilities using the same linear predictor.Under extremely limited overlap, tuning selects λ = 0 and falls back to the full-pool prediction rather than amplifying large inverse-probability weights.Source: G ACCURACY, RANKING STABILITY, AND… · p. 20, Table 12 · p. 21
• In deployed audits, CESS uses recorded controller probabilities and reports an interval rather than a point estimate when a candidate has zero selection probability.The simulation’s limited-overlap condition reduces exploration; its oracle predictor follows the data-generating rule, while its reduced linear predictor omits an outcome-relevant feature.Source: G ACCURACY, RANKING STABILITY, AND… · p. 21