Source-linked AI summary
Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts
Hongmin Li, Wanli Zhao
TL;DR
Final accuracy alone does not show whether a scientific-search agent exposed or inspected a target paper. The paper audits these events through decision checkpoints and finds distinct question-level outcomes across search policies, including different exposure and inspection patterns despite matching aggregate accuracy.
Problem
Final paper-identification accuracy does not reveal whether an agent failed to expose the target, left an exposed target uninspected, or attempted inspection.
Source: §1 · p. 1Method
Results
Keyword search has fewer failures without target exposure or inspection than raw search, but more after exposure without inspection; read-first and keyword have identical aggregate accuracy despite different question-level inspection outcomes.
Source: §7 · p. 9Takeaways & Limitations
Measuring exposure and inspection alongside final accuracy reveals search-behavior and question-level differences that aggregate scores alone obscure.
Source: §8 · p. 10The study is limited to one answerable benchmark subset in a target-enriched environment, and incomplete model specifications, absent repeated executions, and retained excerpts constrain causal and semantic conclusions.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Final-answer accuracy does not reveal whether a scientific-search agent failed to encounter a target paper, attempt to inspect it, or return an accepted answer after inspection. We introduce decision checkpoints that record observations and tool actions without benchmark labels during inference, then join target identities and evaluator labels to assign outcome categories from recorded events. Across five conditions on 540 answerable AutoResearchBench Deep questions in a fixed, target-enriched environment, keyword search achieves 24.6\% accuracy, compared with 17.8\% for raw search. The keyword condition has fewer incorrect answers with neither target exposure nor inspection, but more incorrect answers after the target is exposed and left uninspected. Compared with keyword search, read-first has 27.4\% more recorded evidence-search calls. Target inspection attempts occur on 199 questions under read-first and 191 under keyword search; both conditions achieve 24.6\% accuracy. The checkpoint protocol makes these question-level differences explicit, distinguishing target exposure and inspection from aggregate accuracy and total tool use.
1 Introduction
Final paper-identification accuracy leaves unresolved whether an agent failed to expose the target, exposed it but did not inspect it, or attempted inspection. The paper introduces decision checkpoints to connect these events with answer correctness across search conditions.
1 Introduction
- A wrong answer may follow failure to expose the target, leaving an exposed target uninspected, or attempting to inspect it.Distinguishing these events specifies which part of a search trajectory an intervention is intended to change.Source: §1 · p. 1
- Prior work tracks evidence pipelines, manually analyzes benchmark errors, or diagnoses selected failed trajectories; this study compares exposure, inspection, and accepted answers across the same questions.The paper presents its measurement question as complementary to those approaches.Source: §1 · p. 1
- Decision checkpoints distinguish target exposure and executed inspection attempts from final answer correctness, making search trajectories auditable without benchmark labels during inference.Target identities and evaluator labels are joined to recorded events afterward.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Across five conditions on 540 answerable AutoResearchBench Deep questions, keyword search has fewer failures without exposure or inspection than raw search, but more failures after exposure without inspection.The comparisons use a shared, target-enriched environment.Source: §1 · p. 2
2 Related Work
Related work measures retrieval, evidence use, process progress, and resource allocation through different methods. This paper complements those approaches by measuring target exposure and executed inspection attempts alongside answer outcomes.
Scientific search and stage measurement
- PaperQA2 tracks source retention through retrieval, chunk ranking, summarization, and attribution, while this study links exposure and inspection events to paired answer outcomes.The paper characterizes PaperQA2 as a direct precedent for separating discovery from downstream evidence use.Source: §2 · pp. 2–3
Process and evidence evaluation
- Process-evaluation systems track progress, tool-call milestones, or structured steps, while this study adds target-identity and executed-action labels across successes and failures.Semantic evidence diagnostics instead assess rationale support, refutation, or claim-level entailment.Source: §2 · p. 3
Retrieval access and resource allocation
- Prior resource-allocation work varies document access or optimizes search budgets; this study measures inspection effort and how often it reaches the target under a gate that changes call timing.Source: §2 · p. 3
3 Decision-Checkpoint Auditing
Decision-checkpoint auditing records search trajectories as states, actions, observations, and costs, then matches target identities and evaluator labels after inference. It measures target exposure and inspection attempts as distinct events.
Trajectory representation
- Each checkpoint records the maintained candidate list, prior queries, inspection attempts, retained excerpts, proposed or substituted action, observation, and token use and latency.The analytical state summary is not added to the model’s input context.Source: §3 · p. 3
Retrieval, exposure and inspection
- Exposure occurs when the target identifier appears in a presented search-result list; inspection occurs when an executed within-paper evidence-search call targets that paper.Inspection is an attempt to obtain evidence, not proof that sufficient evidence was acquired.Source: §3 · p. 3
Outcome categories
- After inference, accepted answers are classified as correct; incorrect observable trajectories are categorized by target inspection and exposure, with inspection taking precedence.API exceptions without trajectories are unobservable, and exposure and inspection are also measured independently.Source: §3 · pp. 3–4
Exposure rank and subsequent decisions
- For exposed-but-uninspected incorrect answers, a continuation indicator records whether another valid search, evidence-search, or answer action followed exposure.It does not estimate remaining inspection budget or the counterfactual benefit of inspection.Source: §3 · p. 4
Observation scope
- Exposure is checked against retained tool messages, but incomplete returns and retained excerpts prevent full reconstruction or semantic evaluation of every model input.Source: §3 · p. 4
4 Experimental Setting
The experiments compare five search conditions on answerable Deep questions using a shared retrieval backend and common nominal execution limits. The setup combines metadata and cached full-text retrieval, while answer correctness is evaluated separately from search events.
Task and sample
- 540 answerable questions from the 600-question Deep split are evaluated; the 60 explicitly unsatisfiable questions are excluded, so abstention on unanswerable questions is not assessed.The study is restricted to Deep and one model request alias; 77 questions informed exploratory analysis and intervention design.Source: §4 · p. 4
Retrieval environment
- The +both backend combines BM25 retrieval over 3.16 million arXiv metadata records with BM25 retrieval over 7,703 cached full-text papers, fused by reciprocal rank.The full-text collection includes 3,853 targets and 3,850 matched non-target papers; target identities inform its construction.Source: §4 · p. 4
Search policies
- Raw search uses the complete question verbatim, keyword search uses short distinctive terms, and read-first adds an inspection prompt and gate to keyword search.Decomp searches focused sub-questions; adaptive iteratively revises queries until evidence and candidates stabilize.Source: §4 · pp. 4–5
- Raw and keyword search differ in failure counts: neither exposed nor inspected falls from 333 to 64, while exposed but not inspected rises from 67 to 277; Acc@1 is 17.8% and 24.6%, respectively.Source: Figure 2 · p. 5
Execution and resource measurement
- All conditions share the retrieval backend and nominal limits, and are evaluated with Deep Accuracy@1 alongside tool use and termination measures.Episodes reaching round or context limits remain in the analysis; original model identity and judge settings are not fully specified, so policy comparisons are descriptive.Source: §4 · p. 5
5 Target Exposure and Failure Patterns
Across the same 540 questions, keyword search has fewer failures without target exposure than raw search, but more failures after exposure without inspection. Paired outcomes and exposure-rank patterns show that target discovery does not necessarily lead to an inspection attempt.
5 Target Exposure and Failure Patterns
- Keyword search has fewer never-exposed failures and more exposed-but-uninspected failures than raw search across the same 540 questions.The categories are mutually exclusive; exposure and inspection refer to the target paper.Source: §5 · pp. 5–6, Table 1 · p. 6
- 6.85 percentage points is keyword search’s accuracy advantage over raw search, while failures without target exposure shift from 62% to 12%.The paired comparison covers the same questions under both conditions.Source: §5 · p. 6, Table 2 · p. 6
- 182 of 333 raw questions with neither exposure nor inspection instead fail after exposure without inspection under keyword search.Conversely, 36 questions answered correctly under raw become exposed-but-uninspected failures under keyword search.Source: §5 · p. 6, Table 2 · p. 6
- All 191 keyword target-inspection attempts occur on a later turn after first exposure, and rank associations do not establish a causal effect of rank.Rank, exposure timing, and subsequent trajectory length are uncontrolled.Source: §5 · p. 7, Table 3 · p. 7
- All 277 keyword failures after exposure without inspection contain a subsequent valid action, but this does not quantify unused budget.335 of 476 exposed keyword episodes first show the target by turn 3; the median number of subsequent search, evidence-search, or answer actions is 13. Execution limits occur in 167 of 277 keyword cases versus 20 of 67 raw cases.Source: §5 · p. 7
- Among keyword questions with first exposure in the top 20, 184 of 336 have no later target inspection; at ranks 1–5, 83 of 175 have no later attempt.Percentages use exposed questions in each rank group as denominators; these groups include correct and incorrect answers.Source: §5 · p. 7, Table 3 · p. 7
6 Read-First Gate: Inspection and Answer Outcomes
Read-first adds an inspection gate, but it does not improve aggregate correctness over keyword search. Paired outcomes and resource measures show why target-specific inspection and total tool use need separate accounting.
6 Read-First Gate: Inspection and Answer Outcomes
- The gate requires an inspection attempt before another search or final answer when an eligible unread candidate remains, substituting a call on the highest-ranked eligible candidate if needed.An agent-initiated inspection satisfies the gate regardless of candidate rank or query.Source: §6 · p. 7
- Both keyword and read-first answer 133/540 questions correctly, but 60 questions favor each policy exclusively.The paired correctness comparison is reported in Table 4.Source: §6 · p. 8, Table 4 · p. 8
- Among 5,198 observable read-first evidence-search calls, 1,720 are gate-forced and 3,478 agent-initiated; target inspection occurs through both sources, with overlap.Gate-forced calls reach the target on 22 questions and agent-initiated calls on 187, with ten-question overlap.Source: §6 · p. 8
- Read-first records 27.4% more evidence-search calls and 4.8% more inference tokens than keyword, yet target inspection occurs on 199 versus 191 questions.The target-inspection counts are overall counts; the paired observable sample contains 199 versus 190.Source: §6 · p. 8, Table 5 · p. 9
- Sensitivity checks retain a zero read-first–keyword accuracy difference under identity scoring, while question-bootstrap intervals for that difference span zero.A separate retry exposed and inspected the target but still answered incorrectly; the primary analysis retains the original episode.Source: §6 · pp. 8–9
7 Discussion and Limitations
The discussion separates target visibility, inspection, and answer correctness because aggregate accuracy can mask different question-level outcomes. The findings remain descriptive, with limited generalizability and no identified causal mechanism.
7 Discussion and Limitations
- Visible targets can remain uninspected, including targets near the top of the result list; inspected-but-wrong answers cannot be attributed to reasoning without evidence of excerpt adequacy.The discussion motivates examining candidate selection and post-exposure resource allocation alongside retrieval.Source: §7 · p. 9
- More evidence-search calls coincide with only a modest difference in target-inspection coverage, while paired comparisons show substantial changes in inspection status and correctness.Identical aggregate accuracy can therefore mask different question-level outcomes.Source: §7 · pp. 9–10
- One answerable benchmark subset, target-enriched retrieval, no repeated executions, and retained excerpts that cannot fully reconstruct or semantically assess model inputs constrain interpretation.The authors characterize the findings as descriptive, with limited generalizability and no identified causal mechanism.Source: §7 · p. 10
8 Conclusion
Decision-checkpoint auditing distinguishes target exposure, inspection attempts, and answer correctness in scientific search. Across 540 answerable questions, keyword search has fewer failures without exposure or inspection than raw search, while read-first makes more evidence-search calls than keyword search despite similar inspection counts and equal accepted answers.
8 Conclusion
- Both policies answer 133 questions correctly, with 60 gains and 60 losses in their paired comparison.The comparison covers 540 answerable AutoResearchBench Deep questions in a target-enriched environment.Source: §8 · p. 10
- Keyword search has fewer failures without target exposure or inspection than raw search, but more failures after exposure without inspection.These differing failure patterns distinguish search outcomes that aggregate accuracy alone obscures.Source: §8 · p. 10
- Read-first makes 27.4% more evidence-search calls than keyword search, while target inspection attempts occur on 199 rather than 191 questions.Both conditions achieve 24.6% accuracy.Source: §8 · p. 10
A Supplementary Raw and Keyword Comparisons
Supplementary raw and keyword statistics quantify how often keyword-search errors combine early target exposure with no inspection. The reported overlap is a lower bound, not a count of all uninspected top-20 episodes.
A Supplementary Raw and Keyword Comparisons
- At least 137 keyword questions were both exposed to the target within rank 20 and answered incorrectly without target inspection.The bound follows from 336 top-20 exposures and 277 incorrect, uninspected answers among 476 keyword questions with target exposure.Source: A Supplementary Raw and Keyword… · p. 13, Table 6 · p. 13
- The 137-question lower bound concerns incorrect answers, whereas the 184 uninspected top-20 episodes include correct and incorrect answers.Source: A Supplementary Raw and Keyword… · p. 13
- Table 6 summarizes supplementary exposure, outcome, and resource statistics for raw and keyword search on 540 questions per condition.Source: Table 6 · p. 13
B Measurement and Scoring Procedures
The measurement procedures define target exposure and inspection from recorded search events, score answers separately, and describe how comparisons and uncertainty are assessed. They also specify the limits of recorded excerpts and exploratory semantic-support checks.
B Measurement and Scoring Procedures
- Target identifiers are normalized across whitespace, repository prefixes, version suffixes, and modern or legacy arXiv formats; exposure is the target’s first search-result appearance, and inspection requires an executed call directed at it.Proposed but unexecuted actions do not count as inspection, and repeated identifiers outside search observations do not count as new exposure.Source: B Measurement and Scoring Procedures · p. 13
- Recorded target excerpts cover up to the first 1,800 characters of the latest executed target-call response, and episode coverage requires at least one matching excerpt.This measure does not establish sufficient evidence for the question or represent complete tool returns or model inputs.Source: B Measurement and Scoring Procedures · p. 13
- Answer correctness uses a single-pass title-identity judgment, while a sensitivity re-evaluation of all 2,700 answers resolved and agreed with the primary labels.The judge does not independently verify question constraints, and the original judge configuration remains incompletely documented.Source: B Measurement and Scoring Procedures · pp. 13–14
- RAW prompts require verbatim questions while KW prompts request distinctive keywords; read-first inspection eligibility is limited to valid, uninspected candidates in the latest result list.The execution framework requires nonempty queries but does not enforce RAW’s verbatim instruction.Source: B Measurement and Scoring Procedures · p. 14
- Target inspection attempts occurred on 191 keyword questions and 199 read-first questions, with one unobservable read-first trajectory.The aggregate outcome table reports 133 accepted answers in each condition.Source: B Measurement and Scoring Procedures · p. 14
- Read-first recorded 5,198 evidence-search calls across 540 questions, including 1,720 gate-forced and 3,478 agent-initiated calls among 539 observable trajectories.These executed-call counts do not establish a counterfactual increase in inspection or semantic evidence sufficiency.Source: B Measurement and Scoring Procedures · p. 14
- Paired bootstrap intervals resample question-level correctness differences or target-identifier clusters, with cluster samples retaining all questions and multiplicity.Intervals use the 2.5th and 97.5th percentiles; resampling uses observed outcomes rather than repeating agent executions.Source: B Measurement and Scoring Procedures · pp. 14–15
- Event checks found no discrepancies in 40 stratified trajectories and five anomaly records, while semantic review of 26 excerpts from ten records did not establish full evidence sufficiency or independent human validation.Source: B Measurement and Scoring Procedures · p. 15