PaperlayerView exact revision on arXiv

Source-linked AI summary

Inspire: Benchmarking Scientific Literature Search for Open Research Problems

Jianrong Ding, Zhengyan Shi, Jianyuan Zhong, Kai Qiu, Qi Dai, Yifan Yang, Chong Luo, Qiang Xu

arXiv:2609.33233v1cs.IRcs.AI

TL;DR

Scientific literature search often starts with an open problem rather than a known target or fixed candidate set. INSPIRE benchmarks agents searching prior literature with solution-redacted briefs and logged stagewise evaluation. Current agents recover isolated antecedents more readily than broader portfolios, and exposure is their largest observed bottleneck.

  • Problem

    Scientific search from open problems lacks a known target paper or fixed candidate set, while useful antecedents may not share the problem’s vocabulary.

    Source: §1 · p. 1

  • Method

    INSPIRE evaluates ranked search outputs against hindsight-revealed graded antecedents on a shared open-corpus trajectory, separating exposure, selection, and ranking.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Current agents recover isolated antecedents more readily than broader portfolios, with resource exposure the largest observed loss; the strongest model reaches 0.284 nDCG@10.

    Source: §1 · p. 2, §7 · pp. 9–10

  • Takeaways & Limitations

    Matched hindsight SFT improves all three stages under unchanged test-time information, supporting INSPIRE as an actionable learning signal for targeted model development.

    Source: §7 · p. 10

    The score measures recovery of graded antecedents from one realized lineage, not scientific usefulness beyond that lineage or causal credit.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.

1 INTRODUCTION

Scientific search from an unresolved problem requires agents to form and revise a working relevance criterion while searching. INSPIRE evaluates exposure, selection, and ranking in an open-corpus trajectory and reveals where useful antecedents are lost.

1 INTRODUCTION

  • An unresolved problem leaves useful mechanisms potentially outside its vocabulary, so the agent’s provisional relevance criterion shapes what it finds and retains.Retrieved evidence can revise that criterion and redirect subsequent search.Source: §1 · p. 1
  • Existing benchmarks often stabilize correctness through clues, explicit conditions, anchors, feedback, or candidate pools, and do not measure all three stages on a common trajectory.INSPIRE’s contributions include a controlled multi-model comparison and a matched Base–SFT intervention that improves held-out targets.Source: §1 · pp. 2–3
  • INSPIRE evaluates resource exposure, selection, and ranking on one open-corpus trajectory using a solution-redacted brief and hidden target-specific antecedents.After the episode, graded antecedents from the target’s realized lineage support stage-specific measurements that show whether evidence was not surfaced, omitted, or poorly ordered.Source: §1 · p. 2
  • 0.284 nDCG@10 is the strongest model’s score; it finds at least one antecedent on 72.3% of targets but three on only 32.1% of eligible targets.Across models, zero-exposure rates range from 17.4% to 56.1%, with the largest observed loss before selection.Source: §1 · pp. 2–3

2 A COUPLED VIEW OF AGENTIC SEARCH

Scientific search comprises three observable but coupled stages: exposure, selection, and ranking. In INSPIRE, agents must form a working relevance criterion from the problem and revise it as evidence emerges.

2.1 RESOURCE EXPOSURE, SELECTION, AND RANKING

  • Exposure asks whether a graded paper enters the observed set, selection whether it remains in the submitted set, and ranking how effectively it is ordered.Zero exposure sets a hard ceiling for answers built from the observed trajectory; positive exposure alone does not ensure retention or an effective rank.Source: §2.1 · p. 3
  • The stages are coupled: a provisional relevance hypothesis can guide search and appraisal, while retrieved evidence can update that hypothesis and redirect later actions.The analysis model represents the same evolving criterion as affecting search, paper utility, and belief updates.Source: §2.1 · p. 3

2.2 SUPPLIED AND ENDOGENOUS RELEVANCE CRITERIA

  • INSPIRE’s criterion is endogenous: agents form the working criterion that drives their trajectory, while hindsight supplies an exact graded score after the episode.Its distinctive combination includes solution redaction, interactive date-gated open-corpus search, no anchor or clarification, and logged stage measurements.Source: §2.2 · pp. 3–4

3 INSPIRE BENCHMARK

INSPIRE evaluates literature search using solution-redacted briefs, target-specific cutoffs, and graded antecedents from each target’s realized citation lineage. Its construction keeps the target and evaluation labels hidden from agents while logged trajectories support stage-wise analysis.

3.1 TASK

  • Each task uses a solution-redacted brief, a cutoff three months before the target, and hidden graded antecedents from the target’s cited lineage.Agents see the brief and tool history, but not the target, labels, future citation contexts, or evaluation feedback.Source: §3.1 · p. 4

3.2 CONSTRUCTION

  • Targets are screened and references posted on or after the three-month cutoff are excluded; judges grade eligible cited references before the target identity and grades are withheld from agents.The benchmark constructs briefs that remove the target identity, realized mechanism, and target-specific results, yielding silver labels for a realized citation lineage.Source: §3.2 · p. 4

3.3 STAGE-WISE METRICS

  • Stage-wise metrics use only papers surfaced in logged search trajectories to distinguish exposure, selection, and ranking losses on a shared normalized nDCG scale.The decomposition does not identify the cause of exposure losses; causal attribution requires intervention.Source: §3.3 · p. 5

4 EXPERIMENTS AND RESULTS

Across the shared evaluation setup, Claude Opus 5 leads the end-to-end comparison, but agents recover broader portfolios less often than isolated antecedents. The stage-wise results locate the largest observed loss in resource exposure, while matched hindsight SFT improves held-out performance across the search pipeline.

4.1 SETUP

  • The evaluation gives every system the same brief, date filter, returned fields, budget, output limit, and scorer, while hidden targets and labels remain outside policy history.Search actions and returned identifiers are logged to reconstruct first exposure and the funnel; parametric memory, provider effects, and possible contamination remain uncontrolled sources of variation.Source: §4.1 · p. 6

4.2 END-TO-END AND PORTFOLIO PERFORMANCE

  • Claude Opus 5 leads with 0.284 nDCG@10, 0.215 Recall@10, and 0.723 AnyHit@10, while Claude Sonnet 5 follows at 0.189 nDCG@10.The ordering is broadly consistent across nDCG, recall, MRR, and AnyHit@10.Source: §4.2 · p. 6
  • 72.3% of eligible targets yield at least one antecedent for the leading model, but only 32.1% yield three and 6.7% yield five.Across systems, recovered counts concentrate at zero to two; Figure 3 plots antecedent counts in the top 10 against the percentage of targets, with failures and non-submissions in the zero bin.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 STAGE-WISE PERFORMANCE

  • Resource exposure is the largest observed loss: exposed gain ranges from 0.153 to 0.480, and zero-exposure rates range from 17.4% to 56.1%.For every system, the deficit 1 − G_exposed is at least 0.520 and reaches 0.847; the stage funnel compares exposed gain, selected gain, and final nDCG@10.Source: §4.3 · p. 7, Table 3 · p. 7, Figure 4 · p. 8
  • Selected gain ranges from 0.119 to 0.386; Sonnet and GPT-5.6-Sol expose nearly equal gain, 0.323 and 0.326, but retain 0.269 and 0.233, respectively.Sonnet's higher final nDCG@10 is 0.189 versus 0.164; small selection gaps alone do not establish strong selection when little useful evidence was exposed upstream.Source: §4.3 · pp. 7–8
  • Final nDCG@10 ranges from 0.082 to 0.284, while Rank quality ranges from 0.683 to 0.742; effective ordering cannot compensate for evidence never surfaced.The selected-gain-to-final-score gap isolates loss from submitted order, because selected gain is the best score achievable by reordering the same submission.Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4 MATCHED HINDSIGHT SFT

  • Matched hindsight SFT raises held-out final nDCG@10 from 0.082 to 0.166 under identical test-time information and evaluation conditions.The controlled comparison reports stage-wise changes in exposure, selection, and ranking; SFT raises exposed gain from 0.153 to 0.251 and selected gain from 0.119 to 0.219, while Rank quality rises from 0.683 to 0.759.Source: §4.4 · pp. 8–9, Table 4 · p. 8

5 RELATED WORK

Related work places INSPIRE among research on interactive scientific search, literature discovery, and search agents that combine retrieval with reasoning and tool use. It also connects the benchmark to hindsight supervision and portfolio-oriented retrieval evaluation.

Scientific search

Hindsight supervision

  • Prior work links temporal isolation and future-prior-work search with graded cumulative gain, diversified retrieval, citation-function labels, and learning from privileged or hindsight information (Vu et al., 2024; Ajith et al., 2026; Chamoun et al., 2026; Wu et al., 2026; Wang et al., 2026; J¨arvelin & Kek¨al¨ainen, 2002; Carbonell & Goldstein, 1998; Clarke et al., 2008; Teufel et al., 2006; Valenzuela et al., 2015; MacRoberts & MacRoberts, 1989; Bornmann & Daniel, 2008; Vapnik & Vashist, 2009; Andrychowicz et al., 2017).The related work also situates replay-valid demonstrations within supervised instruction tuning with LoRA.Source: §5 · p. 9

6 LIMITATIONS

The benchmark's citation labels are capped and drawn from recent arXiv citations, so they may omit uncited influences, older foundations, and alternative solution lineages.

6 LIMITATIONS

  • INSPIRE's current capped, recent arXiv citation labels may omit uncited influences, older foundations, and alternative solution lineages.The authors propose human missing-gold audits and further study across disciplines, search backends, judge families, students, training seeds, provider regimes, and controlled parametric-memory settings.Source: §6 · p. 9

7 CONCLUSION

The conclusion frames scientific search as resource exposure, selection, and ranking, evaluated together on logged trajectories using solution-redacted problems and hindsight-revealed graded antecedents. Agents more readily recover isolated antecedents than broader portfolios, and the largest observed loss is in exposure; matched hindsight SFT improves all three stages under unchanged test-time information.

7 CONCLUSION

  • INSPIRE evaluates resource exposure, selection, and ranking on the same logged trajectory using solution-redacted problems, date-gated open-corpus search, and hindsight-revealed graded antecedents.Source: §7 · p. 9
  • Current agents recover isolated antecedents more readily than broader portfolios, with the largest observed loss in resource exposure and further losses in selection and ranking.Source: §7 · pp. 9–10
  • Matched hindsight SFT improves all three stages under unchanged test-time information, and the decomposition enables targeted model development by distinguishing exposure, selection, and ranking losses.Source: §7 · p. 10

AI USE STATEMENT

  • The authors used LLMs for benchmark construction, grading and auditing, synthetic search demonstrations, agent evaluation, and manuscript language assistance, and state that they reviewed and revised AI-assisted text.Source: AI USE STATEMENT · p. 10

A DATASET CONSTRUCTION

INSPIRE pairs solution-redacted briefs with target-specific cutoffs and automatically graded pre-cutoff cited antecedents. Its construction screens recent, visible, sufficiently cited computer-science work and audits briefs for residual target clues.

A.1 OVERVIEW

  • Each retained instance pairs a solution-redacted problem brief and public-record cutoff with graded pre-cutoff cited antecedents annotated by tier and role.Table 5 reports train and evaluation statistics separately; antecedents represent each target’s realized citation lineage.Source: §A.1 · p. 14, Table 5 · p. 14

A.2 TARGET DISCOVERY

  • Target discovery filters out surveys and reviews and rejects papers lacking parseable arXiv HTML or at least two graded pre-cutoff antecedents.These choices define a population of recent, visible, sufficiently cited work.Source: §A.2 · p. 14

A.3 REFERENCE GRADING

  • References dated on or after the target-specific cutoff are excluded, and the active label set retains references within four years, removes T0, and caps the gain-sorted list at ten.Older foundations and overflow remain retrievable but unlabeled; predictions outside the labeled set receive zero benchmark credit, though they may remain scientifically useful.Source: §A.3 · p. 14

A.4 BRIEF GENERATION AND SCREENING

  • Brief generation removes the target name, mechanism, and target-specific results, while a model-based audit flags likely target guesses and defines automatic-clean and flagged partitions.The audit can leave mechanism-level clues, and its slices characterize screening outcomes rather than independently validating every redaction.Source: §A.4 · pp. 14–15, Figure 5 · p. 15

B EXPERIMENTAL DETAILS

The shared experimental protocol records configurations and episode trajectories, checks comparability across runs, and reports completion counts before aggregating results.

B.1 SHARED PROTOCOL

  • Run manifests record target lists, cutoffs, prompts, tool and scorer hashes, budgets, model settings, and software details for each evaluation row.The same interface and budget are used across the evaluated models; development replay checks date filtering, normalization, output limits, and failure-as-zero scoring.Source: §B.1 · p. 15
  • Episode records preserve ordered queries, returned IDs, first exposures, errors, final rankings, terminal status, calls, tokens, latency, and available cost.A comparability check verifies matching targets, briefs, returned fields, cutoff logic, budget, submission schema, and scorer.Source: §B.1 · p. 15
  • The reporter emits expected, attempted, completed, submitted, and scored counts before computing aggregates.Source: §B.1 · p. 15

B.2 MODEL EVALUATION

Model evaluation uses shared targets and a common interface, while trajectory metrics separate exposure, selection, and ranking. The appendix also specifies how replay-valid demonstrations are constructed and how the SFT adapter is trained.

B.2 MODEL EVALUATION

  • Evaluation joins each checkpoint’s outputs by target, uses paired target-bootstrap intervals, and reports descriptive automatic slices with denominators.The main table excludes models with differing policy-visible interfaces or search budgets; outputs are canonicalized to arXiv base IDs and invalid submissions score zero.Source: §B.2 · pp. 15–16

B.3 TRAJECTORY METRICS

  • Trajectory reporting measures exposure, retention, gain, rank quality, and search behavior, with checks reconciling final-stage means and gain bounds.A cross-target join measures backend reachability, but does not test target-specific association or selection.Source: §B.3 · p. 16

B.4 SFT CONFIGURATION

  • The SFT teacher proposes trajectories using private training antecedents, but each retained trajectory must replay through ordinary tools and expose any submitted identifier.Training uses action-token loss for three epochs with LoRA rank 16, scale 32, zero dropout, and learning rate 5 × 10−6; evaluation differs from the base model only by adapter.Source: §B.4 · p. 16

C HINDSIGHT SFT DETAILS

Replay-valid hindsight demonstrations are selected using privileged training-only antecedent information while keeping the student’s test-time observations unchanged. On matched held-out evaluation, SFT improves final nDCG@10 and gains at each reported trajectory stage.

Training protocol

  • The teacher may inspect training-only antecedent tiers, titles, and abstracts, but never identifiers; retained actions must be replayed and surfaced through ordinary search tools.The student receives only deployment transcripts, with private teacher text and demonstration-selection scores removed, so hindsight selects training behavior without expanding test-time observations or actions.Source: C HINDSIGHT SFT DETAILS · p. 16

Matched evaluation

  • 0.166 versus 0.082 nDCG@10: matched SFT raises final performance by 0.084, with a 95% paired target-bootstrap interval of [0.055, 0.115].The matched evaluation changes only the adapter; Figure 6 shows score distributions and aggregate exposure, selection, and final-score gains.Source: C HINDSIGHT SFT DETAILS · pp. 16–17, Figure 6 · p. 17
  • Figure 6 reports improvement in Exposed gain, Selected gain, and final nDCG@10 under the held-out protocol.Its left panel compares matched Base and SFT failure-as-zero nDCG@10 distributions, including probability mass at zero.Source: Figure 6 · p. 17
  • The reported interval reflects variation over the shared target set, while training-seed variation remains outside the uncertainty estimate for the single frozen adapter.Source: C HINDSIGHT SFT DETAILS · p. 17

D EXPOSURE GAIN OVER SEARCH

A prefix-based oracle tracks how much graded antecedent gain is available as search progresses, separating exposure from later selection and ordering losses. Hindsight-SFT exposes higher-value candidate sets earlier and maintains an advantage over Base, while useful evidence continues to arrive after the midpoint.

D EXPOSURE GAIN OVER SEARCH

  • A prefix-based oracle evaluates how well exposed graded antecedents could be ranked if their gains were known and ungraded papers discarded.Its nondecreasing curves end at the Exposed gain values in Tables 3 and 4; normalized progress enables timing comparisons across trajectories, but absolute calls and returned-paper counts may differ, and the retrospective diagnostic does not itself test budget interventions.Source: D EXPOSURE GAIN OVER SEARCH · p. 17, Figure 7 · p. 18
  • At 50% normalized progress, systems have accumulated approximately 64–74% of their eventual oracle gain, and no curve has plateaued early.Even the strongest model continues to gain exposed value in the second half; Gemini 3.1 Pro Preview and Qwen3.6-35B-A3B Base remain low throughout, reflecting slower useful-antecedent accumulation relative to stronger systems.Source: D EXPOSURE GAIN OVER SEARCH · pp. 17–18, Figure 7 · p. 18
  • 0.251 versus 0.153 oracle nDCG@10 at trajectory end, with hindsight-SFT reaching approximately 0.16 by 30% progress—already slightly above Base’s complete-trajectory value.The SFT advantage persists throughout the trajectory; Figure 8’s curve separation measures additional graded gain made available during search, independently of later selection and ordering.Source: Figure 8 · p. 18, D EXPOSURE GAIN OVER SEARCH · p. 18
  • From 50% to 100% progress, oracle nDCG@10 rises by approximately 0.06 for both Base and SFT, while SFT’s endpoint oracle of 0.251 exceeds its final nDCG@10 of 0.166.The curves indicate that continued exploration retains value and that improved exposure still leaves downstream selection and ordering headroom; identifying the learned decisions behind the shift requires a targeted intervention.Source: D EXPOSURE GAIN OVER SEARCH · p. 18, Figure 8 · p. 18

E PROMPT TEMPLATES

  • The search-agent template asks for varied searches, abstract or full-text inspection, and a ranked list of 5–10 papers prioritizing load-bearing mechanisms over topical background.The brief-generation template frames the problem before its solution and redacts the target’s method name, solution mechanism, and unique result, benchmark, or dataset names.Source: E PROMPT TEMPLATES · pp. 19, 21
  • The prior-reference judge assesses how load-bearing each candidate was, using target and candidate abstracts and target full text only when available.The templates specify truncated abstracts and conditional full-text and global-citation inputs.Source: E PROMPT TEMPLATES · p. 19
  • The judging rubric reserves T3/T2 for specialized backbone or component predecessors and downgrades ubiquitous foundations, even when technically cited.It distinguishes T1 background and T0 incidental references, classifies references as conceptual or infrastructural, and directs judges to use in-context citation evidence when target full text is available.Source: E PROMPT TEMPLATES · pp. 19–20

Related papers

Loading 2609.33233v1…