PaperlayerView exact revision on arXiv

Source-linked AI summary

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, Chelsea Finn

arXiv:2610.02202v1cs.AIcs.CLcs.IR

TL;DR

Identifying which prior ideas can advance a new research problem remains a key ability of human researchers. SCHOLARCATALYST collects firsthand judgments from project authors and tests retrieval from pre-project literature; the strongest system recovers only 48% of author-credited papers in its top 20.

  • Problem

    Choosing which existing idea a new problem should build on is still mostly left to human researchers, though an insightful prior paper can help shape a vague question.

    Source: §1 · p. 2

  • Method

    SCHOLARCATALYST asks authors of 207 recent computer science projects to identify and explain prior papers that could have advanced their work, then evaluates retrieval from a temporally filtered corpus.

    Source: §1 · p. 2

  • Results

    48% of author-credited papers is the strongest system’s top-20 recovery, while repeated agent querying does not help (42% vs. 48% for the retriever alone).

    Source: §1 · p. 2

  • Takeaways & Limitations

    If candidate coverage is limiting, progress will require embedding retrievers that surface inspiration papers and agents that build on those results to guide exploration.

    Source: §4.2 · p. 7

    The benchmark covers computer science papers from 2025–2026 with uneven representation across research areas, and its retrospective author labels are subject to hindsight.

    Source: §5 · p. 9

Abstract

from arXiv · show

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

SCHOLAR

The supplied material identifies the paper’s authors and affiliations and illustrates the distinction between topical similarity and papers that inspire research progress.

Authors and affiliations

  • The listed authors are Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko, Rulin Shao, Seungone Kim, Graham Neubig, Pang Wei Koh, Aakanksha Chowdhery, Akari Asai, Omar Khattab, Yejin Choi, Gunhee Kim, and Chelsea Finn.The affiliations listed include Stanford University, Seoul National University, the University of Washington, Carnegie Mellon University, the Allen Institute for AI, and MIT.Source: SCHOLAR · p. 1

Figure 1

  • Figure 1 contrasts a topically related paper that may not inspire progress with a less semantically similar paper that may offer a useful insight.Source: Figure 1 · p. 1

Authors and affiliations

  • The paper marks Sohyeon Kim and Yoonho Lee as equal contributors.Source: SCHOLAR · p. 1

1 INTRODUCTION

The introduction frames identifying useful prior ideas as a core research skill and presents SCHOLARCATALYST as an author-grounded benchmark for studying it. Current systems recover only a fraction of author-credited papers, and agentic search does not outperform embedding retrieval.

1 INTRODUCTION

  • Research taste includes recognizing when a prior idea can sharpen a vague question by revealing a limitation or suggesting an approach to adapt.Source: §1 · p. 2
  • Existing retrieval benchmarks and citation records do not establish which papers could have inspired an early-stage project, especially when the eventual paper did not cite them.Source: §1 · p. 2
  • The benchmark collects firsthand judgments from 184 researchers on 207 projects, validating 894 early-stage questions and labeling prior papers that could have advanced the work.The 191K-paper retrieval corpus is restricted to papers published before each project.Source: §1 · p. 2
  • The strongest system recovers only 48% of author-credited papers in its top 20 across sparse, dense, multi-vector, and agentic retrieval approaches.Agentic search retrieves 42% versus 48% for the embedding retriever.Source: §1 · p. 2

2 RELATED WORK

The related-work section distinguishes SCHOLARCATALYST from scientific-discovery systems and established literature-retrieval benchmarks. Its focus is latent inspiration relevance to early-stage research questions.

Scientific discovery using LLMs

  • Scientific-discovery systems draw on prior papers through citation graphs, metadata, or prebuilt candidate corpora, while SCHOLARCATALYST evaluates retrieval of insightful prior ideas for early-stage questions.Source: §2 · p. 3

Scientific literature retrieval

  • Existing scientific retrieval benchmarks use explicit evidence, facet, or information needs; SCHOLARCATALYST elicits authors’ firsthand judgments about relevance to their research process.Unlike citation-intent-derived labels, its judgments include relevance not documented in the completed paper.Source: §2 · p. 3

Information retrieval systems

  • Embedding and lexical retrievers rank by learned semantic similarity or term overlap, while search agents iteratively query those retrievers and still depend on their top-k candidates.Whether these approaches can retrieve work whose inspiration is not lexically explicit remains unclear.Source: §2 · p. 3

3 SCHOLARCATALYST: EVALUATING SCIENTIFIC LITERATURE RETRIEVAL WITH AUTHORS’ FIRSTHAND ACCOUNTS

SCHOLARCATALYST uses author judgments to evaluate retrieval of prior papers that could advance early-stage research. Its task definition, automated pipeline, and author-review process produce time-filtered queries, candidate papers, and relevance rationales.

3 SCHOLARCATALYST: EVALUATING SCIENTIFIC LITERATURE RETRIEVAL WITH AUTHORS’ FIRSTHAND ACCOUNTS

  • SCHOLARCATALYST asks project authors which prior papers did or could have helped advance their work and why, addressing knowledge largely absent from the public record.Source: §3 · p. 3

3.1 TASK DEFINITION

  • Queries reconstruct the project’s initial research landscape without revealing its eventual solution, using central CoreQ or subfield-specific SubQ questions.Source: §3.1 · p. 4
  • Positive papers are author-judged sources of inspiration, while hard negatives are reviewed and rejected despite being topically related to the query.Source: §3.1 · p. 4
  • The retrieval corpus contains papers published before the source paper to approximate the available literature, and performance is measured with Recall@N and nDCG@N.Source: §3.1 · p. 4

3.2 OUR AUTOMATED PIPELINE FOR EFFICIENT DATA COLLECTION

  • The automated pipeline needs only a source paper’s arXiv ID to prepare author-annotation materials and supports refreshing the benchmark as older instances risk model-training leakage.Source: §3.2 · p. 5
  • The retrieval corpus combines resolved source-paper references with 181K additional arXiv papers from 2020–2024 across major computer-science categories.The resulting corpus contains 190,896 papers.Source: §3.2 · p. 5
  • Pre-annotations propose a CoreQ, SubQs, and rationales; BM25 and Qwen3-Embedding-8B add uncited candidates, which Gemini 3.6 Flash reranks before ten candidates per query are retained.Source: §3.2 · p. 5

3.3 COLLECTING RELEVANCE SUPERVISION FROM AUTHORS

  • Authors can revise queries, judge each candidate, and explain each judgment; manual review left 764 of 894 queries unchanged and reworded others to remove solution-revealing phrasing.Source: §3.3 · p. 6

4 EVALUATION AND ANALYSIS

Across the temporally filtered retrieval evaluation, systems recover only part of the author-identified catalyst papers. The analyses point to limited candidate coverage and weak query similarity as barriers that persist across agentic search, reranking, and query rewriting.

4.1 EXPERIMENTAL SETUP

  • The evaluation tests lexical, multi-vector, dense, and agentic retrieval on a temporally filtered local corpus, with no web search and pre-source-paper model cutoffs in the main setting.The corpus contains 191K papers, smaller than the three million papers on arXiv.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 MAIN RESULTS

  • General-purpose dense retrievers place 39% of author-identified inspiration papers in the top 20 for CoreQ and 51% for SubQ; scientific-document retrievers trail by at least 16 and 27 points, respectively.BM25 and LateOn trail by 15–16 points on CoreQ and 18–25 points on SubQ.Source: §4.2 · p. 7, Table 3 · p. 7
  • Grep finds only 8% of gold query–paper pairs during search, versus 46% for the tool-calling agent, and agentic search does not improve over embedding retrieval.Figure 4 reports the share each GPT-4.1 agent encounters during search and returns in its final top 20.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 ANALYSIS OF AUTHOR-ANNOTATED CATALYST PAPERS

  • More than one inspiration type applies to 55.8% of rationales, and hard negatives are at least as similar to queries as positives on lexical and semantic measures.Authors often identify adapted methods, supporting evidence, generalizations, or limitations as inspirations.Source: §4.3 · p. 8
  • 43.6% of SubQ positives are uncited, while citation-intent labels identify only 46.2% of CoreQ and 34.0% of SubQ positives as uses or extends.Even given the finished source paper and bibliography, Gemini 3.1 Pro covers only 32.2% of CoreQ positives on average and finds none of the key inspirations in 28.0% of instances.Source: §4.3 · p. 8

4.4 ANALYSIS OF FACTORS BEHIND RETRIEVAL PERFORMANCE

  • Stronger rerankers gain more Recall@5 from gold injection, which replaces negative candidates with missing positive papers while keeping pool size fixed.This comparison indicates that reranker selection can benefit only when useful papers enter the candidate pool.Source: Figure 6 · p. 8, §4.4 · p. 9
  • Source title and abstract raise Recall@20 by 0.11 for CoreQ and 0.02 for SubQ; bibliography raises CoreQ by 0.35, while fuller reading changes recall by at most 0.04.The best SubQ reading policy still trails the retriever, 0.50 versus 0.55.Source: Table 4 · p. 8, §4.4 · p. 9
  • Single-query expansion and multi-query generation yield no meaningful improvement, while both hypothetical-abstract variants substantially reduce recall.The authors interpret this as evidence that rephrasing does not recover inspiration papers that the embedding space does not place near the query.Source: Table 5 · p. 8, §4.4 · p. 9

5 CONCLUSION

SCHOLARCATALYST benchmarks retrieval of prior papers whose ideas could advance research, using researchers’ firsthand accounts of their projects. Current systems recover only a fraction of credited papers, while the authors identify broad-field expert intuition as a future milestone.

5 CONCLUSION

  • Current systems recover only a fraction of author-credited papers, and agentic search does no better than embedding retrieval largely because limited candidate coverage keeps useful papers out of reach.The authors frame developing expert-level retrieval as a milestone, including retrieval across fields no single researcher can follow.Source: §5 · pp. 9–10
  • SCHOLARCATALYST covers computer science papers from 2025–2026 with uneven representation across research areas, and its retrospective author labels are subject to hindsight.The authors also identify the task’s performance ceiling as an open question; a preliminary three-author comparison on one paper recovered 43–60% of first-author labels at 84–88% precision.Source: §5 · p. 9
  • Nearly half of credited papers (49.5%) come from outside the project’s subject area, and 43.6% of subfield-specific positives were never cited in the source paper.The authors report 190k papers in their corpus, compared with over 225M indexed by Semantic Scholar.Source: §5 · p. 10

A BENCHMARK CONTRIBUTORS

The appendix acknowledges researcher contributors and describes the benchmark instance and dataset composition. The benchmark pairs research questions with author-validated inspiration papers and uses a temporally restricted corpus.

A BENCHMARK CONTRIBUTORS

  • The authors thank researchers who reviewed reconstructions of their work, revised questions and inspiration papers, and explained which prior work shaped their research.Contributors who agreed to be acknowledged are listed alphabetically.Source: A BENCHMARK CONTRIBUTORS · p. 17

B.1 BENCHMARK INSTANCE

  • Each SCHOLARCATALYST instance anchors a completed source paper’s central question and distinct research directions to author-validated positives, hard negatives, and a temporally restricted corpus.The benchmark contains 207 instances and 894 queries: 207 core research queries and 687 subfield-specific queries.Source: §B.1 · p. 18
  • Core queries can surface ideas organizing a project as a whole, while subfield-specific queries target particular methodological or empirical directions.Their overlap and divergence represent complementary forms of inspiration within one project.Source: §B.1 · p. 18

B.2 DATASET STATISTICS

  • The author-validated set contains 207 source papers and 894 queries, concentrated in machine learning and adjacent AI fields.The queries comprise 207 central and 687 directional questions.Source: §B.2 · p. 18
  • The corpus contains 190,896 papers, including resolved source-paper references and additional arXiv papers from five main computer science domains published between 2020 and 2024.Although some resolved references fall outside that collection window, each query searches only papers published no later than its source paper.Source: §B.2 · pp. 18–19

B.3 QUALITATIVE ANALYSIS OF INSPIRATION RELATIONS

The qualitative analysis finds that inspiration is diverse and poorly captured by topical similarity or citation status. Authors’ rationales also show that a paper’s role depends on the target project.

Diversity of Inspiration Relations

  • Authors most often described adapted techniques, supporting empirical findings, ideas generalized to new settings, or limitations motivating new approaches.A taxonomy labels key-inspiration rationales, with multiple types assigned to 55.8% of them.Source: §B.3 · p. 19
  • Hard negatives were at least as similar to queries as positives on both lexical and semantic similarity measures.The analysis measures lexical similarity using shared query words in titles and abstracts, and semantic similarity using embedding cosine similarity.Source: §B.3 · p. 19

Citation Status Is an Imperfect Proxy for Inspiration

  • Citation status and author-identified inspiration diverge: 43.6% of SubQ positives are uncited, and 67.8% of those share no references with the source paper.Among cited positives, only 46.2% of CoreQ and 34.0% of SubQ positives have a uses-or-extends label.Source: §B.3 · p. 19

Diversity of Inspiration Relations

  • Rationales received 1.69 taxonomy labels on average; 55.8% received at least two, showing that inspiration often operates through multiple mechanisms.The nine-type taxonomy permits multiple labels per rationale.Source: §B.3 · p. 19

Context-Dependent Roles of the Same Source Paper

  • The same prior paper can serve different roles across projects, including a transferable method for one and a motivation for another.Its usefulness depends on the question the project is trying to answer.Source: §B.3 · p. 20

C EXPERIMENTAL SETUP

The experiments evaluate retrieval from temporally filtered literature and compare embedding systems, agents, and LLM rerankers. Reasoning-optimized retrievers do not outperform the strongest general-purpose embeddings, while reranking results vary with backbone capability.

C.2 EVALUATION SETUP FOR PRE-DISCOVERY RETRIEVAL

  • The evaluation restricts each query to papers published no later than its source paper, disallows web search, and retains a median searchable corpus of 190,574 papers.Agents search temporally filtered titles and abstracts; the shared corpus-search tool returns the retriever’s top ten papers.Source: §C.2 · p. 20, §C.3 · p. 20

C.3 AGENTIC SEARCH BASELINES

  • Agent rankings are scored on depth-100 lists that add up to 25 seen-but-unranked papers before retriever back-fill.R@5 and R@20 usually reflect the agent’s own ranking unless it returned fewer than k papers; trajectory recall measures gold papers among those seen.Source: §C.3 · p. 20
  • The tool-calling agent searches the query, pools up to 60 papers over five rounds, then ranks them by how much their ideas could help pursue the query.The deep research agent splits queries into up to six sub-questions, searches and reads papers, then synthesizes a ranking from a pool of up to 150.Source: §C.3 · pp. 20–21
  • Claude Fable 5.1’s June 2026 cutoff follows all but six source papers, so its rows fall outside the main knowledge-cutoff rule.Safety classifiers sometimes decline calls; declines occur on 47 tool-calling, 95 deep-research, and 33 grep-agent queries.Source: §C.3 · p. 21

D ADDITIONAL ANALYSES

  • The appendix’s additional analyses cover embedding retrieval, reranking, knowledge cutoffs, query generation, full-text reading, and agent search failures.The extended embedding comparison evaluates scientific and reasoning-intensive retrievers alongside general-purpose systems.Source: D ADDITIONAL ANALYSES · p. 22, §D.1 · p. 22

D.1 EXTENDED RESULTS FOR EMBEDDING RETRIEVERS

  • None of the evaluated reasoning-optimized embedding models improves over the strongest general-purpose embeddings.The authors suggest current reasoning-intensive retrieval objectives may not capture relationships that make prior papers useful for advancing a project.Source: §D.1 · p. 22

D.2 LLM RERANKING

  • The reranking study tests listwise ranking of 20 candidates and tournament ranking in batches of 20, promoting the top four per batch.It evaluates candidate pools of 20, 40, 80, and 160; Figure 10 compares search and memory variants on 46 CoreQ queries, with Gemini-Emb-2 at 0.38.Source: §D.2 · p. 22, Figure 10 · p. 23
  • Reranking generally improves with backbone reasoning capability; Claude Fable 5.1 achieves the strongest results, while weaker models can underperform the retriever on SubQ queries.Figure 8 reports Recall@5 across candidate pool sizes, with standard and oracle reranking curves and retriever-only performance marked.Source: §D.2 · p. 22, Figure 8 · p. 22

D.3 POST-CUTOFF BACKBONES AND BENCHMARK LEAKAGE

The cutoff-split analysis finds no sign that stronger backbones benefit from having seen source papers. Source abstracts help, but web search adds little and bibliographies nearly reveal the gold papers.

D.3 POST-CUTOFF BACKBONES AND BENCHMARK LEAKAGE

  • The pre-cutoff slice is harder for every system: core research query retriever R@100 is 0.49 versus 0.63 on the post-cutoff slice.The analysis compares each system with the retriever within the same slice; the setup varies source-paper, web, and bibliography access for Gemini 3.7 Flash.Source: §D.3 · pp. 23–24
  • The cutoff split shows no sign that stronger backbones gain from having seen the source papers.GPT-5.6 Sol’s margin over the retriever is 0.07 on the pre-cutoff slice and 0.06 on the post-cutoff slice; Claude Fable 5.1 cannot be tested this way because its cutoff falls inside the post-cutoff slice.Source: §D.3 · p. 24
  • With GPT-5.6 Sol and Claude Fable 5.1, the tool-calling agent exceeds the retriever at core research query R@20 on both slices; other backbones stay close.At the subfield-specific query level, no agent or closed-book row exceeds the retriever on either slice.Source: §D.3 · p. 24
  • With the same source context, the agent’s advantage over memory shrinks from 0.25 for Luna at no reasoning effort to within noise for stronger runs.Raising Luna’s reasoning effort helps memory and hurts the query-only agent, while a stronger tier helps both.Source: §D.3 · p. 24
  • With Gemini 3.7 Flash, the source title and abstract raise core research query R@20 by 0.11 on both slices.Web search leaves recall almost unchanged, while the bibliography alone contains nearly all gold papers, with R@100 above 0.9.Source: §D.3 · p. 24

D.4 QUERY GENERATION METHODS

The section tests query reformulation and reading beyond abstracts as ways to improve agentic retrieval. It describes several generated-query strategies and finds that full-text reading adds little in the tested setting.

D.4 QUERY GENERATION METHODS

  • Single-query expansion appends a short model-generated continuation to the research question and uses the result as the retrieval query.The methods test whether LLM rewriting brings retrieval closer to gold papers that may use different vocabulary from the question.Source: §D.4 · p. 24
  • Multi-query generation produces up to five augmented questions, retrieves separately with each, then fuses the top-1,000 rankings using reciprocal rank fusion.The fusion uses k=60.Source: §D.4 · p. 24
  • HyDE generates five hypothetical abstracts of prior papers and uses each abstract as a retrieval query, fusing the resulting rankings.The user message prefixes the question with “Research question:”.Source: §D.4 · p. 24
  • For one CoreQ instance, dense retrieval on the original question recovers a gold paper that the generated queries miss from their top 100.Tables 23 and 24 show the question alongside single- and multi-query generations and three HyDE abstracts.Source: §D.4 · p. 24

D.5 READING FULL PAPERS VERSUS ABSTRACTS

  • Reading corpus papers beyond their abstracts adds little to GPT-4.1 tool-calling agent Recall@20 across six read policies and abstract-only reading.The comparison uses 50 queries of each type from the full-text subset; the top bar is the retriever alone and the second is the agent without a read tool.Source: Figure 11 · p. 25
  • Because retrievers handle long documents poorly and paper bodies exceed their training input length, the experiments let agents read found papers instead of indexing full text.Reads share a five-round budget with searches; each read is capped at 12,000 characters, with a separate 6,000-character cap for an included bibliography.Source: §D.5 · p. 25

D.6 ANALYSIS OF AGENT SEARCH FAILURES AND RATIONALE EXPLANATIONS

Agentic search rarely finds gold papers beyond the retriever’s pool and often drops papers it does find; recognized papers can still be explained in ways matching authors’ rationales. The section also describes the benchmark construction pipeline, from source-paper processing through candidate retrieval.

D.6 ANALYSIS OF AGENT SEARCH FAILURES AND RATIONALE EXPLANATIONS

  • The retriever’s top 100 covers 70% of gold papers, and agents recover almost none of the rest; follow-up searches add few and can drift away from gold papers.About half of gold papers are returned by no search during an agent run, and the first search accounts for about two-thirds of the gold papers an agent sees; a case shows query drift to generic transfer learning.Source: §D.6 · p. 25, Table 29 · p. 27
  • Among gold papers in the retriever’s top 100, the tool-calling and deep research agents keep 57% and 48%, respectively, in their top 20.In a traced case, the deep research agent retrieves two gold papers but lists neither in its final ranking.Source: §D.6 · p. 25, Table 27 · p. 27
  • For gold papers ranked highly by the tool-calling agent, 76% of 219 comparable model explanations describe the same relationship as the author rationale.Almost all other explanations describe a different but valid relationship; because the same model writes and judges, agreement is an upper estimate and does not show why a paper entered the ranking.Source: §D.6 · p. 26

D.7 COST AND SEARCH EFFORT OF THE AGENTS

  • With GPT-4.1 and o3, grep and deep research agents make 14 to 20 tool calls per query and cost about eight times as much as the tool-calling agent.With Claude Fable 5.1, the tool-calling agent makes more than twice as many searches within five rounds, and every agent costs more than a dollar per query.Source: §D.7 · p. 26

E AUTOMATED DATA CONSTRUCTION PIPELINE

  • The construction pipeline turns raw arXiv papers into benchmark instances through source-paper collection, annotation reconstruction, time-cutoff corpus assembly, and candidate retrieval and validation.The pipeline starts from recent computer science papers and includes citation links in structured paper representations.Source: E AUTOMATED DATA CONSTRUCTION PIPELINE · p. 26

E.1 SOURCE-PAPER PROCESSING

  • Source processing matches recent conference papers to arXiv preprints, converts papers to structured full text, filters incomplete conversions, and retains sources with at least 70% of references resolved.Resolved references are matched to canonical scholarly records using arXiv, Semantic Scholar, and OpenAlex.Source: §E.1 · pp. 26, 28

E.2 PRE-ANNOTATION GENERATION

  • Gemini 3.1 Pro reads each structured source paper and bibliography to reconstruct its CoreQ, SubQs, inspiration papers, and rationales.These reconstructed annotations form the benchmark instances used in evaluation.Source: §E.2 · p. 28

E.3 CANDIDATE RETRIEVAL

  • Candidate retrieval pools sparse and dense results, reranks candidates for scientific relevance, and sends the top 10 for author review as positives or hard negatives.The 190,896-paper corpus combines resolved references with 181K additional arXiv papers from 2020–2024, and applies a per-query publication-date cutoff.Source: §E.3 · p. 28

E.4 AUTHOR VALIDATION OF THE AUTOMATED PIPELINE

Authors validated reconstructed research questions with high accuracy and judged many pooled, non-cited candidates to be genuine inspirations. The pipeline was also applied to additional papers to create a model-reconstructed set.

E.4 AUTHOR VALIDATION OF THE AUTOMATED PIPELINE

  • Authors rated reconstructed CoreQs accurate or mostly accurate for 98.1% of 207 instances, and reconstructed SubQs accurate or mostly accurate for 95.9% of 687 subfields.Fully accurate ratings were 60.9% for CoreQs and 64.9% for SubQs.Source: §E.4 · p. 28
  • Authors retained 85.1% of originally cited candidates as positives and judged 35.1% of pooled non-cited candidates to be genuine inspirations.The non-cited papers were absent from the source paper’s bibliography.Source: §E.4 · p. 29
  • Applying the pipeline to 250 additional papers yielded 250 central queries and 863 directional queries.For CoreQs, the pipeline nominates cited papers as likely key inspirations for comparison with author-labeled positives.Source: §E.4 · p. 29

F AUTHOR DATA COLLECTION

This section describes the collection of author-written research questions, inspiration judgments, and rationales, alongside the documented prompts used for retrieval agents and pre-annotation. Those prompts organize inspiration threads into scientific directions, identify essential cited work, and rank retrieved candidates by scientific contribution rather than topical similarity.

F.1 AUTHOR RECRUITMENT

  • 184 authors contributed 207 instances by annotating their own projects with pre-solution research questions, inspiration labels, hard negatives, and rationales.Authors were recruited from highlighted papers at recent major AI conferences and through public calls to eligible authors.Source: §F.2 · p. 29, §F.1 · p. 29

G.1 AGENT PROMPTS

  • Tool-calling, deep-research, and grep-agent prompts direct searches toward papers whose ideas could help pursue a question, rather than papers that are merely topically related.The prompts set distinct search budgets and workflows, including up to five tool-call rounds, up to 20 searches and 10 abstract reads, or up to 20 shell commands with abstract review.Source: §G.1 · pp. 29–31

G.2 RERANKING AND QUERY GENERATION PROMPTS

  • Reranking and query-generation prompts ask models to rank papers by scientific usefulness and expand questions through specific, diverse search angles.HyDE prompts request five hypothetical abstracts from distinct angles, including adjacent fields, without naming real papers or solving the research question.Source: §G.2 · pp. 31–32

G.3 PRE-ANNOTATION CONSTRUCTION PROMPTS

  • Pre-annotation prompts reconstruct the authors’ pre-solution research question and select cited papers whose contributions most directly inspired the target paper’s central contribution.They require the question to avoid revealing the eventual solution and require rationales to explain the scientific connection and why a selected work is a primary inspiration.Source: §G.3 · pp. 32–33
  • The author interface has four steps: revise the research question, select key prior papers, revise research directions, and judge supporting papers as positive or negative with justification.Authors can also rate the reconstructed questions and mark other cited papers as equal, lesser, or irrelevant influences.Source: Figure 14 · p. 50, Figure 15 · p. 51, Figure 16 · p. 52
  • The prompt identifies 3–5 distinct inspiration threads, each a coherent line of prior work defined at a comparable level of abstraction.It cautions against splitting one line into method variants or merging distinct lines merely because they share a broader research area.Source: §G.3 · p. 54
  • Each thread description explains its shared scientific insight and role in the paper, while its natural-language retrieval query frames one specific pre-solution research question.The query should avoid solution-specific concepts and target prior work that substantially contributed from that perspective.Source: §G.3 · pp. 54–55
  • For each thread, the prompt selects at most five cited papers whose contributions were essential, excluding papers cited only for background, comparison, datasets, benchmarks, or implementation.The rationale explains how the selected work’s shared contribution influenced the target paper and why it mattered more than related cited work.Source: §G.3 · p. 56
  • Candidate reranking prioritizes papers whose primary scientific contributions satisfy the inferred criterion, not those that are merely topically or methodologically similar.The criterion is derived from a reference scientific contribution.Source: §G.3 · p. 57

Related papers

Loading 2610.02202v1…