PaperlayerView exact revision on arXiv

Source-linked AI summary

MUSES: A Benchmark for Prospective Intellectual-Roots Retrieval

Rohan Pandey, Sunjae Kwon, Hong Yu

arXiv:2609.00313v1cs.IR

TL;DR

Existing retrieval systems are not designed to forecast the less familiar prior works that shape an author’s next paper, making prospective intellectual-roots retrieval an open benchmark problem. MUSES builds a million-instance, fixed-corpus benchmark with familiarity tiers and CiteRoots’ rhetorical and author-endorsed layers. Across progressively tighter targets, the strongest practical retriever’s hit@100 falls from 0.534 on CiteNext to 0.171 on author-endorsed CiteNew.

  • Problem

    Existing retrieval systems optimize relevance, popularity, and topical similarity, while prospective retrieval asks which prior works an author will engage with next.

  • Method

    MUSES combines time-safe retrieval over a fixed 2.33 M-paper corpus with familiarity tiers and CiteRoots’ rhetorical and author-endorsed root layers.

  • Results

    Across progressively tighter targets, MC-SPECTER2 hit@100 falls from 0.534 on CiteNext to 0.171 on author-endorsed CiteNew.

  • Takeaways & Limitations

    MUSES provides a shared evaluation suite for prospective retrieval and distinguishes local rhetorical roots from author-confirmed paper-level roots.

  • Takeaways & Limitations

    The benchmark uses a fixed 2.33 M-paper S2ORC slice, so open-corpus retrieval remains future work.

Abstract

from arXiv · show

Scientific discovery depends on finding prior literature that shapes what comes next. Existing retrieval systems optimize for relevance and popularity, often favoring central papers over less familiar works that later prove generative. We introduce \textbf{MUSES}, a million-instance benchmark for prospective intellectual-roots retrieval over a fixed 2.33M-paper corpus, with roughly 140K test instances per familiarity tier. To our knowledge, it is the first prospective benchmark at this scale with a shared retrieval task and author-confirmed paper-level root labels. Alongside it, \textbf{CiteRoots} pairs a scalable rhetorical layer over local citation text (LLM judge $κ= 0.896$ versus human gold) with a paper-level author-endorsed layer ($n = 1{,}518$ generative-inspiration pairs from 753 focal papers). MUSES organizes difficulty along two axes: a \emph{familiarity} axis spanning CiteNext, CiteNew, and CiteNew-Isolated, and a \emph{functional} axis spanning broad citations, rhetorical roots, and author-endorsed roots. Across 9 method classes, a lean multi-centroid retriever built on SPECTER2 is strongest. Hit@100 falls from 0.534 on CiteNext to 0.424 on CiteNew, 0.205 on rhetorical CiteNew, and 0.171 on author-endorsed CiteNew, a $3.1\times$ decline. In a registered eight-lens full-test audit, roughly half of broad-tier test instances remain unsolved at K=1{,}000. Rhetorical role and author endorsement are distinct: the same judge agrees with endorsement at $κ= 0.037$. We release MUSES, both CiteRoots layers, and a distilled open companion judge for future work on prospective retrieval and intellectual roots.

1 Introduction

MUSES frames literature discovery as prospective retrieval: forecasting which prior works an author will cite next rather than retrieving known relevant papers. It combines familiarity- and function-based difficulty axes with broad citation and intellectual-root targets.

  • Motivation: Prospective retrieval asks systems to forecast which prior works an active researcher will engage with in the next contribution from millions of pre-existing candidates.This differs empirically from retrospective relevance lookup.
  • Benchmark: MUSES is a million-instance benchmark over a fixed 2.33 M-paper pool, with approximately 140 K test instances in each familiarity tier.The tiers are CiteNext, CiteNew, and CiteNew-Isolated.
  • Root layers: CiteRoots adds rhetorical and author-endorsed root layers, measuring local citation roles and paper-level author-confirmed generative inspiration.The author-endorsed layer contains 1,518 pairs from 753 focal papers, while the rhetorical judge reaches κ = 0.896 against human gold.
  • Results: 0.534 to 0.171: MC-SPECTER2 hit@100 declines across CiteNext, CiteNew, rhetorical CiteNew, and author-endorsed CiteNew.The endorsed point uses the n=257 CiteNew sub-cohort to match the rhetorical slice.
  • Results: 47.8–50.0 % of broad-tier test instances remain unsolved at K=1000 in the registered eight-lens full-test audit.The benchmark therefore exposes substantial residual retrieval difficulty.

2 Related Work

MUSES extends prior literature-discovery, scientific-retrieval, and citation-intent work by making prospective, author-conditioned intellectual-root retrieval its shared evaluation target. Its author-endorsed layer adds paper-level author confirmation beyond local citation rhetoric.

  • Literature-based discovery: Earlier literature-based discovery and citation prediction work generally remained post-hoc rather than author-conditioned and prospective.Those lines include graph-mining, embedding-based discovery, citation prediction, and recommendation.
  • Scientific retrieval: MUSES differs from BEIR, SciFact, and SciDocs primarily in target: it evaluates prospective citation retrieval rather than topical retrieval, claim verification, or representation quality.The distinction is not merely disciplinary domain.
  • Prospective benchmarks: Recent forward-looking scientific benchmarks address hypothesis discovery, dataset analysis, tool use, research extension, creativity, and procedure generation, while MUSES focuses on prospective retrieval.Related ideation systems and retrieval-augmented research agents likewise target adjacent tasks.
  • Citation intent: MUSES reconstructs citation-intent ideas into a binary ROOT/non-ROOT target so labels define tighter retrieval slices rather than a standalone retrospective classification benchmark.The operational distinction separates generative roots from non-ROOT citation roles.
  • Novelty: The author-endorsed layer asks authors which prior works shaped a focal paper, moving beyond local citation-function inference toward paper-level intellectual roots.This author-confirmed layer yields a benchmark structure not exposed by prior evaluation suites.

3 Benchmark Construction

MUSES constructs a time-safe, fixed-corpus, author-conditioned next-paper retrieval benchmark and adds complementary rhetorical and author-endorsed root labels. Difficulty tightens first by familiarity and then by functional role, leaving progressively fewer targets.

  • Benchmark instantiation: MUSES uses a fixed 2.33 M-paper released pool and 1.04 M author-conditioned instances for time-safe, novelty-stratified retrieval under author-disjoint splits.The primary unit is an author–focal-paper pair, with candidates available before the relevant cutoff.
  • Retrieval task: The ranked output is evaluated against the bibliography of the author’s first subsequent eligible paper after time t.Candidate papers are restricted by a time-safe cutoff.
  • Familiarity tiers: CiteNext contains all next-paper references, CiteNew removes references previously cited by the author, and CiteNew-Isolated also removes targets explainable through coauthor exposure.These tiers progressively reduce prior familiarity.
  • Functional layers: CiteRoots separates local citation role from author-identified paper-level influence, using rhetorical-root and author-endorsed layers to tighten the target beyond broad citation retrieval.The rhetorical layer measures what looks generative in local text, while endorsement measures author-identified paper-shaping influence.
  • Rhetorical roots: The rhetorical layer retains six citation-role categories, collapses them to ROOT/non-ROOT for retrieval, and emits a paper-level positive when at least one linked context is judged ROOT.The canonical judge reaches κ = 0.896 against approximately 1.2 K human-gold annotations.
  • Problem shape: 5,702 and 4,483 focal-paper instances form the rhetorical-root slices, with 6,785 and 5,241 positive edges, respectively.These slices are substantially sparser than the broad benchmark universes.

4 Experiments

Experiments compare nine retrieval classes across familiarity and root-function targets. MC-SPECTER2 is strongest, but tighter targets sharply reduce performance, compress method gaps, and leave a substantial unsolved tail.

  • Broad benchmark: MC-SPECTER2 is the strongest broad-tier method, reaching hit@100 of 0.534 on CiteNext, 0.424 on CiteNew, and 0.366 on CiteNew-Isolated.The ordering remains stable across all three familiarity tiers, with citation-pretrained retrieval ahead of lexical and generic-dense methods.
  • Broad benchmark: 47.8–50.0% of broad-tier instances remain unsolved at K = 1,000, while 62.2–67.5% remain unsolved at K = 100.The core ordering also persists across Biology, Computer Science, Medicine, Engineering, and Chemistry.
  • Root tightening: 0.205 versus 0.424 hit@100: rhetorical tightening produces a 52% relative decline at fixed familiarity.The candidate pool and benchmark instances remain unchanged, so the stricter target increases retrieval difficulty rather than changing the evaluation setup.
  • Root tightening: MC-SPECTER2’s lead over single-centroid SPECTER2 narrows from +0.077 hit@100 on broad CiteNew to +0.045 on rhetorical CiteNew and +0.023 on author-endorsed CiteNew.On the endorsed CiteNew sub-cohort, the corresponding scores are 0.171 versus 0.148.
  • Author endorsement: 0.171 hit@100: the author-endorsed CiteNew sub-cohort shows a further 17% relative decline below rhetorical CiteNew.The endorsed layer is reported as a small calibration slice, with n = 257 for the harder CiteNew sub-cohort and n = 402 for the full evaluable cohort.
  • Author endorsement: κ = 0.896 versus human gold but κ = 0.037 against author endorsement: rhetorical classification and paper-level endorsement measure distinct constructs.Paper-level prompting raises agreement only to κ = 0.160, with harder novelty-controlled cases remaining largely unresolved.

5 Discussion, Limitations, and Future Work

MUSES identifies a simple multi-centroid retriever as strongest while exposing substantial unsolved retrieval difficulty. Its fixed-pool design, citation proxy, and small author-endorsed cohort bound the claims, motivating broader root signals and larger endorsement data.

  • What the benchmark shows: MC-SPECTER2 is strongest across tested methods, using citation-pretrained multi-centroid retrieval without fine-tuning, reranking, or an LLM call.Roughly half of broad-tier instances remain unsolved at K = 1,000, indicating a substantial residual ceiling.
  • Limitations: MUSES uses a fixed 2.33 M-paper S2ORC slice, so open-corpus retrieval remains future work despite cleaner comparisons.The fixed pool preserves time-safety and isolates target difficulty from changing candidate universes.
  • Limitations: Citations proxy intellectual influence, while seminars, conversations, failed experiments, and uncited reading remain unobserved.The residual retrieval ceiling may reflect both genuine discovery difficulty and limits of title-and-abstract retrieval.
  • Limitations: The author-endorsed cohort is small and response-conditioned, so it is a selective calibration slice rather than a representative census of intellectual roots.The paper therefore bounds interpretation of endorsement results to this high-signal cohort.
  • Future work: Future work should scale author endorsement and combine citation trajectories with local-text and paper-level signals.The stated goal is retrieval of prior works likely to shape a researcher’s next contribution.

A.2 Retrieval Benchmark Construction

The retrieval benchmark constructs time-safe author-conditioned instances from S2ORC, freezes a shared candidate universe, and defines rhetorical targets from local citation contexts. Its taxonomy separates generative roots from non-generative citation uses.

  • Retrieval benchmark construction: The benchmark pipeline filters S2ORC papers, builds role-consistent author trajectories, validates next-paper targets, and restricts candidates to papers available by time t.The released shared pool is formed from the time-safe, title-and-abstract-ready substrate.
  • Retrieval benchmark construction: 687,624 train, 182,543 validation, and 168,613 test instances are released.The split counts characterize the benchmark after release construction and filtering.
  • Rhetorical layer: The rhetorical layer classifies citation contexts into ROOT roles—TF, ME, and GM—and WEED roles—CC, TR, and BC.ROOT denotes generative intellectual influence; WEED covers non-generative citation use.
  • Judge taxonomy: The judge uses only the local citation window and assigns exactly one of six operational categories.The categories distinguish theoretical foundation, method extension, generative motivation, contrast, tool/resource, and background context.
  • Rhetorical layer: Paper-level rhetorical positives use a precision-first k≥1 rule: one linked context judged ROOT is sufficient.Stricter k≥2 mirrors are released, but the paper evaluates k≥1 throughout.

B.5 LLM Judge, Distillation, and Agreement

CiteRoots uses a frontier judge to define the rhetorical-label contract and a distilled Qwen3-8B student for scalable inference, with human annotation as the validity reference. The evaluation measures agreement across these labeling stages.

  • Judge and distillation: The rhetorical layer uses a two-stage setup: a frontier judge defines the label contract, and a distilled Qwen3-8B model performs lower-cost inference.Human annotation serves as the validity reference for both models.
  • Agreement evaluation: 1,900 citation contexts were audited by humans, including 200 independently labeled by three raters for inter-rater agreement.The audit was stratified to cover taxonomy decision boundaries rather than only easy background cases.
  • Judge and distillation: The student was trained on 104,976 judge-labeled citation contexts from seven 100K-scale silver shards.The configuration uses Qwen3-8B sequence classification with LoRA adapters and a trainable classification head.
  • Agreement evaluation: The agreement and validation summary is reported in Table 8.The table consolidates the main agreement measurements for the rhetorical layer.

C.2 Retrospective collection, prospective evaluation

Author endorsements are collected retrospectively but mapped back to the same prospective retrieval unit, preserving the pre-t pool while tightening only the gold targets. The resulting cohort is filtered, reviewed, and substantially smaller than the broad benchmark.

  • Collection and evaluation: Author-endorsed labels can only be collected after the focal paper has been written.The retrospective attestation is later resolved to the prospective retrieval-instance structure.
  • Prospective evaluation: The model receives only pre-t history H(a, t), while the focal text, bibliography, and author response remain outside the model input.The retrospective collection supplies gold targets but does not enter inference.
  • Prospective evaluation: The fixed 2.33 M-paper time-safe pool is reused, and only the target set tightens to author-confirmed roots.Endorsed targets outside the pool are tracked separately and excluded from retrieval evaluation.
  • Collection and resolution: Responses undergo automatic parsing, human review of ambiguous cases, and final corpus-identifier resolution; non-endorsed items remain unlabeled.Every released benchmark-facing label is confirmed in human review.
  • Collection and evaluation: Only 34 of 435 context-linked author-endorsed pairs are judged ROOT, with κ = 0.037 for the LLM judge and κ = 0.002 for the student.These figures support the paper’s separability claim between local rhetoric and author endorsement.

D.1 Full Metrics and Method Registry

The appendix provides released materials for reproducing the main experiments and clarifies the metrics reported for rhetorical and endorsed cohorts.

  • Reproducibility: Released GitHub and Hugging Face materials include benchmark artifacts, scoring code, run metadata, and reproduction instructions.They also document source URLs and governing licenses or terms for external datasets, models, and code assets.
  • Metrics: Tables 13 and 14 report hit@10, hit@100, hit@1000, and MRR for rhetorical and author-endorsed cohorts.
  • Uncertainty: Bootstrap confidence intervals for broad-tier runs are provided in the released materials.

D.2 Method Training and Inference Recipes

The method registry compares graph, lexical, dense, sequence, reranking, and fusion approaches under a common time-safe evaluation protocol, with task-specific training varying by method.

  • Priors and graph baselines: Popularity, co-citation, and coauthor 2-hop baselines rank candidates using citation or collaboration structure within a time-safe substrate.
  • Lexical and generic dense retrieval: BM25 ranks candidates from title-and-abstract bag-of-terms queries, while generic dense encoders use normalized-embedding dot products without fine-tuning.The dense methods use BGE-large-en-v1.5 and E5-large-v2 with recommended retrieval prefixes.
  • Citation-pretrained dense retrieval: SPECTER2 variants share an off-the-shelf citation-pretrained encoder but differ in how prior papers are summarized into query vectors.
  • Encoder fine-tuning: BGE-large fine-tuning uses InfoNCE over train-split positive pairs and in-batch negatives, then re-encodes the candidate pool.Training uses approximately 1M positive pairs and takes approximately 36 GPU-hours on one A100-80GB.
  • Sequence and fusion methods: Sequence trajectory encoding uses a SASRec-style transformer over publication-ordered history, while reciprocal-rank fusion combines its ranked list with MC-SPECTER2.The fusion uses the standard k = 60 smoothing constant and no learned parameters.
  • Reranking: The cross-encoder reranks the MC-SPECTER2 top-1000 shortlist, whereas LLM listwise reranking paginates that shortlist into 20-candidate pages without fine-tuning.The cross-encoder is trained with pointwise binary cross-entropy; the LLM merges page-level rankings by averaged within-page rank.
  • Evaluation protocol: All methods use the same released MUSES pool, time-safe candidate restriction, author-disjoint splits, and hit@K/MRR scoring procedure.Run metadata records the retriever configuration, encoder checkpoint hash, and candidate-pool snapshot.
  • Registry coverage: The rhetorical registry covers 21 methods, but the author-endorsed registry evaluates only four dense-history methods on its retrieval-evaluable cohort.The released scoring code allows the endorsed registry to be extended beyond the small cohort.

D.3 K-Saturation for Multi-Centroid Retrieval

Multi-centroid SPECTER2 retrieval largely saturates by K=8, supporting K=16 as the paper’s main operating point while shortlist coverage bounds hit@1000.

  • Saturation: At K=8, hit@100 is within 1.4% relative of K=16, while K=64 adds at most 0.3 absolute points on any metric.
  • Harder familiarity tiers: On CiteNew, K=8/16/64 hit@100 is 0.418/0.424/0.426, and on CiteNew-Isolated it is 0.360/0.366/0.368.
  • Operating point: The endorsed cohort uses a K=8/16/24 grid, bracketing the saturation point and matching the broad-tier ordering K=1 < K=8 < K=16 ≥ K=24.
  • Coverage boundary: Hit@1000 is bounded by whether the true target appears in the MC-SPECTER2 top-1000 shortlist.

D.4 Endorsement Recovery Analyses

The endorsement-recovery analyses show that local rhetorical aggregation is insufficient, paper-level prompting helps only partially, and recovery is easier for habitual than CiteNew endorsements.

  • Passage-level analysis: Passage-level aggregation combines rhetorical proportions, maximum ROOT probability, and cite-passage counts to recover author-endorsed roots.
  • Paper-level prompting: Paper-level prompting reads focal and candidate metadata plus linked cite passages, then returns a yes/maybe/no judgment with justification.
  • Recovery performance: Passage-level aggregation is near chance after class-imbalance correction, while paper-level prompting improves substantially but remains only partially aligned with author confirmation.
  • Endorsement strata: Habitual endorsements are much easier to recover automatically than CiteNew endorsements, even with full cite-passage evidence.The discovery frontier is specifically the CiteNew-Endorsement sub-slice.
Loading 2609.00313v1…