PaperlayerView exact revision on arXiv

Source-linked AI summary

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

Maayan Sharon, Tom Hope

arXiv:2608.27394v1cs.CLcs.IR

TL;DR

Scientific literature can inspire problem-solving through approaches, broader formulations, or concrete instantiations, but conventional retrieval benchmarks do not distinguish these roles. RATIO constructs and evaluates a full-text benchmark for three typed ideation operations using discourse-marker supervision and human–LLM validation. Operation-specific fine-tuning improves retrieval, while absolute performance remains limited and valid alternatives are often missed by single-positive evaluation.

  • Problem

    Conventional scholarly retrieval benchmarks assess topical or query-answering relevance but do not distinguish passages serving different ideation roles.

  • Method

    RATIO uses discourse markers mined from full-text scientific papers to construct retrieval data for ADDRESS, BROADEN, and SPECIFY, with human and LLM validation.

  • Results

    Move-specific fine-tuning substantially improves retrievers across all three operations, while tuned models frequently retrieve valid inspirations beyond the mined positive.

  • Takeaways & Limitations

    RATIO provides a scalable framework for training and evaluating retrieval components that support literature-grounded scientific ideation.

  • Takeaways & Limitations

    Absolute performance remains low, and single-positive metrics provide only a lower bound because valid alternative inspirations may be labeled as negatives.

Abstract

from arXiv · show

Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction - zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision - previously used only for classification - to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers but leaves much room for further improvements. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues on scientific inspiration retrieval.

1 Introduction

RATIO frames literature retrieval as three distinct ideation moves: finding approaches, broader formulations, or concrete instantiations. It introduces a scalable, vetted benchmark because generic topical relevance does not distinguish these roles.

  • Researchers may need inspirations that propose methods, express broader principles, or instantiate principles concretely in new settings.
  • Retrieval helps determine which prior mechanisms, abstractions, and problem formulations are available to human and AI ideation systems.Retrieved stimuli at different abstraction levels have been reported to boost ideation.
  • RATIO defines inspiration retrieval around ADDRESS, BROADEN, and SPECIFY operations rather than topical relevance alone.ADDRESS finds approaches or insights for problems; BROADEN moves toward greater generality; SPECIFY provides concrete cases, mechanisms, or examples.
  • RATIO addresses the gap between conventional benchmarks for topical or query-answering relevance and retrieval conditioned on an ideation role.
  • The benchmark scales relation supervision by mining high-precision discourse markers from full-text papers, then applying expert review, LLM validation, and human-calibrated evaluation.The pipeline extends marker-based distant supervision from sentence-pair classification to retrieval.
  • Operation-specific fine-tuning substantially improves retrieval for all three moves, although absolute performance remains limited.

2 Related Work

RATIO extends scientific retrieval beyond topical and informational relevance by evaluating typed ideation relations over full-text sentence corpora. Its discourse-marker supervision adapts prior relation-learning ideas to large-scale retrieval.

  • Unlike literature-grounded generation studies, RATIO isolates the retrieval capability that determines which scientific inspirations become available.
  • RATIO extends methodology-inspiration retrieval from primarily abstract-level, paper-level supervision to full-text, sentence-level retrieval with three typed operations.
  • RATIO evaluates whether a candidate serves an ideation function—addressing a problem, broadening its formulation, or supplying a concrete instantiation.
  • Prior scientific NLI resources classify given sentence pairs with entailment-oriented labels rather than retrieving relation-appropriate candidates from a large corpus.
  • The benchmark construction combines discourse-marker supervision with validation procedures to produce retrieval resources at larger scale than earlier marker-based relation datasets.
  • RATIO strips discourse markers from inputs and uses them to define candidate relevance for ranking against a large sentence corpus.

3 Problem Definition

RATIO formulates scientific inspiration retrieval as ranking corpus sentences for a specified ideation operation and query. Separate operation-specific retrievers must match candidates to the particular query, not merely recognize an operation-like style.

  • Given a scientific query q, operation r, and corpus C, the task ranks candidate sentences by how well each instantiates r with respect to q.
  • The query may describe a research problem, claim, observation, or methodological objective, while the candidate corpus contains scientific-paper sentences.
  • ADDRESS retrieves approaches for problems, BROADEN retrieves more general formulations, and SPECIFY retrieves concrete cases, mechanisms, or examples.BROADEN and SPECIFY move in opposite directions along an abstraction hierarchy, whereas ADDRESS is a functional response to a research need.
  • Separate retrievers implement relation-conditioned scoring through hard routing to an operation-specific expert.All retrievers rank candidates from the same heterogeneous corpus, so routing does not restrict the search space.
  • Operation-specific training requires query-conditioned compatibility because in-batch negatives can be valid candidates for other queries under the same operation.
  • Selecting an operation identifies the relevance criterion but does not identify which candidates satisfy it for the particular query.

4 Benchmark Construction

RATIO constructs three operation-specific retrieval tasks from discourse-marked sentence pairs mined across full-text scientific papers. The benchmark combines curated and expanded marker lexicons, temporal splitting, shared candidate pools with distractors, and human- and LLM-vetted evaluation data.

  • Marker-based mining: Discourse markers identify adjacent sentence pairs in which the second sentence addresses, broadens, or specifies the preceding sentence.The marker is removed from the gold sentence and retained as construction metadata; the resulting tuple contains the preceding sentence, marker, and remaining proposition.
  • Lexicon construction: The lexicon combines manual curation, rule-based pattern expansion, and LLM generation, with expert review and validation used to filter candidates.LLM expansion retained 3,779 strong-true markers, representing 89% of the final 4,252-marker lexicon.
  • Corpus mining: 809 of 4,252 markers fired on approximately 366.6M consecutive sentence pairs, yielding approximately 3M pairs from 1.1M papers.Two NLP experts independently vetted the 809 firing markers and agreed that every one was valid.
  • Temporal split: The benchmark contains 3,017,476 pairs distributed across SPECIFY, ADDRESS, and BROADEN, with publication-date-based train, validation, and test partitions.The reported counts are 2,779,177 SPECIFY pairs, 222,707 ADDRESS pairs, and 15,592 BROADEN pairs; test data uses 2026 papers.
  • Candidate pools: Shared candidate pools and distractor markers prevent retrieval through relation-specific corpus shortcuts and require query-conditioned relational compatibility.Distractor sentences are gold for no query, while other pool sentences are gold for exactly one query.
  • Silver test set: The silver test set keeps pairs accepted by both retained LLM prompts, producing 17,579 queries with corresponding gold candidates.Expert agreement F1 scores were .87 for ADDRESS, .90 for SPECIFY, and .82 for BROADEN.

5 Experiments and Evaluation

Experiments evaluate baseline and relation-specific fine-tuned retrievers on RATIO’s silver test set and through candidate-level judgments of top-10 results. Fine-tuning substantially improves retrieval and surfaces valid alternatives, but absolute performance remains limited.

  • Experimental setup: The evaluation compares BM25 and three dense retrievers before and after operation-specific contrastive fine-tuning, using separate setups for ADDRESS, BROADEN, and SPECIFY.The experiments include multiple prefix configurations and 1,800 GPU hours of training.
  • Effect of fine-tuning: Move-specific fine-tuning increases ModernBERT-embed-large MRR@10 by 1.6×–2.4× across the three operations, while pre-trained models provide limited gains over BM25.ModernBERT-embed-large is strongest on every operation in the silver-set comparison.
  • Effect of fine-tuning: Relative gains do not track training-set size: ADDRESS gains approximately 2.4× using 7.5% of SPECIFY’s training pairs, whereas BROADEN has the smallest absolute gain at +10.0.The comparison indicates differing operation difficulty and data effects in the reported results.
  • Limitations: Absolute performance remains low: the best model fails for most ADDRESS and BROADEN queries, and MRR@100 exceeds MRR@10 by at most 0.8 points.The reported gap suggests these failures are not near-hits.
  • Top-K candidate evaluation: Candidate-level evaluation judges each top-10 result for relation validity, allowing multiple accepted candidates beyond the mined adjacent-sentence positive.The study evaluates 1,200 queries and reports Hit Rates, Precision, NDCG, and MAP at rank 10.
  • Top-K candidate evaluation: Under hard agreement, fine-tuned ModernBERT-embed-large reaches an accepted candidate for 89.0% of SPECIFY and 76.5% of ADDRESS queries, versus 0.61 accepted ADDRESS candidates per baseline list.The reported comparison uses 400 queries per relation and top-10 candidates.
  • Results and analysis: For tuned ADDRESS, 41.2% of queries have an accepted alternative despite the mined positive being absent, and most accepted alternatives come from other papers.Across operations, 88–90% of accepted non-gold candidates are cross-paper; counting accepted alternatives raises ADDRESS MRR@10 from 24.5 to 41.2.
  • Inspiration potential: Fine-tuned retrievers are preferred in reference-free comparisons for ADDRESS, SPECIFY, and BROADEN at 70.2%, 61.6%, and 56.2%, respectively.The ordering matches the reported relative MRR@10 gains, although BROADEN is least stable.

6 Conclusion

RATIO establishes retrieval conditioned on three ideation roles and provides scalable resources for scientific inspiration retrieval. Experiments show move-specific fine-tuning improves retrieval, while performance remains far from solved.

  • RATIO defines ADDRESS, BROADEN, and SPECIFY as retrieval operations for responses, general formulations, and concrete instances.
  • The benchmark contains over 3 million query–positive pairs mined from full-text papers with vetted discourse markers.Its silver test set uses human-calibrated LLM judgments.
  • Move-specific fine-tuning substantially improves retrieval across the three ideation operations.Tuned models also frequently retrieve valid inspirations other than the mined positive, especially for ADDRESS.
  • Overall retrieval performance leaves substantial room for improvement.
  • The methodology is presented as extensible to other relations and domains.The paper also identifies future work beyond adjacent sentences and computer-science literature.

A Relation-Marker Vocabulary: Full Construction

The relation-marker vocabulary is built through manual curation, LLM generation, Hearst-inspired pattern expansion, and validation. These processes produce relation-specific markers for mining typed sentence pairs.

  • Three LLMs generated marker variants labeled strong-true, weak-true, or false according to contextual reliability.
  • Hearst-inspired templates fill typed slots from curated term lists and expand candidates into grammatical morphological variants.
  • ADDRESS pattern expansion produced 257 valid novel markers from roughly 6K candidates, while BROADEN produced 40 from 810.SPECIFY was not expanded with templates.
  • Embedding-based discovery generated 820 candidates from ten neighbors of each of 82 foundational markers, with roughly 30 additional markers validated by GPT-4 but not reported.
  • Distractor markers cover overly general cues and relations outside ADDRESS, BROADEN, and SPECIFY, with 1,000 validated by an NLP expert.
  • The full lexicon is intended for public release and includes target relations alongside distractor subcategories.

B Data Extraction Details and Figures

Data extraction cleans sentences, stores metadata, adds distractor-derived pairs, and partitions data under constraints related to pretraining exposure, publication growth, and BROADEN coverage.

  • Regex-based preprocessing normalizes sentences and removes malformed text, noise, discourse-marker prefixes, and list prefixes.The cleaning produces continuations suitable for query–marker–positive extraction.
  • Each stored tuple includes corpus ID, publication date, field labels, and section category.A worked extraction example is provided in Figure 3.
  • Distractor-marker pairs are added to the corpus pool to make the benchmark more challenging.
  • Partition boundaries balance model-pretraining exposure, rapid growth in computer-science publishing, and sufficient BROADEN test data.Partitioning by marker produced inflated metrics because of exposure to older evaluation papers.

C Training, Evaluation, Objectives, and Hyperparameters

The experiments compare lexical and embedding retrievers across relation-specific setups, using contrastive training and validation-selected checkpoints. ModernBERT-embed-large performs best, but retrieval misses remain substantial.

  • Experiments use BM25S and Faiss alongside ALL-MPNET-BASE-V2, ModernBERT-embed-large, and Stella-en-1.5B-v5.
  • Training uses Multiple Negatives Ranking Loss with in-batch negatives for contrastive learning.
  • The study evaluates 15 relation-and-setup combinations, including generic, search-query, and system-recommended prefixes.
  • MRR@10 is primary, while MRR@100 tests whether performance depends only on rank depth; MRR@100 exceeds MRR@10 by at most 0.8 points.
  • MRR@10 reaches 39.1 on SPECIFY, 21.5 on ADDRESS, and 22.2 on BROADEN for the recommended ModernBERT prefix.The mined positive reaches the top ten for 56.7%, 35.4%, and 34.6% of queries, respectively.
  • The recommended prefix gives ModernBERT a 3.3-point ADDRESS gain before tuning, shrinking to 0.1 after tuning.For Stella, prefix differences are 0.0 before tuning and 0.6 after tuning.

D Full Automatic Results

The section evaluates automatic retrieval across relation-specific test sets and silver data, using Recall and MRR with M@10 as the primary metric. Fine-tuning improves separation between accepted and rejected pairs, while tuned retrieval reduces lexical overlap relative to off-the-shelf retrieval.

  • Lexical overlap: Tuned retrieval has lower query–candidate lexical overlap than off-the-shelf retrieval in all 36 comparisons.For SPECIFY, tuned top-10 overlap is below the positive reference by −.010 on test, −.019 on silver, and −.029 on gold; rank–overlap Spearman correlation ranges from −.20 to −.28.
  • Automatic retrieval results: M@10 is the primary metric for automatic retrieval results across relation-specific test and silver sets.Results compare baseline and trained systems across input setups and prefixes against relation-shared candidate corpora.
  • Automatic retrieval results: Fine-tuning increases Recall@10 separation between accepted and rejected sampled test pairs, particularly for ADDRESS.Accepted pairs have higher Recall@10 than rejected pairs before filtering in every relation and model state.

G Discovered Positives: Ranking validation set

The ranking-validation analysis examines whether retrieved candidates provide valid ideation alternatives beyond mined positives. Discovered positives overlap queries less than mined positives, while ADDRESS examples especially benefit from explicit anaphoric links.

  • Discovered positives: Discovered positives overlap queries less than mined positives: .075 vs .137 for SPECIFY, .086 vs .124 for ADDRESS, and .093 vs .129 for BROADEN.All differences are statistically significant, with rankbiserial effects of .46, .29, and .27 respectively and p < .001.
  • Validation: Table 10 compares human–human and prompt–annotator agreement across a five-annotator panel and a 300-pair calibration set.The caption identifies precision, recall, κ, and Krippendorff’s α as agreement measures, with agreement reported as high for this difficult task.
  • Relation-specific cues: ADDRESS pairs containing explicit anaphors are accepted at 71.9%, versus 27.7% for bare markers.Explicit anaphors appear in 93.4% of sampled ADDRESS pairs but only three SPECIFY and two BROADEN pairs; the ADDRESS association has Fisher OR 6.7, p < .001.

H TOP-10 Judge Inspiration

The inspiration judge compares fine-tuned and off-the-shelf ranked lists for each ideation relation. ADDRESS receives significantly stronger preference than BROADEN and SPECIFY, while BROADEN shows the weakest judge reliability and a position effect.

  • Inspiration across relations: ADDRESS is preferred over BROADEN (p < 10^-3) and SPECIFY (p = 0.02), whereas SPECIFY and BROADEN are indistinguishable.The comparison concerns win-rate differences among the three ideation relations.
  • Ranking validation: Ranking validation reports hard-agreement and soft-agreement outcomes for 400 queries per relation using top-10 retrieval metrics.The reported metrics include HR@10, P@10, and N@10, comparing matching-relation fine-tuned retrievers with off-the-shelf baselines.
  • Candidate origins: Among judge-accepted non-gold candidates, 86–91% of all top-10 candidates are retrieved from a different paper than the query.The table also reports accepted-candidate counts and distinct-paper counts by setting.

I Prompts

The prompts operationalize relation-specific discourse-marker classification and silver-set validation. They require direct semantic ties, reject ambiguous cues, and separately judge the validity of problem and solution sentences.

  • I.1 Discourse Markers Classifier: ADDRESS, BROADEN: BROADEN prompts define generalization as moving from a specific claim to a wider scope or higher abstraction level.Explicit scope-expansion phrases such as “more generally” and “more broadly” are true cues.
  • I.1 Discourse Markers Classifier: ADDRESS, BROADEN: ADDRESS prompts require the following sentence to be a remedy directly inspired by and targeting the preceding problem.A problem–solution pair must articulate the problem in S1 and the solution or mitigation in S2.
  • I.1 Discourse Markers Classifier: ADDRESS, BROADEN: Explicit anaphoric purposive phrases such as “to address this problem” are accepted, while bare purposive or contribution phrases are rejected.The classifier requires an anaphoric tie to the prior problem rather than relying on generic purpose or contribution language.
  • I.1 Discourse Markers Classifier: ADDRESS, BROADEN: The classifier outputs a JSON label, one-sentence rationale, and confidence value, with no text outside the object.The same strict output format is specified for the prompt listings.
  • I.1 Discourse Markers Classifier: ADDRESS, BROADEN: Ambiguous, additive, causal, exemplary, restatement, contrast, and emphasis phrases are rejected when they do not reliably signal the target relation.The prompts prefer discarding ambiguous connectives rather than admitting noisy examples.
  • I.2 Silver Test Set: Query-Gold ADDRESS, SPECIFY and BROADEN: The silver-set judge evaluates S1 as a valid problem, S2 as a valid solution or mitigation, and then their relation match.The connecting phrase is ignored when assessing whether the sentence content instantiates the relation.
  • I.2 Silver Test Set: Query-Gold ADDRESS, SPECIFY and BROADEN: A valid problem must state a clear, specific core problem in S1 rather than merely asserting a problem or expressing a capability, requirement, or assumption.External references cannot conceal the core problem, although non-core anaphoric references are allowed.
Loading 2608.27394v1…