PaperlayerView exact revision on arXiv

Source-linked AI summary

Nothing Left Behind: Finding Evidence from a Large Randomized Evaluation Corpus

Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg, Linxi Wang, Riccardo Orlando, Piriyakorn Piriyatamwong, Samuel Paul Fraiberger

arXiv:2610.36910v1cs.CL

TL;DR

Evidence synthesis needs a sufficiently complete set of relevant studies, while taxonomy labels can constrain discovery. The paper compares taxonomy search with semantic retrieval followed by query-specific filtering; at 2,500 candidates, the combined approach reports higher macro known-positive paper recall than entity linking. The results support using semantic retrieval for discovery and labels for organization, with search depth and filtering policy shaping query-time cost.

  • Problem

    Evidence tasks such as synthesis need relevant items recovered comprehensively, while labels can miss concepts or over-retrieve when categories are broad.

    Source: §I · p. 1

  • Method

    The system retrieves intervention–outcome pairs by semantic similarity and applies a language model to filter candidates for query-specific relevance.

    Source: §I · p. 1, A. Evidence Structure · p. 2

  • Results

    At K = 2,500, semantic retrieval with Gemini 2.5 Flash filtering achieves 85.0% macro known-positive paper recall, versus 45.1% for entity linking.

    Source: §VIII · p. 6

  • Takeaways & Limitations

    The case study supports semantic retrieval for evidence discovery and labels for organizing results, with search depth and filtering policy central to query-time cost.

    Source: §VIII · p. 6

    The review benchmark measures known-positive paper recall within one development-related RCT corpus, not coverage of the entire literature or every eligible effect.

    Source: §VII · p. 6

Abstract

from arXiv · show

Many AI systems need to find every relevant items in a large collection and then organize what they found. The standard way to do this is to tag each item with labels from a curated vocabulary and search on those labels. Those labels are useful for organizing results, but searching on them misses items that do not fit a label and returns too much when labels are broad. We compare this approach with an embedding-based approach: find candidates by meaning using text embeddings, then have a language model keep only the relevant ones. In a corpus of 7,354 randomized trials, label search found 45% of the evidence a systematic review would include, while the new approach found 92%, at a cost per query that rises with search depth. These results support a two-stage architecture that combines broad candidate retrieval with query-specific evidence filtering.

I. Introduction

Some evidence tasks require a sufficiently complete set of relevant items, not just a few convincing passages, because missing studies can affect synthesis. The paper compares taxonomy-based retrieval with semantic candidate retrieval followed by query-specific filtering in ImpactAI.

I. Introduction

  • Missing relevant studies can change pooled estimates, so evidence search must prioritize recall while balancing screening and computational cost.Precision remains important because returned irrelevant items still require screening.Source: §I · p. 1
  • ImpactAI’s original architecture links questions to curated taxonomy labels, making discovery depend on catalog coverage, granularity, and linking decisions.Broad labels can over-retrieve, while absent concepts may be missed or mapped imperfectly.Source: §I · p. 1
  • The alternative retrieves intervention–outcome pairs by semantic similarity, then uses a language model to filter candidates for query-specific relevance.Labels can organize the selected evidence without determining the initial candidate pool.Source: §I · p. 1

II. Two Retrieval Designs: The ImpactAI Case

  • The case study compares the architectures and examines retrieval, filtering, and operational trade-offs; corpus construction is taken as given.The focus is evidence found before downstream synthesis.Source: §I · p. 1, §II · p. 2

A. Evidence Structure

ImpactAI retrieves evidence at the intervention–outcome-pair level, allowing relevant effects within a study to be distinguished. Its taxonomy-based design supports normalization and organization, but discovery remains bounded by catalog coverage and linking decisions.

A. Evidence Structure

  • ImpactAI’s corpus contains 7,354 randomized trials and 74,826 intervention–outcome pairs, with attributes describing study context and evidence characteristics.The attributes include population, setting, time horizon, implementation, measurement, and sample size.Source: A. Evidence Structure · p. 2
  • An intervention–outcome pair is the retrieval unit, linking intervention and outcome descriptions with study and effect information and its source RCT.Pair-level retrieval is finer-grained than paper-level retrieval because one RCT may examine multiple interventions or outcomes.Source: A. Evidence Structure · p. 2

B. Taxonomy-based Retrieval

  • The taxonomy design detects query spans, assigns catalog entity types, matches candidate entities, and retrieves evidence linked to selected IDs.It normalizes synonymous expressions and supports inspectable concepts and category-based organization.Source: B. Taxonomy-based Retrieval · p. 2
  • Off-catalog concepts lack an exact entity, while compound queries and errors in span detection, typing, selection, or tagging can remove relevant evidence.Finer labels can reduce some ambiguities but increase mapping and maintenance work; broad categories can also erase distinctions in a specific query.Source: B. Taxonomy-based Retrieval · p. 2
  • For a fixed catalog and linking decision, increasing the semantic candidate budget does not change the taxonomy result.This explains the flat taxonomy baseline in the recall comparison.Source: B. Taxonomy-based Retrieval · p. 2

C. Semantic Retrieval and Relevance Filtering

Semantic retrieval compares a query directly with textual descriptions of intervention–outcome pairs, then filters candidates for query-specific relevance. Candidate generation and filtering serve complementary roles, but a top-K budget can limit what the filter can recover.

C. Semantic Retrieval and Relevance Filtering

  • The neural retriever matches queries to intervention–outcome descriptions by embedding-based similarity, without requiring an explicit taxonomy match.An LLM filter then assesses candidates against the query’s requirements.Source: C. Semantic Retrieval and Relevance… · pp. 2–3
  • Broad candidate generation aims to preserve recall, while relevance filtering aims to improve precision by removing irrelevant candidates.The two stages can be evaluated independently.Source: C. Semantic Retrieval and Relevance… · p. 3
  • The system indexes pair descriptions and study context, retrieves the closest K pairs, and retains those judged relevant to the full query.In the October latency configuration, candidates are grouped by source paper with at most 15 pairs per prompt, while each pair receives a separate relevance decision.Source: C. Semantic Retrieval and Relevance… · p. 3
  • Filtering cannot recover a relevant pair omitted during candidate generation, and a top-K retrieval cannot fully recover a relevant set larger than its budget.A positive relevance decision also does not establish that evidence is comparable for pooling.Source: C. Semantic Retrieval and Relevance… · p. 3

III. Experimental Setup

The evaluation asks whether filtering judgments agree with human annotations, how retrieval performs for downstream filtering, how much systematic-review evidence is recovered, and how search depth affects latency.

A. Questions

  • The evaluation tests agreement between Gemini 2.5 Flash relevance judgments and human annotations.Source: A. Questions · p. 3
  • It compares candidate-set retrieval quality and measures end-to-end recovery of evidence identified through systematic reviews.Source: A. Questions · p. 3
  • It examines how search depth affects module latency as an operational trade-off.Source: A. Questions · p. 3

B. Models and Configuration

The evaluation uses multiple datasets to assess relevance filtering, candidate retrieval, and downstream retrieval against review-derived reference sets. The models include embedding retrievers and a taxonomy-based baseline.

B. Models and Configuration

  • Gemini embeddings, Voyage, E5-large-v2, and Granite are compared, with Gemini 2.5 Flash used for taxonomy linking and relevance filtering.The taxonomy quality baseline uses Gemini 2.5 Flash for entity linking.Source: B. Models and Configuration · p. 3

C. Datasets

  • Filtering uses 100 human-generated queries, 100 synthetic queries, and 10 double-annotated queries, each with 30 Gemini-embedding candidates.The double-annotated comparison uses 276 passages with both annotations; 24 lack one annotation.Source: C. Datasets · pp. 3–4
  • Candidate retrieval is tested on 374 Beta Tester queries and 1,000 Synthetic queries using LLM judgments over pooled results.The review-derived benchmark contains 348 formulations from 29 review sources; separate user feedback covers 45 queries.Source: C. Datasets · p. 4

D. Evaluation Units and Metrics

Evaluation distinguishes candidate-pair retrieval from passage-level filtering and paper-level review recall. Gemini 2.5 Flash judgments show moderate agreement with human annotations.

D. Evaluation Units and Metrics

  • The retrieval unit is an intervention–outcome pair, K counts candidate pairs, and filtering agreement and pooled retrieval metrics operate on passages.For review recall, returned pairs are mapped to distinct source papers and recall is averaged equally across query formulations.Source: D. Evaluation Units and Metrics · p. 4
  • Lower-K recall uses ranked prefixes of saved top-2,500 outputs, while larger Beta Tester and Synthetic benchmark scores are relative to pooled LLM judgments.Micro-F1 measures overall agreement; class-specific F1 combines precision and recall for each class.Source: D. Evaluation Units and Metrics · p. 4, A. Filtering Module · p. 4

A. Filtering Module

  • Cohen’s κ is 0.5026 on Beta Tester queries and 0.5045 on Synthetic queries for Gemini 2.5 Flash, versus 0.4340 for human–human agreement.Gemini’s corresponding micro-F1 scores are 0.7513 and 0.7510.Source: A. Filtering Module · p. 4, Table III · p. 4

B. Retrieval Module

Across the Beta Tester and Synthetic benchmarks, Gemini embeddings achieve the strongest reported recall among evaluated embedding models. Gemini is therefore used for candidate generation in later experiments.

B. Retrieval Module

  • At k = 100, Gemini achieves 0.49 F1 on Beta Tester queries and 0.53 F1 on Synthetic queries.At k = 100, the reported F1 scores are 0.27 for Voyage, 0.20 for E5-large-v2, and 0.26 for Granite on Beta Tester queries; Synthetic scores are 0.31 for Voyage and 0.28 for Granite.Source: B. Retrieval Module · pp. 4–5, Table IV · p. 5, Table V · p. 5
  • At k = 1000, Gemini reaches 0.93 recall on Beta Tester queries and 0.96 recall on Synthetic queries, the highest recall at every depth on Synthetic.At k = 1000 on Beta Tester queries, Voyage, E5-large-v2, and Granite reach recall of 0.67, 0.50, and 0.62, respectively.Source: B. Retrieval Module · pp. 4–5, Table IV · p. 5, Table V · p. 5
  • Gemini embeddings are selected for the remaining experiments because they have higher candidate recall across both benchmarks.The relevance filter then screens candidates against the full query.Source: B. Retrieval Module · pp. 4–5

C. Downstream Evidence Retrieval

On the review-derived benchmark, semantic retrieval recovers more known-positive papers than entity linking, while filtering reduces its candidate coverage. Combining both routes adds complementary evidence, with query time increasing at greater search depth.

C. Downstream Evidence Retrieval

  • The separate feedback evaluation reports a 94.7% relevance rate for 348 assessed passages across 45 queries.Paper recall is evaluated separately against review-derived reference sets.Source: C. Downstream Evidence Retrieval · p. 5
  • At K = 2,500, candidate retrieval recovers 96.7% of expected papers on the macro measure, versus 45.1% for EL; filtering retains 85.0% of candidates, a loss of 11.7 percentage points.Table VI separates candidate coverage, filtering, and their union with entity linking.Source: C. Downstream Evidence Retrieval · p. 5
  • The union with EL adds 12.6 percentage points over filtered semantic retrieval at K = 100 and 2.9 points at K = 2,500.The routes recover complementary evidence, and the union retains EL results without applying the semantic filter to that branch.Source: C. Downstream Evidence Retrieval · p. 5

V. What It Costs: Module Latency and Search Depth

  • At K = 2,500, median latency is 1.48 seconds for retrieval alone and 77.22 seconds with filtering, compared with 6.00 seconds for entity linking.Filtering accounts for most query time because it reads every candidate; depth is a product choice between more comprehensive retrieval and a longer wait.Source: §V · pp. 5–6, Table VII · p. 6

VI. Discussion

Semantic retrieval improves evidence coverage, while labels remain useful for organizing results. Search depth brings a recall–precision trade-off, and both architectures have maintenance and scaling costs.

A. Use Meaning to Find and Labels to Organize

  • 85.0% macro paper recall at K = 2,500 versus 45.1% for entity linking shows higher evidence coverage with filtered semantic retrieval.Directly searching evidence descriptions helps with concepts or combinations poorly represented in the taxonomy.Source: A. Use Meaning to Find and Labels to… · p. 6
  • Labels still normalize terminology and organize evidence for browsing and comparison, while semantic retrieval offers more flexible discovery.Semantic results still need organization and comparability checks before pooling.Source: A. Use Meaning to Find and Labels to… · p. 6

B. Recall, Precision, and Cost

  • On Beta Tester queries, Gemini recall rises from 0.47 at K = 100 to 0.93 at K = 1,000, while precision falls from 0.52 to 0.13.At K = 2,500, relevance filtering reduces recall from 96.7% before filtering to 85.0% afterwards.Source: B. Recall, Precision, and Cost · p. 6
  • The operating point depends on the cost of missing evidence versus screening extra candidates and waiting; a hybrid union with entity linking helps most at small budgets.The hybrid approach also adds screening and execution costs.Source: B. Recall, Precision, and Cost · p. 6

C. Maintenance and Scale

  • Taxonomy search requires maintaining its vocabulary and evidence tags as the collection evolves.Source: C. Maintenance and Scale · p. 6
  • Semantic search avoids mapping each new concept to a catalog, but requires embedding, index maintenance, and relevance-model monitoring.Its scores and judgments can also be harder to inspect as a whole than explicit linked labels.Source: C. Maintenance and Scale · p. 6
  • A fixed candidate budget can truncate broad queries as relevant evidence grows; larger budgets may preserve coverage but increase filtering work.Taxonomy retrieval can remain useful for exhaustive enumeration within a known label, and neither design has constant cost as the corpus grows.Source: C. Maintenance and Scale · p. 6

VII. Limitations and Next Steps

The conclusion reports higher known-positive paper recall with semantic retrieval and filtering, alongside increased query-time cost. The evaluation is bounded by one corpus and leaves several operational and downstream questions for future work.

VII. Limitations and Next Steps

  • The results come from one corpus of development-related RCTs, and the review benchmark measures known-positive paper recall within that corpus, not coverage of the entire literature.Both labels and evidence descriptions depend on extraction quality; some benchmarks use LLM judgments, while human feedback and double-annotation samples are limited.Source: §VII · p. 6
  • The latency comparison uses serial modules with different service configurations, and its timed entity-linking service and October batched filter differ from their quality runs.Controlled throughput, monetary cost, and effects on downstream meta-analysis remain unevaluated.Source: §VII · p. 6
  • Next steps include expanding human annotation, testing smaller relevance models, choosing depth per query, and measuring concurrent-load performance.End-to-end synthesis experiments should test whether more complete evidence sets change pooled estimates or conclusions.Source: §VII · p. 6

VIII. Conclusion

  • 45.1% to 85.0% macro known-positive paper recall at 2,500 candidates, with 94.7% high-perceived precision, comes with substantially higher query-time cost.Source: §VIII · p. 6

Related papers

Loading 2610.36910v1…