PaperlayerView exact revision on arXiv

Source-linked AI summary

Structurally-bounded Agentic Graph Exploration for Evidence-Grounded Scholarly DeepSearch

Rima Hazra, Sayan Layek, Somnath Banerjee, Soumen Chakrabarti, Animesh Mukherjee

arXiv:2608.24809v1cs.CLcs.IR

TL;DR

Crase addresses whether scholarly discovery can avoid the oversight problems of open-ended, model-directed search without sacrificing retrieval effectiveness. It uses one seed-search stage, bounded citation expansion, claim-level evidence filtering, and recency-aware graph ranking. The paper reports improved retrieval over agentic baselines with fewer external calls, tokens, time, and cost, while making evidence decisions and termination auditable.

  • Problem

    Open-ended deep research loops make the explored literature, evidence selection, and stopping point difficult to audit.

  • Method

    Crase forms seeds from one search stage, expands them to a 1.5-hop citation graph, prunes unsupported citation edges, and ranks retained papers with a query-personalized recency-aware random walk.

  • Results

    Crase improves retrieval over agentic baselines while using fewer external calls, tokens, time, and cost, with evidence decisions largely aligning with human judgments.

  • Takeaways & Limitations

    Constraining agentic search around explicit, inspectable evidence can improve retrieval effectiveness and auditability.

  • Takeaways & Limitations

    A relevant paper outside the citation neighborhood induced by the initial seeds cannot be recovered through later query reformulation.

Abstract

from arXiv · show

We present Crase, a bounded and inspectable alternative to deep research agents for scholarly search. Instead of an open-ended search loop, Crase queries a search engine once for seed papers, expands them along their 1.5-hop citation neighborhood, prunes citation edges whose claims lack entailment support, and ranks the remaining papers with a recency-aware random walk. This makes the candidate set, the reason each paper is kept, and the stopping condition explicit and fixed before inference. On LitSearch and one further benchmarks over a 500K-paper arXiv corpus, Crase outperforms deep research agents built on proprietary models by up to 3$\times$ recall@50 at roughly a third of the cost.

Introduction

Crase replaces open-ended, model-directed scholarly search with a bounded citation-graph process whose evidence and stopping condition are inspectable. It seeds once, expands locally, filters citation support, and ranks candidates structurally.

  • Motivation: Deep research agents repeatedly search, evaluate evidence, and decide when to stop, making exploration, error propagation, and evidence selection difficult to oversee.The final report reveals little about why sources shaped decisions or why searching terminated.
  • Motivation: Crase asks whether scholarly discovery can remain retrieval-effective while becoming structurally bounded and evidence-inspectable.
  • Method: Crase replaces repeated querying with one seeded search, a 1.5-hop citation neighborhood, claim-level edge filtering, and ranking over the resulting graph.The design uses the citation network as an existing map of literature relevance rather than rebuilding it through sequential searches.
  • Method: The graph boundary, pruning rule, ranking walk, and stopping point are fixed, while the model supplies local plans, claims, and entailment judgments that remain individually checkable.Crase therefore makes candidate inclusion and exclusion traceable without claiming that model judgments are correct.
  • Method: Crase uses focused keyword-style sub-queries to form seeds, expands each seed through references and citing papers, and preserves citation links among retrieved papers.The seed count m and returned-paper count k control the bounded process.

Datasets

The evaluation corpus contains roughly 500K arXiv papers from 2016–2026 across eight AI, ML, NLP, and computer-vision subject categories.

  • Corpus: The corpus contains roughly 500K arXiv papers submitted between January 2016 and July 2026.Included records contain titles, publication years, abstracts, and authors.
  • Corpus: Papers span eight subject categories across Artificial Intelligence, Machine Learning, NLP, and Computer Vision.The categories are cs.AI, cs.LG, cs.CL, cs.NE, cs.CV, cs.IR, cs.MA, and stat.ML.

Evaluation metrics

Evaluation measures both how many relevant papers appear within top-k results and how highly relevant papers are ranked.

  • Recall: Recall@k measures the proportion of ground-truth relevant papers retrieved within the top k results, aggregated across queries.Recall@5 emphasizes early ranking, while Recall@20–Recall@50 capture broader evidence coverage.
  • MAP: MAP@k measures mean average precision at k and rewards ranking relevant papers higher.Average precision uses precision at each relevant rank within the returned list.

Experimental setup

Crase is evaluated against deep-research and graph-based baselines using retrieval metrics across ACL, ICLR, and arXiv, alongside controlled variations in extraction, ranking, and seed selection.

  • Baselines and datasets: DeepResearch-GPT and DeepResearch-Claude search exclusively over graph-expanded paper candidates using title, abstract, and introduction embeddings.Both baselines retain the top-50 papers ranked by relevance to the query.
  • Ablations: Crase ablations vary claim-extraction model and graph-ranking algorithm while holding graph construction, pruning, and candidate pool fixed.The four configurations combine Qwen-32B or Llama-70B extraction with PPR or SALSA ranking.
  • Evaluation protocol: Main-result tables report Recall@K and MAP@K across three datasets and retrieval setups, with best and second-best results marked.Crase-Llama-PPR is reported as strongest on ACL and ICLR and competitive or best across most arXiv settings.
  • Ablations: The Recency variant changes only the age reference, measuring paper age relative to the query’s publication year rather than the current year.Its extraction and ranking remain Qwen-32B and PPR.

Results

Crase generally improves retrieval over agentic and graph-based baselines, with especially strong ACL and ICLR results, lower predetermined interaction cost, and evidence decisions that largely match expert judgments.

  • Retrieval performance: 0.3848 recall@50 on ACL exceeds the best baseline’s 0.2629, while Crase leads at every reported recall cutoff.On ICLR, recall@50 is 0.3720 versus 0.2012 for Spector2-Deepwalk; arXiv is more competitive at low K.
  • Seed selection: Current-year seeds improve nearly all Recall@K and MAP@K scores, with the only reported drop at ICLR recall@5.On ACL, recall@50 rises from 0.2654 to 0.3144 and MAP@50 from 0.0311 to 0.0382.
  • Retrieval performance: 0.3572 recall@50 on arXiv exceeds Spector2-Deepwalk’s 0.3446, although Spector2-Deepwalk leads at recall@5 with 0.0674 versus 0.0533.Crase performs best for the remaining arXiv recall cutoffs.
  • Efficiency: 0.3659 recall@50 on ICLR compares with 0.1220 for both proprietary-agent baselines, while median cost decreases to $0.37 from $1.76 and $2.06.The comparison combines higher retrieval recall with fewer external interactions and lower inference cost.
  • Human auditability: 84.0% of Crase’s retain/prune decisions agree with majority expert judgment, with 88.0% of retained and 80.0% of pruned edges validated.Expert agreement is 68.0% unanimous, with Fleiss’ κ = 0.71.
  • Ablation results: Crase-Llama-SALSA provides the most stable performance across datasets and metrics, while Crase-Qwen-PPR is stronger for recall at larger cutoffs.Llama-SALSA achieves the best MAP at every K on arXiv, whereas Qwen-SALSA leads recall at smaller cutoffs.
  • Complexity and cost: Crase’s invocation count is exactly 1 + |V| + |E|, fixed before inference, whereas generic search-agent cost grows quadratically with an unbounded tool trajectory.Crase is linear in a structurally fixed graph with constant-size prompts and a predetermined stopping condition.
  • Conclusion: The paper concludes that citation-guided exploration and claim-level evidence filtering improve retrieval effectiveness and auditability while using fewer resources than agentic baselines.The conclusion frames explicit structural constraints as the basis for predictable evidence inspection and resource use.

Complexity Analysis

Crase bounds computation by a structurally fixed evidence graph, making model invocations, termination, and memory requirements explicit before inference. Its main costs scale with graph size rather than an unbounded agent trajectory.

  • The evidence graph has at most mL(dmax + 1) nodes, or O(mL dmax), with edges O(|V| ¯d) for sparse citation graphs.Here, L is the number of sub-queries, m the retained seeds per sub-query, and dmax the maximum seed degree.
  • The PPR ranking stage uses sparse power iteration, requiring O(|V| + |E|) per iteration and T = O(α^-1 log(1/ϵ)) iterations.Top-k extraction adds O(|V| log k), while the sparse method avoids the O(|V|^3) closed form.
  • Crase makes exactly 1 + |V| + |E| model invocations, known before inference and independent of intermediate outputs.Its termination condition is exhaustion of E followed by PPR convergence.
  • A generic search-and-synthesis agent instead incurs trajectory-dependent quadratic cost because it appends observations to an ever-growing context and decides when to terminate.Crase keeps peak prompt length at O(ℓ) for extraction or O(nmax^2) for entailment.

Execution Trace of Stage 1

The Stage 1 trace fixes a small external-search budget while deliberately allowing broad lexical retrieval into the seed set. Subsequent graph processing removes off-topic seeds through claim-level pruning.

  • Stage 1: Plan and seed construction: Stage 1 uses one planning call and L = 5 Semantic Scholar index calls, after which no further external search occurs.The five calls correspond to five complementary sub-queries generated from the original information need.
  • Stage 1: Plan and seed construction: Seed construction is deliberately permissive: broad keyword overlap admits quantum-information, applied-physics, and tangential finetuning papers into S.These seeds are subsequently filtered rather than excluded during retrieval.
  • Stage 1: Plan and seed construction: The generated plan contains five keyword-style sub-queries covering faithful measurement, human explanations, helpfulness metrics, explicit guidance, and explanation impact.
  • Stage 1: Plan and seed construction: The trace retrieves papers through relevance, recency, and intersection rankings before unioning and deduplicating them into the seed set.For the example query, the resulting seed set contains 29 papers.
  • Stage 1: Plan and seed construction: Lexically matched but off-topic seeds can enter S and are removed in Stage 2 when claim-level entailment assigns Δ = 0.
  • Stage 1: Plan and seed construction: Intersection retrieval recovers human-feedback literature that one ranking misses, while narrow phrasing can produce sparse overlaps such as two seeds for ρ5.

Claim Extraction

Crase represents each paper with a small, structured set of contribution-level claims to support cross-paper entailment comparisons. The extraction prompt emphasizes standalone, atomic, typed claims anchored to source text.

  • Each paper is represented by 4–5 atomic claims extracted from its abstract and introduction.
  • Each claim is anchored to a short verbatim source phrase so claims can be compared across papers in Stage 2’s entailment matrix.
  • Every claim must be a standalone, self-contained declarative assertion containing one idea rather than a compound assertion.
  • Claims are tagged as EMPIRICAL, THEORETICAL, METHODOLOGICAL, or CONTEXTUAL according to their role.The categories cover results, mechanisms or formal relations, justified design choices, and gaps or motivations.
  • The extractor is instructed to represent the paper’s own contribution rather than claims attributed to prior work.

Test Data Preparation

The held-out arXiv preparation converts each paper into one broad, self-contained search query targeting its primary contribution. The prompt constrains query scope, length, phrasing, and leakage from titles or implementation details.

  • Each held-out paper is converted into exactly one search-style query directly answered by that paper.
  • The preparation prompt forbids references to titles, authors, or implementation details that could leak ground truth.
  • The query expresses a single information need and targets the paper’s primary research problem or main contribution rather than secondary details.
  • Queries are broad enough for scholarly search, concise at ideally 8–25 words, and phrased with varied natural interrogatives.
  • Datasets, benchmarks, model names, and algorithms are included only when central to the research question.

Effect of Exploration Hop

Crase uses a 1.5-hop neighborhood that preserves relations among retrieved neighbors, capturing additional connectivity without the much larger workload of full two-hop expansion. The seed budget further balances candidate coverage against graph-construction and entailment cost.

  • Exploration radius: The 1.5-hop graph contains the same nodes as one-hop expansion but additionally retains citation relations among retrieved neighbors.These relations connect candidate papers to seeds and to one another for subsequent ranking.
  • Exploration radius: A full two-hop expansion yields only a small additional recall gain while increasing candidate-graph and entailment workload by an order of magnitude.
  • Exploration radius: The one-hop graph is cheaper but discards neighboring-paper relations, reducing Recall@50.
  • Seed budget: Increasing the seed budget improves coverage, but gains diminish beyond m = 10 while graph-construction and entailment costs continue growing.The experiments keep L = 5 fixed while varying the maximum seeds retained per subquery.
  • Seed budget: Crase uses m = 10 as a practical trade-off between retrieval quality and the size of the bounded evidence graph.
  • Claim representation: Claim extraction represents papers with small sets of atomic, typed claims anchored to source phrases for direct inspection.The representation can include contextual, methodological, empirical, and other contribution-level claims.

Error Analysis: Seed Corruption and Error

Crase contains errors when corrupted seeds remain near the relevant citation region, but bounded exploration cannot recover from seeds whose neighborhoods miss the ground truth. The stress test therefore separates local error containment from complete off-manifold failure.

  • Stress-test setup: Crase tests seed corruption by replacing clean seeds with lexically similar distractors or random irrelevant papers while leaving Stages 2–3 unchanged.The fixed components include 1.5-hop expansion, coverage-affinity pruning with threshold τ, and PPR ranking.
  • Near-manifold corruption: 56%: Lexically similar distractors reduce Recall@50 from 0.3636 to 0.2045, while preserving ∼81% of clean MAP@50 (0.0185 vs. 0.0228).Their citation neighborhoods can still intersect the relevant region, and unsupported edges receive ∆ = 0 and are pruned.
  • Near-manifold corruption: Recall@50 retains 56% versus 71% at Recall@20, showing that distractors mainly remove the harder-to-reach tail of the evidence set.
  • Off-manifold corruption: Random irrelevant seeds collapse recall to 0 at every cutoff when their citation neighborhoods never reach the ground truth.The resulting empty relevant neighborhood is inspectable before ranking.
  • Operating envelope: Crase contains near-region errors through pruning but cannot recover from errors that move exploration entirely off the citation manifold.This is the trade-off between bounded cost and inspectable evidence versus adaptive re-exploration.

Frequently Asked Questions about Crase

Crase constrains scholarly exploration to an explicit citation neighborhood and makes evidence-selection decisions inspectable, while claim-level pruning and recency-aware ranking address limitations of raw citation graphs. This bounded design trades adaptive recovery outside the initial neighborhood for a fixed candidate boundary and stopping rule.

  • Structural boundedness: Crase fixes the exploration boundary after seed construction, preventing later model outputs from issuing searches or recursively enlarging the candidate set.The search boundary, evidence-selection operations, and stopping rule remain explicit and inspectable.
  • Evidence selection: Claim-level evidence evaluates whether citation edges support contribution-level claims rather than relying on document similarity or binary citation links.This distinguishes topical, methodological, historical, and substantive citations within the graph.
  • Evidence selection: Coverage affinity measures the fraction of a citing paper’s claims receiving support, avoiding dominance by a single highly related claim.This favors references grounding several principal claims while allowing different cited-paper claims to support different citing-paper claims.
  • Graph construction: The 1.5-hop neighborhood expands each seed’s references and citing papers while retaining relations among retrieved nodes, exposing richer local structure without recursively adding another paper layer.It provides more evidence paths than a strict one-hop graph while keeping the candidate bound.
  • Ranking: Recency weighting corrects citation-network age bias as a soft adjustment: strong evidence remains necessary, while older evidence is down-weighted rather than discarded.Recency cannot rescue an unsupported edge because its weight remains controlled by coverage affinity.
  • Trade-offs: The main cost of structural boundedness is that relevant papers outside the seed-induced citation neighborhood cannot be recovered through later query reformulation.Crase exchanges adaptive exploration for an explicit candidate boundary and termination rule.
Loading 2608.24809v1…