Source-linked AI summary
Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities
Shuangshuang Ying, Zheyu Wang, Yunjian Peng, Jin Chen, Yuhao Wu, Hongbin Lin, Dingyu He, Siyi Liu, Gengchen Yu, YinZhu Piao, Yuchen Wu, Xin Gui, Zhongyuan Peng, Xin Li, Xeron Du, Libo Qin, YiXin Cao, Ge Zhang, Stephen Huang
TL;DR
Existing evaluations do not cleanly determine whether models can reason over novel scientific information because retrieval, reasoning, memorization, and web volatility are entangled. DeR2 introduces a controlled benchmark that separates evidence access from reasoning through four regimes and validated frozen document sets. Across models, it reveals mode-switch fragility and procedural concept misuse, showing substantial remaining headroom for retrieval-infused reasoning.
Problem
Existing deep-research benchmarks conflate retrieval, reasoning, toolchain choices, parametric memorization, and open-web volatility, leaving novel scientific reasoning difficult to isolate.
Method
DeR2 decouples evidence access from reasoning with four controlled regimes, two-phase verifiability checks, and frozen document libraries containing relevant evidence, distractors, expert concepts, and validated rationales.
Results
Across strong commercial and open models, DeR2 reveals mode-switching failures, procedural concept failures, multi-concept coordination bottlenecks, and nonlinear noise effects.
Takeaways & Limitations
Robust retrieval-infused reasoning requires evidence-grounded mode control, executable concept use, and adaptive hypothesis management under noisy contexts.
Takeaways & Limitations
Final instruction–answer pairs require human authorship because AI assistance can select peripheral answers or produce overly open-ended questions.
Abstract
from arXiv · showhide
Despite strong performance on existing benchmarks, it remains unclear whether large language models can reason over genuinely novel scientific information. Most evaluations score end-to-end RAG pipelines, where reasoning is confounded with retrieval and toolchain choices, and the signal is further contaminated by parametric memorization and open-web volatility. We introduce DeR2, a controlled deep-research sandbox that isolates document-grounded reasoning while preserving core difficulties of deep search: multi-step synthesis, denoising, and evidence-based conclusion making. DeR2 decouples evidence access from reasoning via four regimes--Instruction-only, Concepts (gold concepts without documents), Related-only (only relevant documents), and Full-set (relevant documents plus topically related distractors)--yielding interpretable regime gaps that operationalize retrieval loss vs. reasoning loss and enable fine-grained error attribution. To prevent parametric leakage, we apply a two-phase validation that requires parametric failure without evidence while ensuring oracle-concept solvability. To ensure reproducibility, each instance provides a frozen document library (drawn from 2023-2025 theoretical papers) with expert-annotated concepts and validated rationales. Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom: some models exhibit mode-switch fragility, performing worse with the Full-set than with Instruction-only, while others show structural concept misuse, correctly naming concepts but failing to execute them as procedures.
1 Introduction
DeR2 addresses limitations in deep-research evaluation by separating retrieval access from evidence-grounded reasoning under controlled conditions. It also provides a frozen, curated benchmark and exposes distinct model failure patterns.
- Motivation: Existing deep-research benchmarks conflate retrieval, toolchain, and reasoning capabilities, limiting diagnosis of errors such as concept omission, concept misuse, and noise-induced derailment.End-to-end scores provide limited guidance for model selection, debugging, or targeted training.
- Benchmark design: DeR2 evaluates the same question under Instruction-only, Concepts, Related-only, and Full-set regimes to isolate retrieval and reasoning losses.The regimes progressively add oracle concepts, relevant evidence, and topically related distractors.
- Benchmark design: A two-phase validation protocol requires parametric failure without evidence while preserving solvability with oracle concepts, reducing memorization leakage.This jointly enforces novelty and tractability.
- Benchmark design: DeR2 uses frozen per-instance document libraries drawn from 2023–2025 theoretical papers, with relevant evidence, distractors, expert concepts, and validated rationales.The frozen libraries address open-web volatility and support reproducible evaluation.
- Findings: Some models show mode-switch fragility, performing better with Instruction-only than Full-set, while others identify correct concepts but fail to execute them procedurally.These failures involve disruption from added documents and replacement of constructive mechanisms with generic heuristics.
2 Benchmark
DeR2 benchmarks scientific problem solving as concept-grounded, multi-step reasoning over controlled document sets. Its construction combines expert annotation, frontier-paper sourcing, difficulty calibration, and scientific quality review.
- Benchmark definition: Each DeR2 example asks a frontier scientific question whose answer requires selecting, instantiating, and coordinating multiple concepts from provided documents.The target is research-style reasoning rather than definition recall or surface-level summarization.
- Benchmark definition: A benchmark item contains an Instruction, Concepts, chain-of-thought rationale, Answer, and a document set containing necessary evidence and on-topic noise.Concepts represent generalizable theoretical knowledge rather than overly specific experimental data.
- Data collection: Annotators construct scoped instruction–answer pairs and complete concept sets and step-by-step rationales from eligible theoretical source papers.Human authorship is required for final instructions and answers because AI tools can select peripheral quantities or generate overly open-ended questions.
- Difficulty calibration: Difficulty calibration runs each condition three times, requiring all Instruction-only attempts to be incorrect and at least one Concepts-only attempt to be correct.The protocol also requires at least one incorrect Concepts-only attempt and applies additional remedies when all concept trials succeed.
- Quality assurance: Reviewers assess format compliance and scientific validity, including instruction scope, answer verifiability, concept necessity, and coherence of the concept-grounded rationale.Reviewers are selected from annotators with at least three accepted annotations.
3 Experiments
DeR2 evaluates retrieval-infused reasoning through four controlled input configurations, separating concept access, document extraction, denoising, and concept composition. Across models, performance and error analyses reveal substantial losses from retrieval, noise, procedural execution, and multi-concept coordination.
- Evaluation Protocol: The four settings progressively provide no evidence, oracle concepts, relevant documents, or relevant documents with distractors, enabling attribution of distinct capability losses.Instruction-only isolates parametric knowledge; Concepts-only isolates concept-level reasoning; Related-only tests document extraction; Full-set adds denoising.
- Overall Results: Concepts-only averaged 75.39%, Related-only 62.89%, and Full-set 51.25%, showing losses from document concept extraction and distractor handling.Instruction-only averaged 55.89%, compared with Full-set at 51.21% in the reported comparison.
- Overall Results: Instruction-only averaged 55.89% versus 51.21% for Full-set, indicating that external context can shift models into a less effective reasoning regime.The paper reports this recurrent pattern across most models.
- Controlled Factors: Full-set performance declines nonlinearly as noise documents increase, while accuracy also decreases as required concept count grows, even with oracle concepts.The results associate these trends with trajectory drift, working-memory demands, dependency tracking, and coordination burden.
- Failure Modes: Models frequently recognize definitions but fail to execute constructive mechanisms, replacing algorithmic steps or instantiations with familiar heuristics.Such chains can remain locally coherent while producing globally incorrect conclusions.
- Failure Modes: Even when correct concepts are available, models may activate incomplete constraints, apply concepts out of order, or lose intermediate invariants.Concepts-only reduces concept-related errors, but remaining failures are predominantly procedural; some settings also show recurring missing-core-concept errors and noise-induced misalignment.
4 Related Work
Prior benchmarks evaluate reading comprehension, RAG, formal reasoning, or long-context processing, but generally do not isolate document-grounded scientific synthesis under noisy multi-document conditions. DeR2 addresses this gap with controlled regimes that separate retrieval, concept extraction, and reasoning.
- Existing Benchmarks: Early QA benchmarks established extractive passage-based evaluation, while multi-hop variants increased evidence aggregation but remained primarily extractive.Later abstractive variants still tend to favor localized extraction over genuine multi-document synthesis.
- Existing Benchmarks: End-to-end RAG benchmarks measure system-level retrieval and generation, making failures from retrieval, reasoning, and evidence integration difficult to distinguish.Many also emphasize factoid lookup or temporal verification rather than multi-step synthesis.
- Existing Benchmarks: Formal reasoning benchmarks test multi-step deduction largely apart from document comprehension, while graduate-level suites remain predominantly parametric.These settings therefore do not directly test synthesis from novel documents.
- Existing Benchmarks: Long-context and document-level reasoning work does not provide a controlled environment for multi-document synthesis with adversarial distractors.This leaves noisy, realistic evidence selection insufficiently measured.
- DeR2: DeR2 organizes each instance around Instruction, Concepts, CoT, Answer, and Document Set, with two-stage verifiability requiring instruction-only failure but oracle-concept solvability.Its four regimes separately measure parametric knowledge, concept reasoning, weak retrieval, and denoising.
- DeR2: Topically adjacent but concept-irrelevant distractors make adversarial noise an explicit variable targeting mode switching and multi-concept coordination under overload.These failure modes are typically masked by prior benchmark designs.
5 Conclusion
DeR2 is a controlled benchmark that separates retrieval from reasoning through structured documents, verifiability checks, and interpretable loss attribution. Experiments expose recurring failures in mode switching, procedural concept use, coordination, and noise robustness, motivating targeted diagnosis and training.
- Conclusion: DeR2 decouples retrieval from reasoning with a two-stage verifiability protocol and structured document sets containing relevant evidence and distractors.The benchmark attributes performance loss to evidence selection and denoising, document-to-concept extraction, and concept composition or scheduling.
- Conclusion: Across strong commercial and open models, Instruction-only can outperform Full-set, definitions can be recognized without executable mechanisms, and Concepts-only is not an upper bound.The benchmark also reports nonlinear noise effects that alter early reasoning trajectories.
- Conclusion: The results identify evidence-grounded mode control, executable concept utilization, and adaptive hypothesis management under noisy contexts as targets for robust systems.DeR2 is released to support model selection, diagnosis, and future training and evaluation.
Contributions and Acknowledgements
The paper acknowledges Multimodal Art Projection, a donation-funded nonprofit open-source AI research community. Its members work across foundation-model pretraining, large-scale data processing, coding, reasoning, and music generation.
- Acknowledgements: Multimodal Art Projection is a nonprofit open-source AI research community funded by donations.The community covers research and applications spanning foundation models, data processing, coding, reasoning, and music generation.
- Contributions and Affiliations: The listed contributors include researchers from ByteDance Seed, Fudan University, Nanyang Technological University, Harbin Institute of Technology Shenzhen, Peking University, and M-A-P.The passage set lists the individual affiliations across these organizations.
A Prompts
The prompts specify four evaluation functions: answer inference, concept extraction, error-cause analysis, and rubric-based scoring. Their outputs require structured formats, explicit concept matching, constrained error labels, and reasoning-quality assessment.
- The related-document prompt instructs a literature-analysis agent to list and read local Markdown files through fetch_md under specified tool-call constraints.
- Answer-inference prompts separately target Concepts-only and Instruction-only settings, requiring step-by-step reasoning followed by a concise final answer.
- Concept-extraction prompts identify every core concept explicitly used in a model’s chain of thought and compare it with the ground-truth list.They calculate true positives, false positives, and false negatives for necessary concepts.
- Error-cause analysis compares predicted and ground-truth concepts and reasoning, then assigns up to two exact labels when matching quality is insufficient.The permitted labels include reasoning, numerical or formal, missing-concept, and concept-use errors.
- The scoring prompt evaluates factual and logical correctness, completeness, and reasoning quality against the instruction and benchmark answer.It requires a single score between 0 and 1 in an exact final-line format.
B.1 bad case of Claude-Opus-4.1 in Instruction-only
The Claude-Opus-4.1 Instruction-only case replaces the paper’s constructive learning procedure with a sample-complexity template, producing a superficially plausible but structurally incorrect complexity analysis.
- Instruction-only case: The task asks for provable learning of a multi-head attention layer from uniform random labeled examples under non-degeneracy assumptions.The target includes estimates of the projection and attention matrices and mean squared error at most (kd)−Ω(m) with success probability at least 1−δ.
- Model reasoning: The model decomposes the problem into input structure, exponentially small target error, sample complexity, and per-sample computational cost.It treats the required accuracy as the main driver of sample demand.
- Model reasoning: (kd)Ω(m) samples and log(1/δ) are derived from a standard-learning-theory concentration argument based on O(1/ϵ2) samples.
- Model reasoning: The model estimates O(mk2d) per-sample cost for attention computation and assumes moment-based or tensor-decomposition methods requiring O(m)-th order statistics.
- Model conclusion: (kd)O(m) · log(1/δ) is reported as total time complexity, with the dominant term attributed to sample complexity and multi-head learning requirements.
- Error analysis: The error analysis identifies omitted convex-body construction, epsilon-net search, linear-span reduction, and brute-force enumeration, while the reasoning is judged completely deviated.The correct logic instead makes constructive search over an m-dimensional subspace dominant, with an O(m3) exponential-search-related term stated in the reference analysis.
B.2 Bad Case of DeepSeek-V3.1 in Concepts-only
The DeepSeek-V3.1 Concepts-only case applies an incorrect descent-count heuristic to a lifted Schubert-calculus construction, yielding 4 equations instead of the true value 7.
- B.2 Bad Case: The task asks for the number of bilinear equations in SA(u, v, w◦w) for specified permutations in S4.
- Model reasoning: The model computes descent sets for u, v, and w◦w, obtaining sizes 2, 1, and 1 before summing them.It first derives w◦w = 1423 and identifies Des(w◦w) = {2}.
- Model conclusion: 4 is the model’s predicted equation count from adding the three descent-set sizes.
- Method: The lifted formulation converts Schubert-cell membership into bilinear equations using auxiliary variables instead of higher-degree determinantal equations.Its construction is tied to Stiefel coordinates and the algebraic characterization of cell membership.
- Error analysis: The prediction is incorrect: the stated true value is 7 rather than 4.
- Error analysis: The model partially retrieves the lifted formulation and Schubert variety but omits maximal descent, constructive auxiliary-variable details, and the theorem governing cell-membership conversion.Its formula replaces the algebraic-geometric construction with simple addition of descent counts.
B.3 Bad Case of Gemini-3-Pro in Full-set
In this Full-set case, the model followed a vacuum Schwarzschild-like derivation despite the prompt’s externally coupled matter field. Consequently, it omitted the electromagnetic contribution and produced an incomplete A(r) lacking the charge correction.
- The task asks for A(r) in bumblebee theory with an external coupled matter field and radial bumblebee field Bµ = (0, b(r), 0, 0).
- The model instead treated the problem as a spherically symmetric vacuum solution based on a Schwarzschild-like bumblebee result.
- The intended setup includes a non-minimally coupled electromagnetic field with its own Lagrangian and modified Maxwell equations.
- The derivation uses the radial bumblebee constraint, [S(r)A(r)]′ = 0, the choice C1 = 1 + ℓ, and Einstein equations to obtain A(r).
- The final answer A(r) = 1 − 2M/r omits the charge-related correction term 2(1+ℓ)q2 because matter contributions were excluded.