PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Context Fidelity via Native Retrieval-Augmented Reasoning

Suyuchen Wang, Jinlin Wang, Xinyu Wang, Shiqi Li, Xiangru Tang, Sirui Hong, Xiao-Wen Chang, Chenglin Wu, Bang Liu

arXiv:2509.13683v1cs.CLcs.AI

TL;DR

LLMs may contradict or fabricate information when answering from provided context, despite the importance of context fidelity. CARE integrates in-context evidence retrieval into reasoning, and the authors report that it consistently outperforms existing approaches on answer generation and evidence extraction across general and counterfactual QA benchmarks.

  • Problem

    LLMs can produce inconsistent answers that contradict or fabricate information relative to the provided context, reducing user trust and practical utility.

    Source: §1 · p. 1

  • Method

    CARE teaches LLMs to identify relevant evidence in the input context and integrate it directly into their reasoning process.

    Source: §1 · p. 2

  • Results

    CARE consistently outperforms existing approaches, including vanilla SFT and traditional RAG, in answer generation and evidence extraction across multiple general and counterfactual QA benchmarks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    CARE is a step toward more reliable AI systems that make better use of available context without expensive retrieval infrastructure.

    Source: §6 · p. 9

    CARE cannot access external knowledge beyond the provided context and may need to be combined with external retrieval for questions requiring absent information.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) often struggle with context fidelity, producing inconsistent answers when responding to questions based on provided information. Existing approaches either rely on expensive supervised fine-tuning to generate evidence post-answer or train models to perform web searches without necessarily improving utilization of the given context. We propose CARE, a novel native retrieval-augmented reasoning framework that teaches LLMs to explicitly integrate in-context evidence within their reasoning process with the model's own retrieval capabilities. Our method requires limited labeled evidence data while significantly enhancing both retrieval accuracy and answer generation performance through strategically retrieved in-context tokens in the reasoning chain. Extensive experiments on multiple real-world and counterfactual QA benchmarks demonstrate that our approach substantially outperforms supervised fine-tuning, traditional retrieval-augmented generation methods, and external retrieval solutions. This work represents a fundamental advancement in making LLMs more accurate, reliable, and efficient for knowledge-intensive tasks.

1 Introduction

LLMs can contradict or fabricate information in their answers despite relevant information being present in the input context, undermining user trust and practical utility. CARE integrates retrieved in-context evidence into reasoning and substantially outperforms vanilla SFT, traditional RAG, and comparable models without in-context retrieval on evidence retrieval and answer accuracy.

  • Context hallucinations are especially consequential in knowledge-intensive tasks, where contradictions or fabricated answers reduce user trust and practical utility.Source: §1 · p. 1
  • Prior RAG requires extensive labeled evidence and extra retrieval infrastructure, while external retrieval may underuse user context and add latency, complexity, and inconsistencies.Source: §1 · pp. 1–2
  • Native retrieval-augmented reasoning teaches LLMs to identify and integrate evidence from the input context directly within their reasoning chain.It uses the model's language understanding without additional indexing or embedding systems.Source: §1 · p. 2
  • CARE uses SFT to establish evidence integration, then RL with retrieval-aware rewards to refine self-retrieval; curriculum learning extends to more complex tasks without additional labeled data.Source: §1 · p. 2
  • The authors present and open-source a curated dataset for training models to perform evidence-integrated reasoning.Source: §1 · p. 2
  • Across real-world and counterfactual QA benchmarks, CARE substantially outperforms vanilla SFT, traditional RAG, and comparable models without in-context retrieval in evidence retrieval and answer accuracy.Source: §1 · p. 2

2 Related Work

Prior work explores prompting strategies for QA reasoning, retrieval-augmented generation, and reinforcement-learning-based retrieval. These lines of work address reasoning and retrieval, but the cited approaches retain challenges around long or noisy inputs, retrieval pipelines, or separating retrieval from reasoning.

2.1 LLM Reasoning on Question-Answering Tasks

  • Prompting approaches include chain-of-thought and structured reasoning methods, yet LLMs still struggle with context coherence on long or noisy inputs.Source: §2.1 · p. 2

2.2 Retrieval-Augmented Generation

  • Traditional RAG retrieves from external corpora, while in-context retrieval reuses input spans; both rely on indexing and embedding pipelines that limit adaptation to complex or evolving contexts.Source: §2.2 · pp. 2–3

2.3 RL-Enhanced LLM Retrieval

  • RL-based retrieval can learn adaptive, task-reward-driven policies, but most existing approaches keep retrieval separate from reasoning, potentially limiting context-aware integration.Source: §2.3 · p. 3

3 The CARE Method

CARE integrates retrieval into reasoning over the supplied context, using supervised fine-tuning (SFT) to establish evidence integration and reinforcement learning (RL) to refine self-retrieval. Its formulation targets questions requiring identification and use of sparse information in long contexts.

3.1 Overview

  • CARE combines native retrieval from the input context with reasoning, integrating retrieved evidence without external retrieval modules or tools.The framework uses an SFT phase followed by RL to improve context loyalty, retrieval accuracy, and answer correctness in QA tasks.Source: §3.1 · p. 3

3.2 Problem Formulation

  • The target problem maps a query and long context containing sparse relevant information to an answer.The formulation focuses on identifying and using key information in lengthy contexts.Source: §3.2 · p. 3

3.3 The Supervised Fine-Tuning Phase

  • SFT data generation creates reasoning chains, integrates labeled supporting facts, and inserts retrieval markers around evidence spans.The SFT phase uses existing QA data with supporting facts to familiarize the model with evidence-integrated output before RL.Source: §3.3 · pp. 3–5

3.4 Reinforcement Learning Phase

  • RL uses GRPO and retrieval-aware rewards to refine self-retrieval and encourage evidence consistency and logical coherence.The reward combines answer accuracy, output structure, and context fidelity; retrieval reward accepts marked text that occurs in the context without requiring ground-truth retrieval data.Source: §3.4 · p. 5
  • The RL curriculum transitions from short-context, short-answer QA to long-context, multihop, long-answer QA.Training begins with the easier dataset and gradually incorporates harder examples, a progression described as mitigating catastrophic forgetting while enhancing retrieval in increasingly complex tasks.Source: §3.4 · p. 6

4 Experiment Settings

Experiments test CARE on real-world long-context and counterfactual QA across multiple LLM families and sizes, using LongBench and CofCA and comparisons with original models, retrieval methods, and online-search approaches.

4.1 Datasets, Benchmarks and Metrics

  • Training uses 7,739 HotpotQA-derived instances with labeled retrieval-augmented reasoning chains for SFT, while DROP and MS MARCO supply RL training data.The evaluation includes LongBench single- and multi-passage QA datasets with F1 scores, plus CofCA counterfactual Wikipedia snippets evaluated with F1.Source: §4.1 · p. 6
  • Evaluation covers MultiFieldQA-En, HotpotQA, 2WikiMQA, and MuSiQue from LongBench, plus CofCA snippets that test adherence when context contradicts parametric knowledge.F1 is reported for the LongBench datasets and CofCA.Source: §4.1 · pp. 6–7

4.2 Models and Baselines

  • The study compares Qwen-2.5 Instruct 7B and 14B and LLaMA-3.1 8B against original models, RL-based online retrieval methods, and CRAG.Available checkpoints differ by baseline: ReSearch and R1-Searcher cover specified model subsets, while CRAG has Qwen2.5 7B and 14B checkpoints.Source: §4.2 · p. 7

4.3 Experiment Settings

  • SFT uses LLaMA-Factory’s default LoRA setting, and RL uses verl’s default GRPO setting; experiments run on 8×A800-SXM4-80GB or 8×H100 80GB.Curriculum RL uses λ1 = 0.7, λ2 = 0.1, λ3 = 0.2, and η = 1.Source: §4.3 · p. 7

5 Results and Analysis

CARE improves QA performance across real-world and counterfactual benchmarks, with evidence retrieval scores also highest across the tested settings. Ablations indicate contributions from RL, native in-context reasoning, and curriculum learning.

5.1 Question-Answering Performance

  • +15.29% average F1 over the original LLaMA-3.1 8B model, including +29.42% on 2WikiMQA and +18.92% on MuSiQue; CARE remains competitive on Qwen despite two dataset exceptions.The reported strongest gains are on multi-hop tasks.Source: §5.1 · p. 7

5.2 Counterfactual QA Performance

  • +13.69% on CofCA with LLaMA-3.1 8B; CARE delivers the strongest counterfactual QA performance, while online search methods underperform original models.The passage suggests external retrieval can be counterproductive when context contradicts parametric knowledge.Source: §5.2 · p. 7

5.4 Evidence Retrieval Evaluation

  • CARE achieves the highest BLEU and ROUGE-L scores in every tested setting, and its relative improvement remains consistent across model scales and families.Scores scale with model size, with Qwen2.5 14B showing the strongest results.Source: Figure 3 · p. 8, §5.4 · pp. 8–9

5.3 Ablation Studies

  • SFT alone offers marginal gains, whereas RL substantially improves QA reasoning; native in-context reasoning variants outperform the R1-like GRPO setting without retrieval reward.The comparison is consistent with retrieval-augmented reasoning grounding answers in contextual evidence.Source: §5.3 · p. 8
  • Curriculum learning improves balance across QA types, especially long-form MFQA and counterfactual CofCA, without additional labeled data beyond initial SFT.The no-curriculum variant performs well on multi-hop datasets, while curriculum supports broader task balance.Source: §5.3 · p. 8

6 Conclusion

CARE teaches LLMs to retrieve and integrate evidence from the input context during reasoning, improving context fidelity with limited labeled evidence. Across general and counterfactual QA benchmarks, it outperforms vanilla SFT and traditional RAG in answer generation and evidence extraction.

6 Conclusion

  • CARE dynamically identifies and integrates in-context evidence within reasoning, improving context fidelity while requiring limited labeled evidence.The conclusion frames the approach as making better use of available context without expensive retrieval infrastructure.Source: §6 · p. 9

Limitations

CARE is limited to information in the provided context, and its evaluation scope and resistance to hallucinations remain bounded by the tested settings and input conditions.

Limitations

  • CARE cannot access knowledge beyond the provided context; cases needing absent information would require combining it with external retrieval, potentially complicating the architecture.Source: Limitations · p. 9
  • Evaluation primarily covers multi-hop, general-domain reasoning; effectiveness on abstract reasoning, numerical computation, creative generation, and domain-specific tasks remains uninvestigated.Source: Limitations · p. 9
  • CARE does not eliminate hallucinations, especially when input information is ambiguous or contradictory.Source: Limitations · p. 9

Ethical Considerations

CARE improves fidelity to supplied context, but does not guarantee factual accuracy when that context is inaccurate or contradictory. The authors also note environmental and bias concerns associated with training and deployment.

Ethical Considerations

  • CARE cannot guarantee factual accuracy when the input contains inaccuracies or contradictions.The authors identify this as a boundary of improved fidelity to the provided context.Source: Ethical Considerations · p. 9
  • Training uses computational resources with an acknowledged environmental impact.Source: Ethical Considerations · p. 9
  • Models may inherit training-data biases despite improved context fidelity, so the authors recommend fairness evaluations before sensitive deployments.Source: Ethical Considerations · p. 9

A Token Cost Efficiency Analysis

The token-cost analysis considers average output tokens per query across real-world QA datasets. CARE produces longer reasoning-chain outputs but avoids the external API and database-retrieval overhead required by baseline methods.

A Token Cost Efficiency Analysis

B.1 Implementation Details

The appendix reports implementation settings, prompts for evidence-integrated reasoning, and a case-study table documenting CARE’s HotpotQA examples.

B.1 Implementation Details

  • SFT runs for 3 epochs, while GRPO training runs for 350 steps across 8 GPUs on one node.The implementation passage also specifies optimizer, batch-size, and hardware settings.Source: §B.1 · p. 13

C System Prompts

  • The prompts require reasoning to identify needed context information and integrate evidence, with retrieved spans marked inside the reasoning process.Source: C System Prompts · p. 13

D Case Study

  • The HotpotQA case-study table includes questions, answers, supporting facts, predictions, reasoning chains, retrieval spans, and F1-based correctness.Source: Table 5 · p. 14
Loading 2509.13683v1…