Source-linked AI summary
Hallucination-Aware Multi-Agent RAG via Internal Attention Rewards
Runsong Jia, Qian Chang, Mengjia Wu, Ciprian Doru Giurcăneanu, Jie Lu, Yi Zhang
TL;DR
RAG can produce claims unsupported by retrieved evidence, while shared task rewards measure answer overlap rather than faithfulness. The paper trains cooperative RAG agents with a shared task reward augmented by a frozen, TruthfulQA-calibrated attention-based risk critic. Across three backbones and three benchmarks, it reports improved task performance and reduced hallucination over the strongest multi-agent baseline.
Problem
RAG outputs can contain unsupported claims, and shared task rewards such as EM or F1 measure answer overlap rather than faithfulness to evidence.
Source: §1 · p. 1Method
Three role-specific LoRA agents for query planning, evidence selection, and answer generation share task reward augmented with a frozen critic’s attention-based hallucination-risk score.
Source: §1 · p. 2, §2.3 · p. 3, §2.4 · p. 5Results
Takeaways & Limitations
A TruthfulQA-calibrated internal risk signal can complement task supervision as reward shaping across RAG tasks.
Source: §1 · p. 2, §4 · p. 8The study uses attention-head norm features and keeps the critic frozen during policy optimization.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) grounds language models in retrieved evidence, but its outputs often contain claims unsupported by that evidence. Recent multi-agent RAG systems decompose the pipeline into cooperative decisions, yet they typically optimize all agents under a shared task reward such as EM or F1, which measures answer overlap rather than faithfulness to evidence. We propose a hallucination-aware multi-agent RAG framework that combines the task reward with an internal hallucination-risk signal computed from the generator's attention-head norms. Three role-specific LoRA adapters, a query planner, an evidence selector, and an answer generator, share a frozen backbone. A lightweight critic is trained once on TruthfulQA truthful/false pairs and then kept frozen, providing a complementary risk score that is added to the shared reward during cooperative PPO training. Across three backbones (Qwen3-8B, Llama3-8B, Llama2-13B) and three benchmarks (RAGTruth, HotpotQA, 2WikiMultiHopQA), our method improves task performance and reduces hallucination rates over the strongest multi-agent baseline. Ablations and critic diagnostics show that the TruthfulQA-calibrated signal provides useful reward shaping that transfers across RAG tasks.
1 Introduction
RAG can produce unsupported claims even when it answers correctly, while task-only rewards prioritize answer overlap over evidence faithfulness. The paper proposes a shared multi-agent reward that adds an internal attention-based risk signal to task reward.
1 Introduction
- RAG answers can name the correct entity yet add unsupported relations, dates, or numbers, and failures may originate at multiple pipeline stages.Ambiguous questions, noisy or partially relevant passages, and generation itself can all contribute.Source: §1 · p. 1
- Shared task rewards such as EM or F1 favor answer overlap, so unsupported claims may be barely penalized even when the answer matches the reference.A correct answer can still add details absent from the evidence.Source: §1 · p. 1, Figure 1 · p. 2
- Existing faithfulness signals assess the final answer or steer only the generator, while a decoding-time internal signal could provide a cheaper dense reward and address planning and evidence selection too.Reference-based metrics and LLM-as-a-judge approaches evaluate the surface answer; the latter requires an external call for every rollout.Source: §1 · pp. 1–2
- The framework decomposes RAG into query planning, evidence selection, and answer generation, using role-specific LoRA adapters and a shared reward combining task correctness with an internal faithfulness signal.A lightweight critic over attention-head norms is trained on TruthfulQA truthful/false pairs, kept frozen, and its risk score is combined with task reward during cooperative PPO training.Source: §1 · p. 2
- Across three backbones and three QA benchmarks, the method improves task metrics and reduces hallucination rates over the strongest multi-agent baseline.The critic is calibrated on TruthfulQA and transferred across RAG benchmarks without retraining.Source: §1 · p. 2
2 Learning Setup
The learning setup decomposes RAG into three cooperative decisions and combines task correctness with an internal attention-based faithfulness signal. A frozen backbone and critic support role-specific policies trained with shared reward feedback.
2 Learning Setup
- Three role-specific LoRA adapters on a frozen backbone implement query planning, evidence selection, and answer generation, with each stage optimized as an explicit decision.The critic’s risk estimate complements the task reward as a reward-shaping term.Source: §2 · p. 2
2.1 Problem Formulation
- The objective pairs reference correctness with evidence faithfulness, since an answer can overlap with a reference while containing unsupported claims.The question and evidence pool condition a natural-language answer.Source: §2.1 · pp. 2–3
2.2 Framework Overview
- In a fixed-order cooperative episode, the planner rewrites the query, the selector chooses a subset of evidence, and the generator produces the answer.All agents share one terminal reward; the frozen backbone and critic preserve the critic’s calibrated feature space during RL.Source: §2.2 · p. 3
2.3 Multi-Agent RAG Policies
- The planner clarifies entities, constraints, or sub-questions without triggering retrieval; the selector chooses conditioning evidence, and only the generator produces the free-form answer.Although only the generator emits the answer, all three agents receive the shared reward, shaping their intermediate decisions toward correct, faithful answers.Source: §2.3 · p. 3
2.4 Attention-Based Hallucination Critic
- The frozen critic maps generator-side attention-head features to a hallucination-risk score, using truthfulness-calibrated head norms and a lightweight MLP.Heads are selected using a TruthfulQA-derived calibration set; the critic assigns higher risk to false answers and lower risk to truthful ones, then remains fixed during RL.Source: §2.4 · pp. 3–5
2.5 Hallucination-Aware Multi-Agent RL
- Cooperative PPO gives all three agents the same terminal reward, combining task reward with the frozen critic’s hallucination penalty, while updating only their role-specific adapters.A supervised warm start teaches valid query plans, evidence selections, and answers before RL refines the policies.Source: §2.5 · p. 5
3 Experiments
Across three QA benchmarks and three backbones, the method improves task metrics and reduces unsupported answers relative to the strongest multi-agent baseline. Ablations and critic diagnostics examine the reward signal and the attention-head features it uses.
3.1 Experimental Setup
- The evaluation uses fixed evidence pools across RAGTruth, HotpotQA, and 2WikiMultiHopQA with Qwen3-8B, Llama3-8B, and Llama2-13B, reporting task metrics and RAGTruth AHR.The fixed-evidence setting focuses comparisons on policy learning rather than retriever variability.Source: §3.1 · p. 6
3.3 Faithfulness Analysis
- On RAGTruth, AHR is lowest across all three backbones, with absolute reductions over MMOA-RAG of 10.74 points for Qwen3-8B, 5.62 for Llama3-8B, and 3.34 for Llama2-13B.AHR measures the fraction of responses containing unsupported claims; task performance improves in parallel.Source: §3.3 · p. 6, Table 2 · p. 7
3.4 Ablation Studies
- Removing the critic lowers average R/A by 3.31 points and F1 by 2.59; freezing any role-specific adapter also degrades performance, with the generator update having the largest effect.In these role ablations, the frozen adapter remains in the inference pipeline at its SFT initialization while the other agents continue updating.Source: §3.4 · pp. 6–7
3.2 Main Results
- On Qwen3-8B, the method gains +2.35 R/A, +0.95 EM, and +1.29 F1 over MMOA-RAG; it also leads on average across all three metrics for Llama3-8B and Llama2-13B.Across datasets, reported gains include +2.07 Recall on RAGTruth, +2.22 Accuracy and +2.34 EM on HotpotQA, and +2.76 Accuracy and +1.89 EM on 2WikiMultiHopQA.Source: §3.2 · p. 6, Table 1 · p. 7
3.5 Critic Diagnostics and Head Analysis
- The held-out TruthfulQA critic reaches 75.92% AUROC and produces actionable, non-saturated risk scores in RAG optimization, supporting its use as an auxiliary reward rather than a standalone detector.The critic is calibrated on TruthfulQA truthful/false pairs and kept frozen during RAG policy optimization.Source: §3.5 · pp. 7–8
- Truthfulness discrimination is concentrated in a subset of attention heads spanning multiple layers, supporting selection of the strongest truth-correlated heads as critic inputs.Figure 4 visualizes the stronger of two head-level vote accuracies in each cell.Source: §3.5 · p. 8, Figure 4 · p. 8
- Selected head norms change locally near answer-salient tokens, including entity mentions and factual predicates, suggesting sensitivity to content-bearing answer positions.Figure 5 shows token-wise trajectories for representative selected heads.Source: §3.5 · p. 8, Figure 5 · p. 8
4 Conclusion
The conclusion presents internal attention-head diagnostics as an auxiliary reward for cooperative multi-agent RAG and reports improvements across three benchmarks and three backbones, with reduced hallucination on RAGTruth.
4 Conclusion
- The framework combines task performance with a frozen critic’s hallucination-risk estimate to train role-specific agents for query planning, evidence selection, and answer generation.The conclusion describes the internal attention-head signal as auxiliary to reinforcement learning, not a replacement for task supervision.Source: §4 · p. 8
- Experiments show consistent improvements across RAGTruth, HotpotQA, and 2WikiMultiHopQA on Qwen3-8B, Llama3-8B, and Llama2-13B, while reducing answer hallucination on RAGTruth.The conclusion does not provide metric values in these passages.Source: §4 · p. 8
- The authors conclude that lightweight model-internal signals can shape RAG policies toward more faithful generation when used alongside task supervision.They propose combining external evidence supervision with model-internal diagnostics as a direction for more reliable RAG systems.Source: §4 · p. 8
Limitations
The work uses attention-head norms as a lightweight, stable internal signal and keeps its critic frozen during policy optimization; richer diagnostics and adaptive critic updates remain future directions.
Limitations
- The critic uses attention-head norm features as its internal signal, making the reward inexpensive to compute during reinforcement learning.The authors identify hidden-state dynamics, MLP activations, and structured attention patterns as possible richer diagnostics for future study.Source: Limitations · p. 9
- The critic remains frozen during policy optimization to preserve a fixed reward function and avoid changing internal-feature calibration as agents update.Source: Limitations · p. 9
- The framework evaluates a lightweight, stable instantiation; whether richer diagnostics or adaptive critic updates can help while maintaining training stability remains open.Source: Limitations · p. 9
A.1 Related Work
The related-work discussion organizes the paper around three research lines and two questions: where supervision comes from and when it is applied.
A.1 Related Work
- The paper draws on research about faithfulness in RAG systems, reinforcement learning for cooperative LLM agents, and internal signals for truthfulness.Source: §A.1 · p. 11
- The discussion distinguishes supervision from external text versus internal representations.Source: §A.1 · p. 11
- It also distinguishes inference-time correction from training-time reward shaping as points of comparison across the research.Source: §A.1 · p. 11
A.1.1 Faithfulness and Multi-Agent Optimization in RAG
Prior work addresses RAG faithfulness through inference-time correction and training, while another line decomposes RAG into learnable cooperating stages. The paper builds on multi-agent optimization and asks whether internal signals can complement text-level rewards.
A.1.1 Faithfulness and Multi-Agent Optimization in RAG
- Inference-time approaches include Self-RAG’s reflection tokens for adaptive retrieval and self-evaluation (Asai et al., 2024), CRAG’s evaluator and web-search fallback (Yan et al., 2024), and RefChecker’s claim-triplet verification.Source: §A.1.1 · p. 11
- Training approaches include RAG-HAT’s DPO over faithful/unfaithful pairs and RAFT’s targeted fine-tuning for domain-specific RAG (Zhang et al., 2024).Source: §A.1.1 · p. 11
- Query rewriting, iterative retrieval-reasoning, and adaptive retrieval treat individual RAG stages as learnable (Mao et al., 2024; Trivedi et al., 2023; Jeong et al., 2024).Source: §A.1.1 · p. 11
- Multi-agent and related methods optimize retrieval and generation, with shared F1-based rewards in MMOA-RAG (Chen et al., 2026), ranking and generation in RankRAG (Yu et al., 2024b), and reading notes in Chain-of-Note (Yu et al., 2024a).The paper builds on multi-agent decomposition and PPO, examining internal-computation rewards as a complement to text-level objectives.Source: §A.1.1 · p. 11
A.1.2 Reinforcement Learning for LLM Alignment
Prior alignment work mainly uses text-domain rewards, while this paper explores internal representations as a complementary training signal.
A.1.2 Reinforcement Learning for LLM Alignment
- Prior RL alignment methods use preferences, ratings, or match scores as text-domain rewards for broad behavioral targets.The section cites PPO, DPO, GRPO, Constitutional AI, and RAG-DDR among this line of work.Source: §A.1.2 · p. 11
- PPO is retained as the training machinery while the paper investigates supervision from policy internal representations during generation.Source: §A.1.2 · p. 11
- DPO directly optimizes preference pairs, whereas GRPO estimates baselines from group statistics without a separate value network.Source: §A.1.2 · p. 11
A.1.3 Internal Signals for Truthfulness
Prior work indicates that internal model states carry truthfulness information; this paper investigates using that information as a training-time reward.
A.1.3 Internal Signals for Truthfulness
- Prior studies use internal truthfulness signals either for post-hoc hallucination detection or to steer activations during inference.The cited examples include MIND and INSIDE for detection, and ITI for inference-time steering.Source: §A.1.3 · p. 12
- NoVo uses majority voting over attention-head norms to achieve strong TruthfulQA accuracy, while other cited work extracts latent knowledge from activations without supervision.Source: §A.1.3 · p. 12
- The paper explores a third application: using internal signals as a training-time reward to shape policy learning.Source: §A.1.3 · p. 12
A.2 Experimental Details
The appendix details the fixed-evidence evaluation, training setup, metrics, and baselines, then reports critic-transfer diagnostics and illustrative pipeline materials.
A.2.1 Datasets Details
- The evaluation spans RAGTruth, HotpotQA, and 2WikiMultiHopQA under fixed evidence pools, isolating policy learning from retriever variability.RAGTruth focuses on faithfulness; the other two benchmarks require multi-hop reasoning over supporting evidence.Source: §A.2.1 · p. 12
A.2.2 Implementation Details
- Three role-specific LoRA adapters share frozen backbones and are warm-started with supervised fine-tuning before PPO; a frozen critic maps selected attention-head norm statistics to scalar risk.The critic is trained on TruthfulQA truthful/false pairs.Source: §A.2.2 · p. 12
A.2.3 Evaluation metrics.
- RAGTruth uses Recall, EM, and F1 plus AHR; HotpotQA and 2WikiMultiHopQA use Accuracy, EM, and F1, with AHR measuring answers containing unsupported claims.Source: §A.2.3 · pp. 12–13
A.2.4 Details about Baseline Models
- Baselines range from vanilla RAG and self-reflective systems to multi-agent methods, including MMOA-RAG, which jointly optimizes planning, selection, and generation with F1 rewards.The ablation study keeps the Q–E–G inference pipeline intact while freezing a named agent's adapter during PPO.Source: §A.2.4 · p. 13, §A.3.1 · p. 13, Table 3 · p. 14
A.3.2 Additional Critic Transfer Diagnostics
- 75.92% AUROC on held-out TruthfulQA pairs indicates a truthfulness signal in selected head-norm features; the frozen critic is used as a reward-shaping term for RAG policy optimization.The diagnostics characterize the critic as a weak complementary signal rather than a standalone RAG hallucination judge.Source: §A.3.2 · pp. 13–14
- An evidence-aware critic reaches 74.05% AUROC on held-out RAGTruth data; critic-score correlations with answer length and answer–evidence overlap are −0.1103 and −0.1077.Two independent judges produce similar AHR scores, with trends consistent across model families.Source: §A.3.2 · pp. 13–14, Table 4 · p. 14
- Among tested settings, λ = 0.08 gives the strongest RAGTruth Recall, EM, and F1; calibrated critic scores outperform shuffled scores that break sample-level correspondence.The diagnostics conclude that the critic is useful as calibrated reward shaping, with penalty weight balanced against task reward.Source: §A.3.2 · p. 14, Table 5 · p. 14
A.4 Prompt Templates
- Shared prompt templates fill question and document slots at inference, and Agent E also receives Agent Q's planned sub-questions as a soft conditioning signal.The authors state that LLM use was limited to grammar and spelling checks.Source: §A.4 · p. 14, §A.6 · p. 15, Figure 6 · p. 16