Source-linked AI summary
OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG
Fengran Mo, Zhan Su, Yuchen Hui, Jinghan Zhang, Jia Ao Sun, Zheyuan Liu, Chao Zhang, Tetsuya Sakai, Jian-Yun Nie
TL;DR
RAG answers can suffer when retrieved documents are noisy or vary in relevance, while standard decoding does not explicitly use the retriever’s relevance judgments. OpenDecoder adds external quality indicators to LLM decoding and trains robustness to noisy contexts, outperforming prior approaches across five benchmarks. The framework is also designed to accommodate other external indicators and post-training purposes.
Problem
RAG performance depends on retrieved-document usefulness, but noisy or imperfect context can degrade answers and standard decoding does not explicitly incorporate document relevance.
Method
OpenDecoder constructs explicit quality indicators from retrieved documents, incorporates them into attention computation for decoding, and uses robustness training with varied noisy inputs.
Results
OpenDecoder outperforms vanilla RAG and other strong baselines across five benchmark datasets and diverse noisy environments.
Takeaways & Limitations
Incorporating explicit retrieved-information indicators improves LLM tolerance to noisy RAG contexts, while the paradigm can incorporate other external indicators.
Takeaways & Limitations
The specific quality features and indicators may vary with requirements such as faithfulness or trustworthiness.
Abstract
from arXiv · showhide
The development of large language models (LLMs) has achieved superior performance in a range of downstream tasks, including LLM-based retrieval-augmented generation (RAG). The quality of generated content heavily relies on the usefulness of the retrieved information and the capacity of LLMs' internal information processing mechanism to incorporate it in answer generation. It is generally assumed that the retrieved information is relevant to the question. However, the retrieved information may have a variable degree of relevance and usefulness, depending on the question and the document collection. It is important to take into account the relevance of the retrieved information in answer generation. In this paper, we propose OpenDecoder, a new approach that leverages explicit evaluation of the retrieved information as quality indicator features for generation. We aim to build a RAG model that is more robust to varying levels of noisy context. Three types of explicit evaluation information are considered: relevance score, ranking score, and QPP (query performance prediction) score. The experimental results on five benchmark datasets demonstrate the effectiveness and better robustness of OpenDecoder by outperforming various baseline methods. Importantly, this paradigm is flexible to be integrated with the post-training of LLMs for any purposes and incorporated with any type of external indicators.
1 Introduction
RAG supplements LLMs with retrieved information, but noisy or imperfect documents can degrade answers because decoding relies mainly on internal attention. OpenDecoder incorporates explicit relevance signals into decoding and trains robustness to varied retrieval quality, outperforming baselines across five benchmarks.
- Retrieved information can vary in usefulness, and noisy or defective context may degrade LLM-generated answers.
- Existing workflow-based methods guide LLMs to identify or filter useful retrieved information before generation but retain the standard decoder attention mechanism.
- OpenDecoder modifies LLM decoding by incorporating external relevance signals alongside internally produced attention scores.
- The method constructs relevance indicators, teaches LLMs to use them during answer decoding, and improves robustness by training with documents of varying relevance.
- Five benchmark experiments covering general and multi-hop question answering show that OpenDecoder outperforms vanilla RAG and other strong baselines across diverse noisy environments.
2 Related Work
Related work improves RAG through retrieval refinement, robust input handling, and decoding optimization, but generally assumes retrieved information is relevant. OpenDecoder instead modifies attention according to retrieved-document relevance.
- RAG methods retrieve external resources for knowledge-intensive generation, while retrieval refinement seeks to improve the quality of searched information.
- Retrieval errors remain common because of search and corpus limitations, motivating robust RAG methods that optimize inputs or integrate knowledge more reliably.
- LLM decoding research addresses efficiency, effectiveness, and faithfulness through attention optimization, inference scaling, document compression, and misinformation management.
- Unlike prior approaches that assume input information is relevant, OpenDecoder modifies the attention mechanism according to retrieved-information relevance.
3 OpenDecoder
OpenDecoder incorporates explicit document-quality indicators into LLM decoding rather than relying only on prompt-based attention. It constructs relevance features, modulates attention with them, and trains the model to generate more robustly from noisy retrieved context.
- 3 OpenDecoder: OpenDecoder addresses noisy RAG by guiding decoding with explicit relevance information instead of relying solely on prompt design and internal attention.The approach aims to help the model distinguish useful evidence from irrelevant retrieved information during answer generation.
- 3.2 Constructing Indicators via Extracting Features from External Information: Three indicator types are constructed from retrieved documents: retriever relevance, LLM-ranker semantic relevance, and query performance prediction scores.These features can be used individually or combined to guide how strongly the model focuses on different context parts.
- 3.3 Learning to Leverage Explicit Indicators Features for Decoding: The indicators are normalized into token-level scores and incorporated into the LLM’s attention computation through learned parameters.Query and instruction tokens receive score 1, while document scores are normalized to [0, 1].
- 3.3 Learning to Leverage Explicit Indicators Features for Decoding: Modulating attention reshapes token importance during autoregressive decoding, allowing the model to emphasize useful information and reduce reliance on irrelevant documents.When all retrieved documents have very low relevance, relatively higher query and instruction scores guide reliance on parametric knowledge.
- 3.3 Learning to Leverage Explicit Indicators Features for Decoding: OpenDecoder fine-tunes the original LLM with learned attention-modulation parameters and generates answers using the updated model.The training objective maximizes the probability of producing the ground-truth answer while teaching the model how much to rely on external or internal knowledge.
4 Experimental Setup
The evaluation measures OpenDecoder across five question-answering datasets and three retrieval-noise settings. Comparisons use common backbone and retrieval conditions across multiple baseline systems.
- 4 Experimental Setup: OpenDecoder is evaluated on five datasets spanning general and multi-hop question answering.The datasets are NQ, TriviaQA, PopQA, HotpotQA, and 2WikiMultiHopQA.
- 4 Experimental Setup: The experiments use Normal, Noisy, and another setting with different levels of noise in retrieved documents.Normal Evaluation uses the retriever’s original top-10 documents, while Noisy Evaluation replaces the second half with partially relevant and irrelevant documents.
- 4 Experimental Setup: The comparison includes vanilla RAG, vanilla supervised fine-tuning, RobustRAG, AstuteRAG, and InstructRAG.These baselines represent retrieval-augmented generation, fine-tuning, filtering, retrieval refinement, and denoising approaches.
- 4.4 Implementation Details: All methods use the Qwen-2.5-3B-Instruct backbone and the same fixed retrieved documents for fair comparison.Table 1 reports results across three evaluation settings and noisy retrieval environments.
- 4.4 Implementation Details: The retrieval setup uses a 2018 Wikipedia dump, E5 retrieval, and ten retrieved documents, with noisy training proportions of 5 relevant, 3 partially relevant, and 2 irrelevant documents.Partially relevant and irrelevant documents are randomly sampled and fixed across methods for fair comparison.
5 Experimental Results
Across five datasets and three noisy-context settings, OpenDecoder consistently improves robustness over baseline approaches. Ablations and targeted analyses indicate that explicit relevance indicators, robust training, feature handling, and document ordering all affect performance.
- 5.1 Main Results: OpenDecoder consistently outperforms most baselines across three evaluation settings and significantly surpasses Vanilla SFT without external indicators.The experiments cover five datasets with different levels of noise in retrieved documents.
- 5.1 Main Results: Compared with RobustRAG and RbFT, OpenDecoder produces more robust answers in noisy and extremely noisy settings.The authors attribute this to explicit indicators that provide useful signals and alleviate bias from the internal decoding process.
- 5.1 Main Results: Increasing retrieved-document noise causes larger performance drops on NQ and TrivialQA than on HotpotQA and 2wiki.The authors report that factoid questions with retrieved support evidence are more sensitive to noisy inputs.
- 5.2 Ablation Study: Combining OpenDecoder mechanisms achieves better results across three evaluation settings and five datasets, while robust training contributes to stable performance.Ablations also find that feature aggregation helps on some datasets and that effective features may vary with dataset distribution.
- 5.3 Feature Aggregation and Normalization: Aggregating retrieval scores alone can suffice for general QA, whereas aggregating more feature scores helps on multi-hop QA datasets.The results suggest that complex questions benefit from more external indications, while additional features can introduce interference on general QA datasets.
- 5.3 Feature Aggregation and Normalization: Max normalization performs better than Min-Max on general QA, while Min-Max performs better on multi-hop QA; Exponential normalization causes a large performance drop.Thus, normalization choice can substantially change performance within the same OpenDecoder pipeline.
6 Conclusion
OpenDecoder modulates LLM internal information processing with explicit document-quality indicators to improve robustness when retrieved context contains varying noise. Its framework extracts quality features from retrieved documents, modifies attention-score computation, and adds robustness enhancement during training.
- OpenDecoder modulates LLM internal information processing with explicit indicators to improve robustness in noisy-context answer decoding.
- The framework extracts quality indicators from retrieved documents and applies them to modify attention-score computation in LLM networks.
- Robustness enhancement is integrated into training so OpenDecoder can handle varied noisy environments.
A Datasets Details
The evaluation uses five benchmarks covering factoid, reading-comprehension, factual, and multi-hop question answering. The datasets differ in question sources, evidence organization, and reasoning demands.
- Five benchmarks evaluate OpenDecoder: NQ, TriviaQA, PopQA, HotpotQA, and 2WikiMultihopQA.The unified training set combines NQ and HotpotQA for fine-tuning OpenDecoder.
- NQ contains anonymized aggregated Google queries, while TriviaQA provides trivia questions with independently gathered evidence documents.
- PopQA tests factual recall and entity ambiguity, whereas HotpotQA and 2WikiMultihopQA test multi-hop reasoning across contexts or Wikipedia passages.
B Baseline Details
The study implements all compared baselines with identical retrieved document sets across evaluation settings to ensure fair comparison. Instructions vary by method family, while RbFT shares OpenDecoder’s hyperparameter setting.
- All compared baselines use the same retrieved document sets across evaluation settings for fair comparison.
- Vanilla RAG, Vanilla SFT, and OpenDecoder use the same answer-generation instruction.
- RobustRAG, InstructRAG, and AstuteRAG inherit their original repository instructions, while RbFT uses its original instruction with OpenDecoder’s hyperparameter.
C Investigation of Scaling Model Size
Scaling model size improves tolerance to contextual noise for both vanilla SFT and OpenDecoder, with OpenDecoder’s explicit-indicator benefits becoming more pronounced at larger sizes. The gains are especially evident on complex QA datasets, but smaller models do not consistently perform better across datasets.
- Investigation of Scaling Model Size: Larger models improve contextual-noise tolerance for both vanilla SFT and OpenDecoder in noisy, normal, and extreme-noisy settings.
- Investigation of Scaling Model Size: OpenDecoder’s effectiveness in leveraging explicit indicators becomes more pronounced as model size increases.
- Investigation of Scaling Model Size: 1.5B models do not consistently achieve better performance across all datasets.
- Investigation of Scaling Model Size: Effectively integrating external signals with internal reasoning and aggregating multiple guidance features requires higher model capacity and implicit learning during training.
- Investigation of Scaling Model Size: Scaling gains are more pronounced on complex QA datasets than on general datasets, suggesting stronger implicit reasoning ability in larger models.
D Discussion on Time and Space Efficiency
OpenDecoder matches Vanilla SFT in computation complexity for both offline training and online inference, while adding storage overhead for explicit guidance across affected Transformer layers.
- OpenDecoder and Vanilla SFT have the same computation complexity, O(|d|^2h + |d|h^2), in the RAG setting.Here, d is the average document-token count and h is the decoder-only LLM hidden dimension.
- The equal computation cost holds for both offline training and online inference because relevance scores are produced with retrieved documents and score normalization is negligible.
- OpenDecoder adds O(nh) storage overhead compared with Vanilla SFT for normalized token-level scores across n impacted Transformer layers.The stored normalized score has shape R^{h×h}, matching the Query, Key, and Value metrics in the attention network.