PaperlayerView exact revision on arXiv

Source-linked AI summary

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation

Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, Ji-Rong Wen

arXiv:2406.18676v2cs.CLcs.AIcs.LG

TL;DR

RAG can mitigate hallucinations, but retrieved documents may fail to match the knowledge diverse LLMs need and can conflict with their reasoning preferences. DPA-RAG constructs preference data, aligns a reranker and LLMs to those preferences, and outperforms strong baselines across four knowledge-intensive QA datasets.

  • Problem

    Retriever–reader preference gaps can leave retrieved documents misaligned with LLM reasoning needs or conflicting with their self-knowledge, complicating reliable RAG.

    Source: §1 · pp. 1–2

  • Method

    DPA-RAG builds preference knowledge using five query-augmentation strategies and quality filtering, then aligns rerankers and LLMs through multi-grained reranker tasks and a pre-aligned stage before SFT.

    Source: §1 · pp. 2–3

  • Results

    DPA-RAG outperforms all strong baselines across four knowledge-intensive QA datasets.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The authors report that their analyses provide practical insights for developing reliable RAG systems.

    Source: §1 · p. 3, §5 · p. 11

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) has demonstrated effectiveness in mitigating the hallucination problem of large language models (LLMs). However, the difficulty of aligning the retriever with the diverse LLMs' knowledge preferences inevitably poses an inevitable challenge in developing a reliable RAG system. To address this issue, we propose DPA-RAG, a universal framework designed to align diverse knowledge preferences within RAG systems. Specifically, we initially introduce a preference knowledge construction pipline and incorporate five novel query augmentation strategies to alleviate preference data scarcity. Based on preference data, DPA-RAG accomplishes both external and internal preference alignment: 1) It jointly integrate pair-wise, point-wise, and contrastive preference alignment abilities into the reranker, achieving external preference alignment among RAG components. 2) It further introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT), enabling LLMs to implicitly capture knowledge aligned with their reasoning preferences, achieving LLMs' internal alignment. Experimental results across four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all baselines and seamlessly integrates both black-box and open-sourced LLM readers. Further qualitative analysis and discussions also provide empirical guidance for achieving reliable RAG systems. Our code is publicly available at https://github.com/dongguanting/DPA-RAG.

1 Introduction

RAG can help mitigate hallucinations, but retrieved documents may not match the knowledge an LLM needs and can conflict with its own reasoning preferences. DPA-RAG addresses this gap through preference-data construction, reranker alignment, and LLM self-alignment.

1 Introduction

  • For knowledge-intensive tasks, relying only on internal LLM knowledge can expose answers to factual inconsistency and hallucination, motivating retrieval-augmented technology.RAG assists LLMs in integrating relevant external knowledge.Source: §1 · p. 1
  • Retriever and reader components have distinct architectures, training objectives, and task formats, so retrieved documents may fail to meet LLMs’ reasoning needs or conflict with their self-knowledge.The paper frames this preference gap as a challenge for reliable RAG.Source: §1 · pp. 1–2
  • Documents ranked 100th can support correct answers, while some high-similarity documents can mislead LLMs more than lower-ranked alternatives.The preliminary analysis on GPT-3.5 spans three QA benchmarks and includes direct responses and answers referencing retrieved documents.Source: §1 · p. 2, Figure 1 · p. 2
  • DPA-RAG combines pair-wise, point-wise, and contrastive reranker alignment with a pre-aligned stage before standard SFT for LLM self-alignment.The reranker targets external alignment across RAG components; the pre-aligned stage guides LLMs toward knowledge aligned with their reasoning preferences.Source: §1 · pp. 2–3
  • Five query augmentation strategies and quality filtering synthesize high-quality preference data to address its scarcity and limited diversity.The framework uses the resulting preference knowledge to align downstream models.Source: §1 · pp. 2–3
  • DPA-RAG outperforms all strong baselines across four knowledge-intensive QA datasets, with analyses offering practical insights for reliable RAG systems.The authors describe it as a plug-and-play solution.Source: §1 · pp. 2–3

2 Related Work

Prior work aligns language models to human preferences and develops methods for retriever supervision, multi-round retrieval, corpus filtering, and reranking. The paper identifies limited multi-level alignment with diverse LLM knowledge preferences as a gap its approach targets.

2 Related Work

  • Preference-alignment research uses human preferences, including reinforcement learning and DPO-style optimization, while human or expert-model annotations increase alignment cost.RAG-focused studies also use LLM-generated supervision, multi-round retrieval, and training-corpus noise filtering.Source: §2 · p. 3
  • Existing RAG alignment approaches often lack multi-level alignment, limiting adaptation to diverse LLM knowledge preferences.DPA-RAG is presented as aligning retrievers with those preferences without external expert annotations.Source: §2 · p. 3
  • Reranking work spans sentence-transformer fine-grained ranking, point-wise generative reranking, and pair-wise or list-wise LLM rerankers.Personalized sorting methods also address distinct user preferences, inspiring the paper’s effort to distill LLM preferences into the reranker.Source: §2 · p. 3

3 Methodology

DPA-RAG constructs preference knowledge to align the retriever, reranker, and LLM, using both external component alignment and internal LLM self-alignment. Its method moves from task definition and preference-data construction to reranker training and a pre-aligned LLM stage before SFT.

3 Methodology

  • DPA-RAG combines preference knowledge construction, a preference-aligned reranker, and LLM self-alignment to align knowledge preferences across RAG components.The framework’s inference process uses the aligned reranker to filter retrieved knowledge before the LLM reads it.Source: §3 · p. 3

3.1 Task Definition

  • The task definition uses a retriever to return top-k documents, a reranker to filter them for LLM preference alignment, and an LLM to generate the answer.The method also optimizes the LLM for RAG tasks through a self-alignment mechanism.Source: §3.1 · p. 4

3.2 Preference Knowledge Construction

  • Preference knowledge is mined by comparing direct answers with answers using sampled documents, then retaining examples containing aligned or unaligned knowledge.Four documents are sampled at ranks 1, 25, 50, and 100; examples labeled both correct or both incorrect are discarded because their document roles are difficult to distinguish.Source: §3.2 · pp. 4–5
  • Five query augmentations—rephrasing, complexity, decomposition, constraints, and SPARQL—expand the preference data, and an NLI filter removes contradictory augmentations.The preference data is described as only 20% of the training data; approximately 20% of augmented data is labeled contradiction and excluded to maintain intent consistency.Source: §3.2 · p. 5

3.3 Reranker-LLM Alignment

  • The reranker combines point-wise labels, pairwise preferences, and contrastive learning to identify and prioritize documents aligned with LLM preferences.Its document preference score weights the LLM preference score and the retriever’s similarity score; contrastive learning treats aligned documents as positives and other-instance documents as negatives.Source: §3.3 · pp. 5–6
  • MGDA-UB dynamically balances the jointly optimized preference tasks to seek a Pareto-optimal solution.The method uses multi-task learning to optimize the reranker’s multi-grained alignment objectives while addressing potential task bias and conflict.Source: §3.3 · p. 6

3.4 LLM Self-Alignment

  • Before SFT, the LLM is trained to judge a sampled aligned or unaligned document as positive or negative, encouraging implicit identification of preference knowledge.The pre-aligned stage uses that document alongside randomly sampled documents in a top-k set.Source: §3.4 · pp. 6–7
  • After pre-alignment, the LLM undergoes QA supervised fine-tuning using training examples whose documents are reordered and filtered by the preference-aligned reranker.The pre-alignment capability is described as helping the LLM focus on aligned knowledge during SFT.Source: §3.4 · p. 7

4 Experiments

Across four knowledge-intensive QA datasets, DPA-RAG improves on traditional RAG baselines, and analyses examine its alignment behavior, scaling, data quality, and training strategy. The experiments cover open-domain, multi-hop, and knowledge-base QA, using Hit@1 and F1 score.

4.1 Datasets and Metrics

  • The four datasets span open-domain, multi-hop, and knowledge-base QA; evaluation uses Hit@1 and F1 score.NQ and TQA represent open-domain QA, HotpotQA represents multi-hop QA, and WebQSP represents knowledge-base QA.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 Main Results

  • Over 5% performance improvement across all four datasets for DPA-RAG (LLaMA2-7B) versus traditional RAG baselines, with gains consistent across various readers.The main-results table reports DPA-RAG and different baseline types on four QA benchmarks.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • Removing any DPA-RAG component lowers performance, with the largest drop after removing the preference-aligned reranker.The complete framework also exceeds the combined gains from its reranker and pre-aligned task; smaller BGE and ColBERTv2 rerankers can match or outperform LLM-based rerankers, and DPA-RAG beats direct logit-based alignment methods.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Quantitative Analysis

  • From 500M to 7B parameters, baseline RAG performance rises sharply, then stabilizes above 7B; DPA-RAG gains remain stable as model size increases.The gains are larger on HQA, where unaligned knowledge is more extensive and average F1 is below 50, than on TQA, where average F1 exceeds 60.Source: §4.3 · p. 9, Figure 3 · p. 9
  • DPA-RAG achieves the highest Aligned Knowledge scores and reduces Unaligned Knowledge; its improvement in Both Correct exceeds that in Aligned Knowledge.The reported comparison covers NQ and TQA.Source: §4.3 · pp. 9–10, Figure 4 · p. 9
  • All five query augmentation strategies improve LLM performance to differing degrees, and NQ performance trends upward with the combined data-quality score.Complexity and Decomposition notably improve both complexity and diversity scores; the analysis relates augmentation effectiveness to query complexity and diversity.Source: §4.3 · p. 10, Table 4 · p. 10, Figure 5 · p. 10
  • Sequential training after the pre-aligned phase yields stable performance gains, whereas mixing preference and QA data leads to performance decline and fluctuations.The authors suggest optimization conflicts in multi-task training may explain the mixed-training behavior.Source: §4.3 · p. 10, Figure 5 · p. 10

5 Conclusion

DPA-RAG addresses knowledge-preference gaps among RAG components through preference-based reranking and LLM self-alignment, and the paper reports gains across four knowledge-intensive QA datasets. Its workflow mines and augments preference data, aligns the reranker, and pre-aligns open-source readers before SFT.

5 Conclusion

  • DPA-RAG mines and filters LLM-preferred knowledge, adds five query-augmentation strategies, and combines pair-wise, point-wise, and contrastive reranker alignment with LLM self-alignment.The conclusion identifies these as external alignment among RAG components and internal alignment for LLMs.Source: §5 · pp. 10–11
  • DPA-RAG outperforms strong baselines across four knowledge-intensive QA datasets, with further analysis offering practical insights for reliable RAG systems.The conclusion also describes its external reranker alignment and internal LLM self-alignment.Source: §5 · p. 11

A.1 The Overall Algorithm Workflow of DPA-RAG

  • Reranker training mines, augments, and filters preference data before multi-grained distillation alignment; open-source readers use the aligned reranker and a pre-aligned task before vanilla QA SFT.For open-source readers, the workflow also reranks retrieved training and test documents; closed-source readers use the preference-aligned reranker to sort test documents before prediction.Source: §A.1 · p. 22

A.2 Preference Scoring Mechanism for Different LLMs

  • For models below 7B parameters, the scoring mechanism uses output logits; 7B and 13B models provide preference scores from 1 to 5, combined with retriever similarity.For GPT-3.5 and GPT-4, the paper uses pairwise document comparisons because they produced more precise preference ordering than individual paragraph scoring.Source: §A.2 · p. 22

B.1 Datasets

The appendix describes four QA datasets and how preference examples are mined, augmented, filtered, and used in reader prompts. Its templates distinguish a pre-aligned judgment task from vanilla answer generation.

B.1 Datasets

  • The four datasets described are NaturalQuestions, TriviaQA, HotpotQA, and WebQuestionsSP.Their passages characterize NQ and TQA as question-answer benchmarks, HQA as requiring multi-step reasoning, and WebQSP as involving SPARQL queries.Source: §B.1 · pp. 22, 24
  • Preference construction selects retrieved documents at ranks 1, 25, 50, and 100 and distinguishes aligned from unaligned knowledge using LLM and document answer correctness.The algorithm adds an aligned example when the LLM answer is wrong but a selected document supplies the correct answer.Source: §B.1 · p. 23
  • Five query-augmentation strategies—Complexity, Constraint, SPARQL, Decomposition, and Rephrasing—expand preference data, which is filtered to retain queries entailing the original.The training set includes the original training data and augmented preference examples.Source: §B.1 · pp. 22–23

B.2 Prompt Templates

  • The vanilla SFT prompt supplies the top-k documents and query, then requests a short answer based on the documents or internal knowledge.The algorithm loads pre-aligned parameters before supervised fine-tuning.Source: §B.2 · p. 24, §B.1 · p. 24
  • The pre-aligned prompt asks the reader to answer using retrieved documents or internal knowledge and judge whether the preference document is positive or negative knowledge.The input combines one preference document with randomly selected documents.Source: §B.2 · pp. 24–25
  • The augmentation prompt provides the original query, reference documents, title, and rewriting requirements to generate new queries.The prompt template labels the output field “New Queries.”Source: §B.2 · p. 25

B.3 Implementation Details

Implementation uses a DPR retriever, BGE reranker, and LLM readers, with specified training settings and averages reported across five random seeds.

B.3 Implementation Details

  • DPR encodes questions and documents, then retrieves the top 100 relevant Wikipedia documents by dot-product similarity.Source: §B.3 · p. 25
  • The BGE reranker is fine-tuned for 10 epochs with batch size 16 and learning rate 1e-5 to order the retrieved documents and return the top three.Source: §B.3 · p. 25
  • LLM QA fine-tuning runs for three epochs with batch size 128 and learning rate 7e-5, using eight A100 80g GPUs.The setup uses a 3% warmup and reports average performance over five experiments with different random seeds.Source: §B.3 · p. 25
  • The paper states that datasets and evaluation benchmarks are open-source and promises to release its code after blind review.Source: §B.3 · p. 25

B.4 Baselines

The comparison set includes reranker-based and preference-aligned RAG baselines, alongside descriptions of the LLM readers used by DPA-RAG.

B.4 Baselines

  • Reranker baselines include RankGPT, LRL, PRP, RankLLaMA, BGE, BCEmbedding, and ColBERTv2.The listed approaches include listwise prompting, pairwise ranking, pointwise reranking, and embedding-based rerankers.Source: §B.4 · p. 26
  • Preference-aligned baselines include KnowPAT, REPLUG, RA-Judgement, RRHF, RAFT, and FILCO.The descriptions span preference-set construction, retriever optimization, judgment, response ranking, and context selection.Source: §B.4 · p. 26
  • Reader models described include LLaMA2, LLaMA3, Qwen1.5, Qwen2, Mistral, Phi2, GPT-3.5, and GPT-4.Source: §B.4 · pp. 26–27
  • Table 5 reports detailed ablations for LLaMA2-7B on NQ and TQA, and defines Point-wise, Pair-wise, and CPA as Point-wise, Pair-wise, and Contrastive Preference Alignment, respectively.Source: Table 5 · p. 27

C.1 Detailed Results for Ablation Studies

The ablations report gains from the framework’s components, query augmentation, and multi-grained alignment tasks. Pre-alignment also shows steady gains, particularly on TQA, where internal alignment has greater potential than external alignment.

C.1 Detailed Results for Ablation Studies

  • Removing any DPA-RAG component lowers results; query augmentation brings substantial gains, while quality filtering adds slight gains.The authors attribute the augmentation gains to added supervision for both the reranker and reader, and filtering gains to maintaining intent consistency between original and augmented data.Source: §C.1 · p. 27
  • Each multi-grained distillation task independently gives stable gains on NQ and TQA, with point-wise preference alignment producing the largest alignment gains.MGDA-UB adds stable gains beyond jointly optimizing the three tasks.Source: §C.1 · p. 28
  • Pre-alignment yields steady gains, especially on TQA, where internal alignment has greater potential than external alignment, unlike on NQ and HQA.The authors use this difference across datasets to argue for dual alignment across domains.Source: §C.1 · p. 28

C.2 Details about Diverse Query Augmentations

The section reviews query-augmentation examples and training-data intent tags. The sampled tags are concentrated in historical, sports, and entertainment queries, and the examples’ complexity generally follows the reported complexity and diversity trends.

C.2 Details about Diverse Query Augmentations

  • The augmented-query examples’ complexity generally matches the complexity and diversity trends reported for the data.The examples were generated using gpt-3.5-turbo-0613 APIs with different augmentation requirements.Source: §C.2 · p. 28
  • Among 10,000 randomly selected SFT samples, the most common intent tags concern history, sports, and entertainment.The sample pool included original data and five sets of augmented data.Source: §C.2 · p. 28
  • Figure 6 encodes each tag by its first two words and scales segment size by frequency; only tags appearing more than 600 times are shown.The visualization is a sunburst chart.Source: Figure 6 · p. 29, §C.2 · p. 29

C.3 Case Studies for Preference Alignment

The case study and manual analysis examine supporting and misleading knowledge in retrieved documents. Across the examples, DPA-RAG’s reranking and pre-aligned stage are described as helping retain or prioritize knowledge useful for reasoning.

C.3 Case Studies for Preference Alignment

  • In the case study, the authors manually marked supporting and misleading knowledge in the top three retrieved documents in green and red, respectively.Source: §C.3 · p. 30
  • For one example, baseline retrieval includes aligned and unaligned knowledge, and the unaligned material misleads reasoning; DPA-RAG reranking filters it and retains aligned knowledge.The analysis reports successful reasoning after reranking.Source: §C.3 · p. 31
  • For the cities example, DPA-RAG filters unaligned knowledge and finds aligned knowledge in lower-similarity documents, reranking it to the front before successful reasoning.Source: §C.3 · p. 33
  • For the Kilimanjaro example, DPA-RAG ranks aligned knowledge first, filters unaligned knowledge, and the analysis reports successful inference.Source: §C.3 · p. 35
  • For the van Gogh example, baseline documents contain aligned knowledge but in relatively low quantity and order; DPA-RAG reranking and pre-alignment help the LLM focus on useful information.The analysis reports successful reasoning.Source: §C.3 · p. 37
Loading 2406.18676v2…