PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Attributed Text Generation of Large Language Models via Preference Learning

Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, Min Zhang

arXiv:2403.18381v1cs.CLcs.AI

TL;DR

Because attribution work often focuses on retrieval and evaluation, APO treats attribution as preference learning and develops curated and synthesized preference data for progressive optimization. Across three datasets, it improves citation F1 and response quality over compared baselines.

  • Problem

    Attribution methods often focus on retrieval and automatic evaluation, while LLMs can generate hallucinations that undermine the reliability of their content.

    Source: §1 · p. 1

  • Method

    APO frames attribution as preference learning, combining curated post-training examples and automatically synthesized preference pairs with progressive optimization using fine-grained information.

    Source: §1 · p. 2

  • Results

    Across ASQA, StrategyQA, and ELI5, APO surpasses compared baselines with improved citation F1 and response quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported experiments support APO's effectiveness for attributed text generation on the three evaluated datasets.

    Source: §7 · p. 8

    The datasets may not represent real-world diversity, and automatically generated preference data may not cover all hallucination scenarios.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.

1 Introduction

The introduction frames hallucinations and limits in existing attribution approaches as motivation for optimizing how models generate attributed text. It presents APO as a preference-learning framework with curated post-training data, synthesized preference pairs, and progressive optimization.

1 Introduction

  • Prior attribution work often emphasizes retrieval and evaluation, while the authors argue generation itself should also be optimized to address answer and attribution hallucinations.The introduction also notes that common preference-learning approaches face costly data collection and other practical difficulties.Source: §1 · p. 1
  • Across ASQA, StrategyQA, and ELI5, APO surpasses compared baselines in citation F1 and response quality.The authors describe this as improvement across all three datasets.Source: §1 · p. 2
  • APO models attribution as preference learning and uses progressive preference optimization inspired by human citation practices.The method leverages fine-grained information and experience replay without explicit reward modeling or reinforcement learning.Source: §1 · p. 2
  • The framework combines 6,330 curated post-training examples with 95,263 automatically synthesized attribution preference pairs.The curated examples are collected and filtered from existing datasets.Source: §1 · p. 2
  • The authors report applying preference learning to attribution tasks and show that APO can also be used with synthesized preference scenarios.They describe this as the first application of preference learning to attribution, to their knowledge.Source: §1 · p. 2
  • The paper establishes a data-collection pipeline for attribution and proposes progressive optimization using fine-grained information to address sparse rewards.The authors also benchmark existing direct preference optimization methods for attribution tasks.Source: §1 · p. 2

2 Related Work

The related work covers verification methods that identify support for generated text and preference-optimization methods for improving generation. The paper positions APO as extending attribution beyond verification and preference methods beyond coarse-grained feedback.

2.1 Text Generation for Verification

  • Verification research includes AIS, which turns outputs into standalone propositions and evaluates whether a source supports them; this paper instead frames attribution improvement as preference learning.The cited work includes Rashkin et al. (2021) on AIS and later extensions to QA and generated text with citations.Source: §2.1 · p. 2

2.2 Preference Optimization Methods

  • RLHF is described as complex, time-consuming, and unstable, while direct preference optimization removes reward-model training; existing methods do not fully account for fine-grained preferences.The paper describes direct preference optimization methods as more data-efficient and stable.Source: §2.2 · pp. 2–3

3 Preliminary

The preliminary section outlines preference learning as a pipeline from pretraining and optional supervised fine-tuning through preference data collection to optimization. It then introduces RLHF and direct preference optimization as alternative optimization approaches.

Preliminary

  • The preference-learning pipeline comprises pretraining and optional supervised fine-tuning, preference-data collection, and preference optimization.The reference policy follows pretraining or supervised fine-tuning, and preference data can consist of context-response pairs labeled by preference.Source: §3 · p. 3
  • In preference-data collection, the reference policy generates two responses for a context, and humans label which response they prefer.The labeled context, responses, and preference indicator form the preference dataset.Source: §3 · p. 3
  • RLHF estimates a reward from preference data and uses PPO to maximize it while applying a KL constraint to the reference model.The reward function is learned using the Bradley-Terry model.Source: §3 · p. 3
  • DPO eliminates reward-model training by deriving a loss directly for the current policy.Source: §3 · p. 3

4 Methodology

The methodology frames attributed generation as producing statements with citations, then trains a model through post-training and preference optimization. Its preference data and optimization operate at statement level, with replay and inference-time refinement also included.

4.1 Problem Formulation

  • The task is to produce a response of distinct statements, each associated with citations from a document corpus.Statements are used because a sentence typically expresses a coherent, concise claim that is easier to verify.Source: §4.1 · p. 3

4.2 Overall Framework

  • APO combines post-training to ground the base model for attribution with preference optimization addressing generation and attribution hallucinations.The post-training goal is to generate an answer and add citations for statements when needed.Source: §4.2 · p. 4, §4.3 · p. 4

4.3 Post-training

  • Post-training uses 6,330 processed samples drawn from EVIGSE, ExpertQA, and HARGID attribution datasets.The selected datasets were described as diverse and high-quality, with annotations from human experts or powerful LLMs.Source: §4.3 · p. 4

4.4 Preference Optimization

  • Automatic preference collection generates positive and negative statement pairs from relevant and supported labels, including examples of irrelevant support and unsupported content.Negative examples distinguish irrelevant-but-supported text from relevant-but-unsupported text, including fabricated statements and mistaken synthesis; omission is also listed as a subtype.Source: §4.4 · p. 5
  • Statement-level preference optimization targets fine-grained deterministic preferences; experience replay adds autoregressive language-modeling loss at fixed intervals to help alleviate overfitting.The progressive loss has an implicit mean-pooling procedure, while replay uses old training samples to address overfitting and text degradation associated with distribution shift.Source: §4.4 · pp. 5–6

4.5 Inference and Refinement

  • At inference, the model generates an initial response and then propagates citations backward from later cited statements to earlier uncited ones.The procedure traverses statements from the end, updating a citation set and assigning it to statements without citations.Source: §4.5 · p. 6

5 Setup

The experiments evaluate long-form QA on ASQA and ELI5, alongside reasoning-focused StrategyQA, using citation and task-quality metrics. Baselines and implementation settings are specified for comparison.

5.1 Datasets and Evaluation Metrics

  • Evaluation uses official test sets from ASQA and ELI5 long-form QA, plus StrategyQA, which focuses on open-domain questions with implicit reasoning steps.The paper reports citation recall, precision, and F1 across datasets, alongside EM-R for ASQA, claim recall for ELI5, and accuracy for StrategyQA.Source: §5.1 · p. 6

5.2 Competitive Methods

  • The comparison includes in-context citation prompting, two post-hoc attribution baselines, Self-RAG, and AGREE.Self-RAG adaptively retrieves documents; AGREE uses test-time adaptation and automatically collected data for self-grounding.Source: §5.2 · p. 6

5.3 Implementation Details

  • Unless otherwise specified, experiments retrieve the top five documents and use llama-2-13b-base for fair comparison.Inference temperature is 0.01; preference optimization uses LoRA for one epoch with alpha 2 and rank 16.Source: §5.3 · p. 6

6 Results

APO outperforms baselines in correctness and citation quality across three datasets, while ablations and error analyses reveal distinct contributions from preference data, training strategy, and retrieval context.

6.1 Main Result

  • Across ASQA, StrategyQA, and ELI5, APO outperforms baselines in correctness and citation quality; versus the post-training baseline, preference optimization yields an 8.0 average increase in citation F1.APO outperforms Self-RAG by 8.8 points on the EM-R metric.Source: §6.1 · pp. 6–7, Table 2 · p. 7

6.2 Ablation Study

  • Removing hallucinated-statement preference data significantly lowers citation F1, while removing unintentional-omission data yields poor generation quality.Mistaken-synthesis data contributes little to performance improvement but can improve groundedness under human evaluation.Source: §6.2 · p. 7, Table 3 · p. 7
  • Response-level optimization such as vanilla DPO improves the model slightly less, and excluding ASQA questions from preference data lowers generation and citation quality.The authors attribute the latter decline to high-quality in-domain ASQA questions in the preference data.Source: §6.2 · p. 7, Table 3 · p. 7

6.3 Different Prompting Strategy

  • APO's Citation F1 performs best on ASQA with VANILLA and ORACLE contexts and improves on ELI5 with SUMM and SNIPPET contexts.The authors report that context format affects attribution performance.Source: §6.3 · pp. 7–8, Table 4 · p. 8

6.4 Different PO Methods

  • All tested preference optimization methods boost performance over post-training and SFT baselines, although performance varies across metrics.The method transfers to several preference optimization methods.Source: §6.4 · p. 8, Table 5 · p. 8

6.5 Error Analysis

  • Nearly half of manually reviewed attribution errors are fabrication errors, including unsupported text or information attributed to irrelevant documents.The review also finds omission errors and less-common synthesis errors; truncated low-quality documents can prompt extrapolation.Source: §6.5 · p. 8, Table 6 · p. 8

7 Conclusion

The paper presents APO as a preference-learning framework for attributed text generation, combining curated post-training data with automatically synthesized preference pairs. Across three datasets, it reports leading citation F1 and improved response quality.

7 Conclusion

  • APO frames attribution for generated text as a preference-learning task.The framework uses curated post-training collections and an automated preference-data synthesis algorithm intended to reduce manual labeling costs.Source: §7 · p. 8
  • Experiments on three datasets demonstrate APO's effectiveness, with leading citation F1 and improved response quality.Source: §7 · p. 8
  • The authors identify extending APO to real-world applications as future work.Source: §7 · p. 8

Limitation

The paper notes limitations in dataset scope, coverage of hallucination scenarios, scalability, and the approximation of human citation practices. These boundaries leave generalization and real-world use incompletely assessed.

Limitation

  • The narrow dataset scope may not represent the diversity of real-world applications, and synthesized data may not cover all hallucination scenarios.Source: Limitation · p. 9
  • Scalability and economic feasibility on larger datasets remain unexplored despite the method's aim of addressing high labeling costs.Source: Limitation · p. 9
  • The framework's approximation of human citation processes may not capture all the complexities of scholarly writing.Source: Limitation · p. 9
  • Robustness to deliberate hallucination and adaptability to rapidly evolving NLP fields are not fully assessed.Source: Limitation · p. 9

Ethical Statement

The ethical statement highlights risks in trusting generated citations without verification, using them in time-critical decisions, assuming their reliability, and handling copyright. It calls for critical user engagement and verification.

Ethical Statement

  • Generated citations may increase trust without verification and pose concerns in time-critical decisions or when assumed inherently trustworthy.Source: Ethical Statement · p. 9
  • The statement calls for users to critically engage with and verify machine-generated content to mitigate misinformation and hallucinations.Source: Ethical Statement · p. 9
  • Copyright concerns and potential legal challenges are also identified as issues in using attributed LLM text.Source: Ethical Statement · p. 9

A Details about Preference Optimization Methods

This appendix compares preference-optimization methods and documents preprocessing and templates used to prepare post-training and preference data.

A Details about Preference Optimization Methods

  • RLHF uses a learned reward model and on-policy PPO, while deterministic preferences can cause approximation error and out-of-distribution reward inference, making the process complex and unstable.Its reward model uses the Bradley-Terry assumption that preference scores can be approximated by point-wise rewards.Source: A Details about Preference… · p. 13
  • DPO can overfit deterministic preferences by ignoring KL regularization; IPO instead derives an off-policy loss that regularizes the policy toward the reference.KTO maximizes generation utility without preference pairs, while SLiC calibrates sequence likelihoods to align with reference sequences.Source: A Details about Preference… · p. 13

B Details about Pre-processing

  • Preprocessing removes ExpertQA samples with empty references or duplicate-context document IDs, and EVIGSE samples with “None” references or no reference documents.The labels and citation formats for these datasets are also normalized.Source: B Details about Pre-processing · p. 13

C Post-training Templates

E Case Study

The case study classifies attribution errors through examples, while the related-work discussion describes retrieval-augmentation approaches and identifies attribution hallucination as an insufficiently addressed issue.

E Case Study

F Related Works of Retrieval Augmentation of LLMs

  • Prior studies had not adequately addressed attribution hallucination; related approaches include memory-based citation generation, reasoning-chain retrieval, and retrieval verification.The discussion describes long- and short-term memories for claim and citation generation and SearChain's retrieval augmentation at reasoning-chain nodes.Source: E Case Study · p. 14, F Related Works of Retrieval… · p. 14
  • The automatic preference-data algorithm takes initial data, a critic, a generator, a retriever, and error instructions, and outputs an augmented preference dataset.It iterates over questions, documents, statements, and their attributed passages.Source: F Related Works of Retrieval… · p. 15
  • The algorithm retains examples when referenced passages are relevant, retrieves top-m passages, and uses a critic to assess relevance and support.For each statement, the algorithm can add passages assessed as supporting it to its attributed passages.Source: F Related Works of Retrieval… · p. 15
Loading 2403.18381v1…