PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning Reduces Hallucination in Conversations

Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

arXiv:2212.10400v1cs.CLcs.AI

TL;DR

Language models can draw on knowledge stored in their parameters, yet may generate irrelevant or factually incorrect conversational responses. MixCL uses contrastive learning with hard negatives and span-level mixing to reduce hallucination. On Wizard-of-Wikipedia, it improves over LM-based methods and is comparable to KB-based methods, with efficiency and scalability advantages.

  • Problem

    Language models can produce plausible-looking but irrelevant or factually incorrect dialogue responses despite storing knowledge in their parameters.

    Source: §1 · p. 1

  • Method

    MixCL contrasts responses with confusing negative knowledge, using retrieved or model-generated negatives and mixed contrastive learning at the span level.

    Source: §1 · p. 2, §5.1 · p. 3

  • Results

    MixCL improves response quality and relevancy by 5% to 15% over previous LM-based methods and achieves performance comparable to state-of-the-art KB-based methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MixCL offers efficiency and scalability advantages, including 5× faster inference than the compared KB-based approach.

    Source: §1 · p. 2, §8 · p. 7

    The work identifies knowledge forgetting as a limitation and leaves reproduction for less resource-rich languages to future work.

    Source: §8 · p. 7

Abstract

from arXiv · show

Pre-trained language models (LMs) store knowledge in their parameters and can generate informative responses when used in conversational systems. However, LMs suffer from the problem of "hallucination:" they may generate plausible-looking statements that are irrelevant or factually incorrect. To address this problem, we propose a contrastive learning scheme, named MixCL. A novel mixed contrastive objective is proposed to explicitly optimize the implicit knowledge elicitation process of LMs, and thus reduce their hallucination in conversations. We also examine negative sampling strategies of retrieved hard negatives and model-generated negatives. We conduct experiments on Wizard-of-Wikipedia, a public, open-domain knowledge-grounded dialogue benchmark, and assess the effectiveness of MixCL. MixCL effectively reduces the hallucination of LMs in conversations and achieves the highest performance among LM-based dialogue agents in terms of relevancy and factuality. We show that MixCL achieves comparable performance to state-of-the-art KB-based approaches while enjoying notable advantages in terms of efficiency and scalability.

1 Introduction

LM-based dialogue agents can use knowledge stored in their parameters, but may still produce irrelevant or factually incorrect responses. MixCL addresses this with contrastive learning and achieves strong results against LM-based and KB-based dialogue agents.

1 Introduction

  • LM-based agents can generate plausible-looking statements that are irrelevant or factually incorrect, while hallucination in eliciting knowledge from LM parameters remains underexplored.The introduction attributes hallucination partly to maximum-likelihood training with teacher forcing, which can encourage blind imitation of training data.Source: §1 · pp. 1–2
  • In a pilot study of 200 BART responses, annotators labeled 24% as intrinsic hallucinations and 27% as extrinsic hallucinations.Three well-informed experts assessed knowledge relevancy and factuality on Wizard-of-Wikipedia responses.Source: §1 · p. 1
  • MixCL contrasts ground-truth responses with confusing negative knowledge, sampled by retrieval or model bootstrapping, using span-level mixes tailored to two hallucination types.Its entity-based and constituency-based mix-up strategies target intrinsic and extrinsic hallucinations, respectively; end-to-end optimization avoids retrieval at inference.Source: §1 · p. 2
  • MixCL improves response quality and relevancy by 5% to 15% over previous LM-based methods, with performance comparable to state-of-the-art KB-based methods.The reported approach also speeds inference up 5× and shows superior scalability.Source: §1 · p. 2

2 Related work

Prior work grounds open-domain dialogue using retrieved knowledge or knowledge stored in language-model parameters. MixCL differs from earlier contrastive approaches by targeting hallucination at the span level with hard negatives.

2.1 Knowledge-grounded dialogues

  • KB-based dialogue methods retrieve knowledge from a corpus, but face retrieval errors, inefficiency, and challenges integrating knowledge at multiple granularities.Prior work studies knowledge selection and knowledge-grounded response generation.Source: §2.1 · p. 2

2.2 Language models as knowledge bases

  • LM-based dialogue methods use knowledge stored in model parameters, but existing approaches still face hallucination; MixCL targets this implicit knowledge-elicitation problem.Prior work includes finetuning, knowledge-enhanced finetuning, topic-aware adapters, prompting, and lexical knowledge internalization.Source: §2.2 · p. 2

2.3 Contrastive learning

  • Unlike prior contrastive methods for retrieval, generation, and summarization, MixCL reduces hallucination at the span level with effective negative sampling.Earlier contrastive work addressed properties such as repetition, maliciousness, dullness, inconsistency, and sentence-level summarization hallucination.Source: §2.3 · p. 2

3 Problem formulation

The problem formulation represents a dialogue context, response, and knowledge corpus, and distinguishes explicit-retrieval KB agents from LM agents that use parameter-stored knowledge. The paper focuses on the latter.

3 Problem formulation

  • Given dialogue context x and knowledge corpus K, the agent predicts an informative response y using knowledge from K.The corresponding response and ground-truth knowledge are denoted y and k.Source: §3 · p. 2
  • KB-based agents explicitly retrieve knowledge from K before response generation using a retriever and a response generator.The paper presents this as one of two approaches to knowledge-grounded dialogue.Source: §3 · p. 3, Figure 2 · p. 3
  • LM-based agents generate responses from knowledge stored in model parameters, without an explicit retrieval step; the paper focuses on this approach.The model is first pretrained on K to store knowledge in its parameters.Source: §3 · p. 3

4 Preliminaries

The dialogue agent is a pretrained encoder-decoder language model finetuned to generate responses. The section contrasts denoising pretraining and teacher-forced maximum-likelihood training with the hallucination problem attributed to MLE.

4 Preliminaries

  • The proposed encoder-decoder model is pretrained on a knowledge corpus and finetuned on dialogue data to generate informative responses.The model uses BART, pretrained with denoising self-supervision.Source: §4 · p. 3
  • Denoising pretraining reconstructs sampled corpus text k from corrupted text k-hat, with corruption operations including masking, deletion, and infilling.The corpus K may consist of text such as Wikipedia.Source: §4 · p. 3
  • Teacher-forced MLE predicts ground-truth response tokens, but can encourage blind imitation and error accumulation that leads to hallucinating content at inference.The paper proposes MixCL to address this problem.Source: §4 · p. 3

5 MixCL

MixCL samples confusing negative knowledge and applies mixed contrastive learning to reduce the generation probability of negative tokens. Its span-level method combines negative sampling, span extraction and mixing, a contrastive objective, and joint finetuning losses.

5 MixCL

  • MixCL samples confusing negative knowledge, then applies mixed contrastive learning to lower the generation probability of negative tokens.Source: §5 · p. 3, Figure 3 · p. 4

5.1 Negatives sampling

  • Negative knowledge comes from retrieved irrelevant-but-confusing snippets or model-generated hallucinations, with β controlling the sources’ relative contribution.Source: §5.1 · pp. 3–4

Mixed-contrast loss.

  • The mixed objective contrasts positive and negative content at the span level, addressing standard token- or sentence-level losses’ omission of fine-grained spans.Source: §5.2 · p. 4

Extracting spans.

  • To target entity confusions in intrinsic hallucinations and irrelevant fragments in extrinsic ones, MixCL extracts entities with NER and constituents with constituency parsing.Source: Extracting spans · p. 4

Mixing examples.

  • It randomly replaces a positive span with a negative span of the same type, forming a mixed sequence and token-source labels φ.Source: Mixing examples · p. 4

5.3 Optimization

  • Finetuning jointly minimizes LMCL, LLM to mitigate catastrophic knowledge forgetting, and LMLE to optimize response generation, with weights for all three losses.Source: §5.3 · p. 5

6 Experimental setup

The experiments use Wizard-of-Wikipedia with seen and unseen test splits, automatic metrics, and expert human judgments. MixCL is compared with LM-based and KB-based baselines under the described evaluation and implementation settings.

6.1 Datasets and evaluation metrics

  • Wizard-of-Wikipedia contains 22,311 conversations across 1,365 topics with manually labeled turn knowledge; evaluation uses test-seen and test-unseen splits.Source: §6.1 · p. 5
  • Automatic metrics cover text overlap, knowledge and entity F1, and knowledge-selection accuracy; human evaluation scores informativeness, relevancy, factuality, and humanlikeness.Three experts evaluate randomly sampled examples from both test segments.Source: §6.1 · p. 5

6.2 Baselines

  • MixCL is compared with LM-based and KB-based baselines, including KnowledGPT, KnowBART, GPT-2, BlenderBot, KnowExpert, and MSDP.KB methods are evaluated with oracle candidates and with retrieval from the full Wikipedia corpus; models are re-evaluated using official checkpoints and the same evaluation function.Source: §6.2 · p. 5

6.3 Implementation details

  • MixCL uses BART-Large (400M) and Wikipedia as its knowledge corpus, and uses greedy decoding for testing.The loss weights are initialized and scheduled as described in the implementation settings.Source: §6.3 · pp. 5–6

7 Experimental results

On Wizard-of-Wikipedia, MixCL performs strongly against LM-based methods in automatic and human evaluations, while matching or competing with KB-based approaches. Ablations, efficiency analysis, and case studies further examine its components and performance.

7.1 Results of automatic evaluation

  • MixCL achieves the highest scores among LM-based methods on almost all automatic metrics, with 5%–15% relative improvements over previous-best LM-based baselines.It performs well on both response quality and knowledge relevance metrics.Source: §7.1 · p. 6
  • MixCL outperforms two non-LM KB methods by a large margin and KnowledGPT and KnowBART on test seen, but trails the best KB baselines on test unseen, probably due to knowledge forgetting.Under oracle conditions, KB-based methods perform better, but their manually selected knowledge candidates include unavailable ground truth.Source: §7.1 · p. 6

7.2 Results of human evaluation

  • Human evaluation finds that MixCL consistently outperforms LM-based baselines on all metrics and KB-based baselines on some, including relevance, factuality, and humanlikeness.The Table 2 metrics are informativeness, relevance, factuality, and humanlikeness.Source: §7.2 · pp. 6–7, Table 2 · p. 6

7.3 Ablation studies

  • Removing the mixed contrast objective, hard-negative sampling, model-generated negatives, or LM loss degrades performance; MLE-only training substantially reduces KF1.The LM loss is especially useful on unseen topics, while model-generated negatives provide harder examples and the mixed contrast objective particularly supports knowledge relevance.Source: §7.3 · p. 7, Table 3 · p. 7

7.4 Efficiency analysis

  • MixCL is 5× more efficient than state-of-the-art KB methods while achieving competitive response generation performance.Its improvements with model size are more noticeable than those of KB-based methods.Source: §7.4 · p. 7, Figure 4 · p. 7

7.5 Case study

  • Case studies find that MixCL incorporates knowledge more effectively and generates more engaging, human-like responses than baselines.Source: §7.5 · p. 7

8 Conclusions

The paper concludes that MixCL reduces hallucination in conversational language models and performs comparably to state-of-the-art KB-based approaches, with efficiency and scalability advantages. Knowledge forgetting remains a limitation.

8 Conclusions

  • Wizard-of-Wikipedia experiments show MixCL outperforming existing LM-based methods and achieving comparable performance to state-of-the-art KB-based approaches.Human evaluation and ablations also support its effectiveness in reducing hallucination; the paper reports efficiency and scalability advantages.Source: §8 · p. 7
  • Knowledge forgetting remains a limitation, and the authors plan to reproduce their findings for less resource-rich languages.Source: §8 · p. 7

A Results of pilot experiment

The pilot annotation found hallucinations in 51% of BART responses, divided between non-factual intrinsic hallucinations and irrelevant extrinsic hallucinations. The remaining responses were categorized as other.

A Results of pilot experiment

  • 51% of annotated BART responses were hallucinatory: 24% intrinsic and 27% extrinsic.The other 49% were categorized as other.Source: A Results of pilot experiment · p. 10
  • 24% were intrinsic hallucinations, including non-factual statements (11.5%), incorrect entities (9%), and ambiguous statements (3.5%).These categories describe non-factual statements such as incorrect dates or ambiguous claims.Source: A Results of pilot experiment · p. 10
  • 27% were extrinsic hallucinations: out-of-context responses (14.5%), confusion of similar knowledge (6.5%), or non-specific knowledge (6%).Examples include using knowledge inconsistent with the dialogue context or offering generic topic information.Source: A Results of pilot experiment · p. 10
  • The other 49% included qualified responses (32.5%), mechanical knowledge use (8%), no knowledge use (5.5%), and repetition of prior-turn knowledge (3%).Source: A Results of pilot experiment · p. 10

B Implementation details

In the case studies, MixCL produced a response judged strongest for relevance, factuality, and humanlikeness in one example, while other examples show context failures and a non-factual response. The implementation used staged loss weights, negative sampling, and standard training and inference settings.

B Implementation details

  • Training used loss ratios shifted from 0.4/0.3/0.3 to 0.5/0.5/0, eight negatives, AdamW at 2e−5, and up to five epochs; inference averaged 0.05 seconds per example.Training completed within 10 hours on a single GPU; inputs were truncated to 128 tokens and outputs to 64.Source: B Implementation details · p. 10

C Case study

  • In Table 4, MixCL used knowledge to give a more specific and relevant response than baselines that expressed preferences, repeated prior knowledge, or hallucinated.Source: C Case study · p. 10, Table 4 · p. 11
  • In Table 5, MixCL used knowledge appropriately but repeatedly asked about the user’s favorite store, ignoring dialogue context.The authors still describe its response as competent compared with baselines.Source: C Case study · p. 10, Table 5 · p. 11
  • In Table 6, MixCL’s response was best for relevance, factuality, and humanlikeness, while DukeNet and KnowledGPT retrieved irrelevant knowledge after misreading context.MSDP repeated the last utterance.Source: C Case study · p. 10, Table 6 · p. 11
  • Tables 7 and 8 illustrate that MixCL can produce a non-factual response, while it can also use appropriate knowledge distinct from the human response.Source: C Case study · pp. 10–11, Table 7 · p. 12, Table 8 · p. 12

Related papers

Loading 2212.10400v1…