PaperlayerView exact revision on arXiv

Source-linked AI summary

Diving Deep into Modes of Fact Hallucinations in Dialogue Systems

Souvik Das, Sougata Saha, Rohini K. Srihari

arXiv:2301.04449v1cs.CLcs.AI

TL;DR

Fact hallucinations in KG-grounded dialogue systems introduce unsupported entities, but entity-level detection remains underdeveloped and human annotation is difficult. The paper identifies fine-grained hallucination modes, creates FADE through perturbation strategies, and evaluates detection baselines. The resulting models achieve strong detection performance and generalize zero-shot to established benchmarks, although extrinsic-trained models transfer poorly.

  • Problem

    Entity-level hallucination detection in KG-grounded dialogue systems is underdeveloped, while human annotations are expensive and often inaccurate.

    Source: §1 · p. 1

  • Method

    The authors analyze hallucination modes through human feedback, create FADE with fine-grained perturbation strategies, and train multiple pre-trained detection baselines.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The utterance-level classifier reaches 90.75% F1 and the entity-level classifier reaches 75.59% F1; models trained on FADE achieve over 90% F1 and outperform current baselines by 10.16% and 17.5% on two tasks.

    Source: §7 · p. 7, §9 · p. 9

  • Takeaways & Limitations

    FADE’s perturbation strategies support entity-level hallucination detection and robust zero-shot evaluation on BEGIN and FaithDial.

    Source: §9 · pp. 8–9

    Models trained on extrinsic datasets do not generalize well to benchmark datasets whose hallucinations are mostly related to the provided evidence.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Knowledge Graph(KG) grounded conversations often use large pre-trained models and usually suffer from fact hallucination. Frequently entities with no references in knowledge sources and conversation history are introduced into responses, thus hindering the flow of the conversation -- existing work attempt to overcome this issue by tweaking the training procedure or using a multi-step refining method. However, minimal effort is put into constructing an entity-level hallucination detection system, which would provide fine-grained signals that control fallacious content while generating responses. As a first step to address this issue, we dive deep to identify various modes of hallucination in KG-grounded chatbots through human feedback analysis. Secondly, we propose a series of perturbation strategies to create a synthetic dataset named FADE (FActual Dialogue Hallucination DEtection Dataset). Finally, we conduct comprehensive data analyses and create multiple baseline models for hallucination detection to compare against human-verified data and already established benchmarks.

1 Introduction

Knowledge-grounded dialogue models can produce factually unfaithful responses that introduce unsupported entities or information, risking conversational usefulness. The paper addresses this gap by categorizing hallucinations, creating synthetic training data, and evaluating detection models.

Motivation

  • Unsupported responses can make knowledge-grounded conversations uninformative and jeopardize conversational flow despite large models’ strong linguistic abilities.Source: §1 · p. 1
  • Human annotation is costly and error-prone because annotators may lack incentive, show personal bias, or attend poorly to the provided knowledge.Source: §1 · p. 2

Contributions

  • The paper extends fact hallucination in KG-grounded dialogue systems into eight categories through human analysis of generated responses.The categories are intended to capture more nuanced hallucination behavior than previously studied broad distinctions.Source: §1 · p. 2
  • The authors design perturbation strategies that simulate hallucination modes and build the FADE synthetic dataset for entity-level hallucination detection.Source: §1 · p. 2
  • The study compares multiple pre-trained detection baselines across component and mixed datasets and tests zero-shot generalization on BEGIN and FaithDial.Source: §1 · p. 2

2 Different Modes of Hallucination in KG-grounded Dialogue Systems

The paper defines hallucination as dialogue content unsupported by valid KG paths, subdivides it into fine-grained modes, and studies their occurrence in generated responses. Human feedback finds that extrinsic-soft hallucination dominates, while intrinsic-hard hallucination is least common.

2.1 Background

  • A dialogue system hallucinates when generated text is not supported by a valid path in the k-hop KG subgraph anchored around the context entity.Source: §2.1 · p. 2

2.3 Definitions

  • The paper subdivides KG hallucinations into extrinsic-soft, extrinsic-hard, extrinsic-grouped, intrinsic-soft, intrinsic-hard, intrinsic-repetitive, and history-corrupted intrinsic or extrinsic modes.The extrinsic modes introduce unsupported spans, whereas intrinsic modes misuse entities or triples within the provided subgraph; history corruption is influenced by hallucinated conversational history.Source: §2.3 · pp. 2–3

2.4 Human Feedback Analysis

  • The human-feedback study evaluates GPT2 responses generated with greedy, beam-search, and nucleus-sampling strategies, retaining annotations with moderate-to-high agreement after rejecting poor-quality judgments.The study sampled responses across four decoding settings and reported an average Krippendorff alpha of 0.74 on remaining annotations.Source: §2.4 · p. 3
  • Extrinsic-soft hallucination is the dominant form, supporting the observation that language models often generate entities similar to the gold entity.Source: §2.4 · p. 3
  • Intrinsic-hard hallucination is the least frequent type, suggesting that models usually retain some connection to the supplied KG triples.Source: §2.4 · p. 3

3 Dataset Creation

FADE combines component datasets generated through extrinsic and intrinsic entity perturbations with mixed datasets representing multiple hallucination types. The construction uses retrieval and similarity-based selection, then analyzes dataset composition and verifies perturbation quality with human annotations.

Dataset Construction

Extrinsic Perturbation

  • Extrinsic perturbation retrieves candidate entities from type-specific BM25 indexes built from entity-associated KG triples.Each entity is represented through indexed documents formed by concatenating its KG triples, and candidate scores average BM25 results across subject, relation, and object queries.Source: §3.1 · p. 4

Intrinsic Perturbation

  • Intrinsic perturbation ranks entities from the original entity’s 1-hop graph using hybrid retrieval that combines GPT-2 embeddings, similarity scoring, and relation rarity.The query-term embeddings and unigram probabilities are used to score KG triples before candidate selection.Source: §3.1 · p. 4

Dataset Analysis

  • The base dataset contains 77,430 data points, while valid KG paths limit perturbed examples to roughly 12,000–23,000 extrinsic and 9,000–18,000 intrinsic utterances.Extrinsic datasets include approximately 550–11,300 utterances with multiple perturbations.Source: §3.2.1 · p. 5, Table 4 · p. 5, Table 5 · p. 5
  • Extrinsic-soft hallucinations mainly involve PERSON entities and closely affiliated substitutions, whereas extrinsic-hard hallucinations have a more uniform NER distribution.Intrinsic-hard hallucinations include unusual relations such as Country of Origin and Country of Nationality.Source: §3.2.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

Dataset Validation

  • Human verification of 600 sampled mixed-dataset examples used at least three annotators per example and additional graduate-student checks of perturbation adherence.Source: §3.4 · p. 6

4 Task

The paper defines two hallucination-detection tasks: classifying whether an utterance is hallucinated and locating hallucinated spans within it.

Task Definition

  • Utterance classification predicts whether the current utterance is hallucinated given dialogue history, knowledge triples, and the utterance.Source: §4 · p. 6
  • Token classification performs sequence labeling over the current utterance to identify hallucinated spans using the same dialogue and knowledge inputs.Source: §4 · p. 6
  • The task formulation uses dialogue history D, knowledge triples K_n, and current utterance x_n+1 as inputs.Source: §4 · p. 6

5 Baseline Models

The baseline systems fine-tune pre-trained transformer encoders for utterance-level hallucination classification and token-level span detection.

Model Families

  • The baselines use BERT, XLNet, and RoBERTa to detect hallucinations from contextual inputs and embedded world knowledge.These models are intended to identify self-contradictory or anti-commonsense content.Source: §5 · p. 6

Utterance Classifier

  • The utterance classifier encodes dialogue history and knowledge as one sequence, the current utterance as another, and predicts a binary hallucination label.Max pooling over final hidden states feeds a tanh-activated MLP trained with cross-entropy.Source: §5 · p. 6

Token Classifier

  • The sequence classifier encodes the inputs jointly and predicts BILOU labels through a tanh-activated MLP.The five labels are B, I, L, O, and U, and training uses cross-entropy.Source: §5 · p. 6

6 Experimental Setup

Experiments evaluate compact pre-trained transformer baselines across component and mixed datasets using classification, calibration, and imbalance-sensitive metrics.

Model Configuration

  • The experiments compare BERT-base-uncased (110M), RoBERTa-base (125M), and XLNet-base-cased (110M) configurations.Larger versions are omitted because scaling them in production is costly.Source: §6 · p. 6

Architecture Variants

  • The best architecture uses four dialogue-history turns, max pooling, no knowledge/current-utterance hidden-state concatenation, and an MLP sequence tagger.Source: §6 · p. 6

Evaluation Metrics

  • Evaluation reports precision, recall, and F1 for sequence tagging, plus accuracy, precision, recall, F1, and AUC for utterance classification.G-Mean and Brier Skill Score additionally assess sensitivity-specificity balance and probabilistic error.Source: §6 · pp. 6–7

Benchmark Protocol

  • Tables 6 and 7 report test benchmarks for component and mixed datasets using models trained on 25% of the total data.Source: Table 6 · p. 7, Table 7 · p. 7

7 Results and Discussion

FADE-trained models detect hallucinations effectively on human-verified data, but transfer less reliably to benchmarks whose hallucinations differ from the paper’s target phenomena.

  • Among component datasets, utterance-level classifiers outperform token taggers in F1, while intrinsic hallucinations are generally easier to detect than extrinsic ones.Extrinsic-soft hallucinations are the most frequent type but receive the lowest F1 score.Source: §7 · p. 7
  • Among mixed datasets, XLNet fine-tuned on the extrinsic+ dataset achieves the best F1 scores, whereas the observed dataset remains difficult because it contains many extrinsic-soft cases.Source: §7 · p. 7
  • F1 scores exceed 90% on human-verified data, outperforming the current baseline by 10.16% and 17.5% across the two tasks.The comparison uses a pre-trained model with fewer parameters.Source: §7 · p. 7, Table 8 · p. 7
  • FADE-trained models underperform the best baseline by 6% on BEGIN and 10.17% on FaithDial because those benchmarks contain fundamentally different hallucinations.The benchmark hallucinations are mostly related to the provided evidence, unlike the paper’s target fact hallucinations.Source: Table 9 · p. 7, §7 · p. 8
  • Approximately 25% of the training split is optimal for generalization to benchmark datasets.The estimate comes from evaluating models trained on 10%–30% of the training data.Source: §7 · p. 8, Figure 7 · p. 8
  • The models identify misplaced or unexpected entities in visualized examples, but sometimes fail when conversational history is convoluted.Figure 6 contrasts entity-level detection, unexpected response content, and failures caused by complicated history.Source: §7 · p. 8, Figure 6 · p. 8

8 Related Work

Prior work addresses dialogue hallucination through controlled generation, token-level criticism, knowledge retrieval, and benchmark evaluation, but entity-level detection remains insufficiently established.

  • Existing systems mitigate hallucination with control tokens, token-level critics, or knowledge-retrieval mechanisms during generation.Source: §8 · p. 8
  • Benchmarks including BEGIN, DialFact, FaithDial, and AIS provide evaluation frameworks, but their ability to detect entity-level hallucination is unknown.Source: §8 · p. 8
  • This work extends earlier extrinsic-versus-intrinsic analyses by subdividing hallucinations into finer-grained categories.Earlier work examined two broad circumstances in which language models behave unfaithfully to the provided KG.Source: §8 · p. 8

9 Conclusion

The paper analyzes nuanced entity-level hallucinations, creates FADE through targeted perturbations, and demonstrates strong detection performance with partial benchmark generalization.

  • Human feedback analysis reveals more nuanced hallucination modes than the straightforward cases previously studied in KG-grounded dialogue.Source: §9 · p. 8
  • FADE uses fine-grained perturbation strategies to mimic observed real-world hallucinations in synthetic datasets.Source: §9 · p. 8
  • The detector achieves 75.59% F1 for hallucinated-entity prediction and 90.75% F1 for utterance-level hallucination classification.Source: §9 · p. 9
  • The models generalize in zero-shot predictions on BEGIN and FaithDial, while the authors identify more sophisticated perturbations as a future extension.Source: §9 · p. 9

Limitations

The paper reports limitations involving classifier disagreement and weak transfer from extrinsic-hallucination training to benchmarks dominated by evidence-related hallucinations.

  • Token-level and utterance-level classifiers can produce contradictory results, although this occurs in only a small percentage of examples.Source: Limitations · p. 9
  • Models trained on extrinsic datasets do not generalize well to benchmarks whose hallucinations are mostly related to the provided evidence.Source: Limitations · p. 9

F Supplementary results

The supplementary results report model benchmarks on component and mixed FADE datasets, with models trained on 25% of the total data. They also document the BERT and XLNet hyperparameters used in these experiments.

Related papers

Loading 2301.04449v1…