PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data

Katja Filippova

arXiv:2010.05873v1cs.CL

TL;DR

Noisy training data can cause neural text generators to produce fluent but unsupported content. The paper estimates each example’s noise and supplies it as a controllable input attribute without changing the model architecture. On WikiBio, the approach improves faithfulness without sacrificing fluency or coverage and improves upon prior work.

  • Problem

    Heuristically mined parallel data can contain output phrases unsupported by the input, causing neural generators to learn hallucinations.

    Source: §1 · p. 1

  • Method

    The method estimates each source-target pair’s noise and prepends a categorical hallucination degree to the input while leaving the model architecture unchanged.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Human evaluation on WikiBio shows significantly improved faithfulness with no loss in fluency or coverage, improving upon prior work.

    Source: §4.3 · p. 4, §6 · p. 5

  • Takeaways & Limitations

    Hallucination amount can be controlled on noisy data without dismissing inputs or modifying the encoding and decoding architecture.

    Source: §1 · p. 2, §6 · p. 5

    Word-overlap noise estimation assumes source and target share a language and have comparable term distributions, while PARENT can miss paraphrases or reward unsupported reference matches.

    Source: §3.1 · p. 3, §4 · p. 3

Abstract

from arXiv · show

Neural text generation (data- or text-to-text) demonstrates remarkable performance when training data is abundant which for many applications is not the case. To collect a large corpus of parallel data, heuristic rules are often used but they inevitably let noise into the data, such as phrases in the output which cannot be explained by the input. Consequently, models pick up on the noise and may hallucinate--generate fluent but unsupported text. Our contribution is a simple but powerful technique to treat such hallucinations as a controllable aspect of the generated text, without dismissing any input and without modifying the model architecture. On the WikiBio corpus (Lebret et al., 2016), a particularly noisy dataset, we demonstrate the efficacy of the technique both in an automatic and in a human evaluation.

1 Introduction

Heuristic data collection introduces unsupported target phrases, leading neural generators to hallucinate fluent but unfaithful text. The paper proposes controlling this behavior with an input-side hallucination degree while retaining noisy data and the model architecture.

  • Training corpora built with heuristic rules contain noise that neural models can learn as data artifacts.Source: §1 · p. 1
  • Hallucinations here are generated content unfaithful to the input, primarily caused by imperfect training-data quality.Source: §1 · p. 1
  • The proposal adds a controllable hallucination degree to the input without modifying the encoding or decoding architecture.Source: §1 · p. 2
  • The noise degree is estimated per training example using the difference between language-model and conditional-generator losses during forced-path decoding.Source: §1 · p. 2
  • On WikiBio, which contains extra information in 62% of references, the models outperform Liu et al. (2018), and human evaluation confirms faithfulness need not trade off against coverage.Source: §1 · p. 2

2 Controlling Hallucinations

The paper treats hallucination amount as a controllable generation attribute by prepending a noise degree to each input. This extends the input-side control paradigm beyond attributes such as length, sentiment, style, and theme.

  • Controlled generation adds a target attribute as an additional input so inference exposes a corresponding control knob.Source: §2 · p. 2
  • The paper adds a categorical hallucination degree as a prefix to each input, allowing the model to control unsupported information in outputs.Source: §2 · p. 2
  • The same input can be paired with three hallucination degrees, producing corresponding outputs from the controlled WikiBio model.Source: §2 · p. 2

3 Detecting Hallucinations in the Training Data

The paper estimates hallucination in source-target pairs with two complementary methods: word overlap and a comparison of conditional versus unconditional language models. These scores are discretized into tags for controlling generation.

3 Detecting Hallucinations in the Training Data

  • Five quantile-based hallucination tags are learned during training, and inference uses the lowest-hallucination tag, hal 0.Each source-target pair receives a score hal ∈[0, 1], which is converted into five intervals covering 20% of the range.Source: §3 · p. 2

3.1 Word Overlap

  • Word overlap estimates unsupported target content by comparing source and target words, but applies only when both are in the same language and sufficiently similar.The authors note that this condition can vary within a dataset, such as across more abstractive and extractive news headlines.Source: §3.1 · pp. 2–3

3.2 When a LM Knows Better

  • The LM-based method compares an unconditional target-only LM with a source-informed conditional LM trained on the same data and architecture.The conditional model is denoted LMx, while LM predicts from the decoded prefix alone.Source: §3.2 · p. 3
  • A target token is treated as hallucination evidence when LMx predicts it incorrectly but LM assigns it a smaller loss, normalized by target length.This reflects the intuition that source-supported tokens favor LMx, whereas unsupported but prefix-plausible tokens can favor LM.Source: §3.2 · p. 3

4 Experiments

The experiments test controlled hallucination tags on WikiBio using automatic and human evaluations, comparing the default model with noise-aware and prior systems. The controlled models substantially improve faithfulness while preserving coverage and fluency.

4 Experiments

  • The experiments compare an unmodified seq2seq model with versions conditioned on Word Overlap or LM-based noise annotations, alongside Liu et al. (2018) and Tian et al. (2019).Automatic evaluation uses BLEU and PARENT, with PARENT additionally checking entailment by the input table.Source: §4 · p. 3, §4.1 · p. 4
  • Human evaluation measures fluency, faithfulness, and coverage so that concise but unsupported or templated outputs are not favored.Faithfulness is counted as Faithful or Mostly faithful, while coverage counts table-cell information expressed in the generated sentence.Source: §4 · pp. 3–4

4.2 Removing Noisy Examples

  • Training only on the cleanest 20% of examples yields more precise but low-recall and low-BLEU outputs, so this variant is not evaluated by humans.The clean-data model is not competitive with the other systems.Source: §4.2 · p. 4, Table 1 · p. 4

4.3 Results

  • Automatic PARENT-F scores are similar across models, but Liu et al. (2018) trades lower precision for its best PARENT-F, motivating human evaluation for final conclusions.PARENT may miss paraphrases or reward reference matches unsupported by the table, so the authors complement it with human judgments.Source: §4 · p. 3, §4.3 · p. 4
  • All hallucination-reducing models achieve 93-96% fluency and outperform models that do not address hallucinations.The evaluation first checks naturalness because perfect faithfulness and coverage can be obtained by concatenating input fields.Source: §4.3 · p. 4, Table 2 · p. 5
  • The controlled generators improve faithfulness over LIU-ET-AL. and the default SEQ2SEQ by 15-25 points, while the LM-based version matches default seq2seq coverage.Compared with TIAN-ET-AL., the controlled methods are comparable or better on strictly faithful ratings, while TIAN-ET-AL. has lower coverage.Source: §4.3 · p. 4, Table 2 · p. 5

5 Discussion

The discussion shows that hallucination control can improve faithfulness without sacrificing fluency or coverage, while different noise estimators and evaluation metrics impose important trade-offs. The controllable framework also supports deliberately increasing unsupported information, but its estimator choices and metric behavior leave clear scope for refinement.

Main findings

  • Human evaluation significantly improved faithfulness at no loss in fluency or coverage, while the WikiBio results improved upon prior work.The discussion also reports superior performance to Liu et al. (2018) on the noisy WikiBio dataset.Source: §5 · p. 5

Noise estimation

  • The overlap estimator may be preferable with comparable source-target vocabularies, whereas halLM makes no data assumptions and performed better in human evaluation.halLM also permits paraphrasing and straightforward inferences that overlap can miss.Source: §5 · p. 5

Possible extensions

  • Alternative noise detectors, including embedding-space similarity and word alignment, may perform better within the controlledhallucination framework.The authors present these as possible extensions rather than evaluated improvements.Source: §5 · p. 5

Controllable generation

  • The hallucination knob can increase unsupported information for applications requiring fluent adversarial sentences, not only suppress it.Figure 2 shows hallucinations increasing with the knob value.Source: §5 · p. 5

Evaluation caveat

  • All tested models outperform Liu et al. (2018) on PARENT and human evaluation but do not approach its BLEU score, whose suitability for generation beyond MT is questioned.The authors do not explain the BLEU discrepancy and cite Reiter (2018) on BLEU's limitations.Source: §5 · p. 5

Length and coverage

  • 17.2, 17.8, and 18.7 sentencepiece tokens rank halLM, halWO, and Tian et al. (2019) consistently with their input-field coverage.This comparison suggests that reducing length alone does not establish the desired informativeness trade-off.Source: §5 · p. 5

6 Conclusions

The paper proposes controlling hallucinations caused by noisy training data by informing the model about each source-target example's noise level, without changing the model architecture. On WikiBio, this reduces hallucinations without coverage loss and improves human-rated faithfulness without sacrificing fluency.

Takeaway

  • The approach treats hallucinations caused by training-data noise as a controllable property of generated text.The conclusion presents this as a simple but powerful idea.Source: §6 · p. 5

Contribution

  • The method reduces hallucinations at no coverage cost by informing the model how noisy each source-target example is, without changing the model architecture.It uses two proposed ways of detecting training-data noise.Source: §6 · p. 5

Evaluation

  • Human evaluation shows significantly improved faithfulness with no loss in fluency or coverage.The reported results improve upon prior work on noisy WikiBio.Source: §6 · p. 5
Loading 2010.05873v1…