Source-linked AI summary
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, Siva Reddy
TL;DR
Knowledge-grounded conversational systems hallucinate, but it is unclear whether the problem lies in training datasets, models, or both. The paper conducts a human study of benchmark and model responses, finding that benchmarks contain more than 60% hallucinations and models generally amplify them. These findings question the quality of existing datasets and models and motivate careful dataset inspection and trustworthy-system design.
Problem
Knowledge-grounded conversational models produce factually incorrect statements, while prior work has focused on ameliorating models without auditing conversational benchmarks.
Method
The paper conducts human annotation of three knowledge-grounded benchmarks and responses generated by several state-of-the-art conversational models.
Results
More than 60% of responses in the three datasets are hallucinated, and models generally produce an even larger proportion of hallucinations than the training data.
Takeaways & Limitations
The findings question the quality and training suitability of current conversational datasets and the robustness of existing models.
Takeaways & Limitations
The audited benchmarks were collected through AMT and may contain disturbing or toxic examples that pose psychological risks to annotators.
Abstract
from arXiv · showhide
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of >60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.
1 Introduction
The paper investigates whether hallucinations in knowledge-grounded conversational systems originate from datasets, models, or both. Auditing three benchmarks, it finds that hallucinated responses are widespread and that trained models can amplify them.
- Knowledge-grounded conversational models generate factually incorrect statements, while prior work primarily attempts to ameliorate models rather than audit benchmarks.
- Crowd-sourced benchmarks may contain hallucinations because workers ignore knowledge snippets, rely on personal knowledge, or prioritize informativeness over faithfulness.
- Neural conversational models may replicate or amplify dataset hallucinations because they optimize for the data distribution rather than faithful outputs.
- The study annotates responses from Wizard of Wikipedia, CMU-DOG, and TOPICALCHAT using discourse and attribution-oriented coding frameworks.
- More than 60% of responses in the three datasets are hallucinated, mainly through subjective information and unsupported objective facts.
- Model-generated responses contain an even larger proportion of hallucinations than the training data, raising questions about dataset quality and model robustness.
2 Hallucinations in Benchmarks
The human evaluation defines attribution and discourse categories, then measures hallucination across three knowledge-grounded benchmarks. It finds substantial benchmark contamination by hallucinated content, commonly expressed through disclosure and edification.
- The study evaluates three crowdsourced benchmarks—Wizard of Wikipedia, CMU-DOG, and TOPICALCHAT—whose dialogues pair topical conversations with turn-relevant knowledge snippets.
- Responses are classified as entailment, hallucination, generic, or uncooperative according to whether their information is supported by the knowledge snippet and fits conversational cooperation.
- The VRM scheme labels responses by speech act, including disclosure, edification, advisement, confirmation, question, and acknowledgement.
- Expert annotation achieved Fleiss’ Kappa scores of 0.89 for BEGIN and 0.78 for VRM on Wizard of Wikipedia, indicating substantial agreement.
- More than 60% of benchmark responses are hallucinated overall: Wizard of Wikipedia reaches 74.4% in 4K samples, CMU-DOG has 61.4% pure hallucination, and TOPICALCHAT has 63.9%.
- Disclosure is the dominant hallucination mode across WOW, CMU-DOG, and TOPICALCHAT at 50.9%, 56.2%, and 61.5%, respectively.
- Edification is also common in hallucinated responses, reflecting unsupported factual additions, whereas entailment responses show more than 70% edification.
3 Hallucination Amplification in Models
The study tests whether conversational models amplify hallucinations in benchmark-trained generation. Across models, hallucinations generally increase, while CTRL reduces hallucination at the cost of uncooperative, often extractive responses.
- Model setup: The authors fine-tune GPT2, DoHA, and CTRL on the benchmarks and evaluate generated responses with nucleus sampling at p = 0.6.GPT2 concatenates knowledge and history; DoHA uses two-view attention; CTRL uses control tokens targeting less subjective, more entailed content.
- Amplification: Models generally amplify hallucinations relative to human gold responses, except CTRL on WOW.The analysis measures the percentage of each response class across the three benchmarks.
- Amplification: GPT2 increases full hallucination by 19.2% on WOW, 15% on CMU-DOG, and 15.1% on TOPICALCHAT.It simultaneously reduces entailment by 17.4%, 9.3%, and 11.9%, respectively.
- Model trade-offs: CTRL hallucinates least but produces many uncooperative responses by copying large portions of the evidence or the entire evidence.Its extractive behavior is consistent with optimization for overlap with the knowledge.
- Possible contributors: The authors identify exposure bias and maximum-likelihood training on noisy data as possible contributors to model deficiencies.Teacher forcing can aggravate error propagation, while noisy references can produce unstable behavior.
- Hallucination strategies: DoHA and GPT2 predominantly amplify disclosure, whereas CTRL relies on edification and increases uncooperative responses.CTRL’s uncooperative responses rise by 33.5%, 12.9%, and 20.2% in the reported benchmark comparisons.
4 Related Work
Prior hallucination research spans neural machine translation and summarization, while hallucination in knowledge-grounded dialogue remains a relatively new research problem.
- Research scope: Recent hallucination studies address neural machine translation and summarization, while knowledge-grounded dialogue has received attention more recently.The passage characterizes knowledge-grounded neural dialogue hallucination as a nascent research problem.
5 Conclusion
The paper finds hallucination to be widespread in both dialogue benchmarks and models. It reports that models trained on these benchmarks can amplify hallucinations and calls for higher-quality data and trustworthy system design.
- Conclusion: Hallucination is prevalent in the three examined dialogue benchmarks and in models trained on them.The analysis identifies disclosure and edification as the most common human hallucination strategies.
- Implications: The findings motivate clean, high-quality data releases and careful design of trustworthy conversational systems.The authors also advocate inspecting dataset samples before use or public release.
Impact Statement & Ethics
The impact statement addresses risks to annotation workers and risks from deploying conversational systems trained on low-quality data. It emphasizes that trustworthy deployment requires attention to both benchmark quality and model behavior.
- Annotation Risks: AMT annotation tasks may expose workers to racist, expletive, toxic, or otherwise disturbing conversational content.The authors acknowledge possible psychological distress and provide a warning allowing workers to skip distressing tasks.
- Deployment Risks: Low-quality benchmark data may harm conversational-model deployment, especially in high-stakes domains.The authors state that even robust models remain bounded by data quality and that enforcing faithfulness is non-trivial.
A Datasets
The analysis covers three widely used knowledge-grounded conversational benchmarks: Wizard of Wikipedia, CMU-DoG, and TopicalChat. Each pairs dialogue responses with source knowledge relevant to the conversation.
- Wizard of Wikipedia contains wizard–apprentice dialogues centered on a topic, with Wikipedia snippets provided to the wizard at each turn.The wizard communicates topic information while the apprentice seeks information.
- Wizard of Wikipedia includes 82,722 training, 8,800 validation, and 8,690 test grounded responses after filtering unselected knowledge passages.
- CMU-DoG contains movie-domain conversations, with each response grounded in a Wikipedia section.Workers either persuade others to watch the movie or discuss the document content.
- TopicalChat covers varied topics and supplies facts from Reddit, Wikipedia, and news articles under symmetric and asymmetric knowledge-access settings.It contains 292,215 training, 23,601 validation, and 23,623 test grounded responses.
B Implementation Details
The implementation trains or uses GPT2, DoHA, and CTRL models across the benchmark datasets, using specified optimization, selection, and decoding procedures. The annotation materials also include VRM definitions and expert-annotation agreement information.
- GPT2 training uses Adam, dropout, batch size 32, learning rate 6.25 × 10^-5, linear decay, and a three-utterance history limit.Training early-stops at epochs 7, 8, and 8 for WOW, CMU-DOG, and TOPICALCHAT, respectively.
- DoHA uses a public CMU-DoG model and newly trained WOW and TopicalChat models following the reported training procedure.The WOW and TopicalChat models require average runtimes of 5 and 10 hours, respectively.
- CTRL is reimplemented with Hugging Face Transformers and PyTorch Lightning using reported training details and author discussions for accuracy.
- All models use the validation set for best-model selection, Nvidia V100 32GB training, and nucleus sampling with p=0.6 at inference.
F AMT Human Annotation
The AMT annotation protocol hierarchically identifies unsupported information, classifies its type, and distinguishes supported, faithful, or generic content. Quality controls combine qualification, piloting, inspection, feedback, and rejection procedures.
- The interfaces support annotation of BEGIN and VRM categories, while VRM definitions are presented with examples.
- Annotators first judge whether a response contains information unsupported by the source, then classify the unsupported information type.Possible types include unsupported opinion, unsupported fact, personal experience, feelings, advice, and acknowledgements with human information.
- Responses without hallucinations are further classified as faithful or generic, and faithful responses are assessed for cooperation with the human’s response.
- AMT workers must pass a 14-question qualification test, be located in the United States or Canada, and complete a monitored pilot round.
- Poor-quality annotations are rejected during the main stage, and repeated rejections can block workers indefinitely; four workers annotated 4,000 responses.
- Workers were paid approximately 18–20 USD per hour and informed that released research annotations would keep their worker IDs anonymous.
H Limitation
The work audits hallucinations in gold and machine-generated responses rather than investigating their root causes. Its analysis is limited to English benchmarks, leaving multilingual datasets and model-level causes for future study.
- The main contribution is a data-quality audit examining hallucination types in both gold and machine-generated responses.
- The study explicitly does not investigate the root causes of hallucination in models.
- The analysis is restricted to English benchmarks.
- Future work can examine model-level causes and hallucination in multilingual datasets.
L Machine-generated Responses
This section presents breakdowns of BEGIN classes and VRM speech acts across machine-generated responses, alongside response examples and hallucinated benchmark outputs. The materials cover Wizard of Wikipedia, CMU-DOG, and TOPICALCHAT.
- Table 6 provides generated-response samples from GPT2, DOHA, and CTRL on Wizard of Wikipedia and CMU-DOG.
- Figures 6–8 break down BEGIN classes and VRM speech acts for machine-generated responses from WoW, CMU-DOG, and Topical.The figures correspond respectively to WoW, CMU-DOG, and Topical machine-generated responses.
- Table 5 illustrates the BEGIN breakdown and different VRM linguistic phenomena for responses from Wizard of Wikipedia.
- Table 7 presents hallucinated responses from Wizard of Wikipedia, CMU-DOG, and TOPICALCHAT, marking hallucinated content in red.