Source-linked AI summary
HalluHard: A Hard Multi-Turn Hallucination Benchmark
Dongyang Fan, Sebastien Delsad, Nicolas Flammarion, Maksym Andriushchenko
TL;DR
LLMs can produce plausible but unsupported claims, with the problem becoming harder to evaluate in evolving multi-turn contexts. The paper introduces HALLUHARD and a citation-based, web-search judging pipeline, finding substantial hallucination and persistent content-grounding difficulty even with web search. It further identifies model capacity, turn position, effective reasoning, and knowledge type as influential factors.
Problem
Existing hallucination benchmarks often use easy domains, constrained formats, or single-turn prompts that diverge from realistic multi-turn, open-ended use.
Method
The paper introduces HALLUHARD and evaluates cited factual claims with an adaptive pipeline that retrieves and reads full-text sources, including PDFs.
Results
Hallucinations remain substantial with web search, while hallucination behavior varies with model capacity, turn position, effective reasoning, and data type.
Takeaways & Limitations
Later-turn error propagation and niche knowledge are especially challenging, whereas effective thinking and stronger web-enabled verification can reduce some hallucinations.
Takeaways & Limitations
Niche facts remain intrinsically difficult because new niche facts continually emerge, limiting the reach of verification-based mitigation.
Abstract
from arXiv · showhide
Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade. We introduce $\textbf{HalluHard}$, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding. We operationalize groundedness by requiring inline citations for factual assertions. To support reliable evaluation in open-ended settings, we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content. Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search ($\approx 30\%$ for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates. Finally, we show that hallucination behavior is shaped by model capacity, turn position, effective reasoning, and the type of knowledge required.
1. Introduction
HALLUHARD targets hallucinations in realistic, multi-turn, open-ended interactions, where context evolves and early inaccuracies can propagate. Its citation-based evaluation finds that frontier models still hallucinate substantially, including with web search.
- Motivation: HALLUHARD addresses the gap left by benchmarks that use easy domains, constrained formats, or simplified single-turn prompts.The benchmark is designed for multi-turn interactions and more challenging task settings.
- Evaluation: Models must support factual claims with explicit citations, enabling verification against retrieved full-text sources, including PDFs.The judge follows citations and reads the referenced material rather than relying only on surface-level matches.
- Evaluation: A hallucination can involve either a reference-grounding failure or a content-grounding failure.The judge distinguishes unsupported citations from claims whose cited sources do not substantiate the generated details.
- Findings: Model capacity, turn position, reasoning effort, and data type significantly affect hallucination rates.The study reports that content grounding remains challenging even when web search is enabled.
- Benchmark: 950 seed questions span 4 challenging task domains in the proposed benchmark.The contribution emphasizes multi-turn hallucination evaluation across four domains.
2. Related Work
Related work distinguishes in-context from in-parameter hallucination and documents recurring factuality problems. However, existing evaluation traditions do not cleanly isolate hallucination propensity or fully characterize its causes.
- Hallucination Definitions: In-context hallucination measures whether outputs are grounded in provided context but is entangled with instruction-following ability.This entanglement may prevent it from cleanly measuring hallucination propensity.
- Hallucination Definitions: In-parameter hallucination evaluates consistency with information encoded in model parameters and is commonly tested with short-form factual prompts or verified long-form claims.World knowledge is often treated as a proxy for in-parameter knowledge acquired from web-scale data.
- Empirical Studies: Prior studies use hallucination evaluations to characterize recurring patterns, including factuality changes on novel knowledge and refusal behavior on unanswerable questions.The cited work links evaluation to empirical understanding of hallucinatory behavior.
3. Why a New Benchmark?
Existing benchmarks and judges are often too easy or too shallow for open-ended, niche, multi-turn factuality evaluation. HALLUHARD responds with citation-checkable claims and evidence retrieval that can inspect full source text.
- Groundedness Operationalization: Wikipedia-based inconsistency does not necessarily establish inconsistency with a model’s training data because models may rely on other parametric sources.This limits what such benchmarks can conclude about in-parameter knowledge.
- Groundedness Operationalization: Verbatim source quotations provide evidence that a model drew from a retrievable document, while an unlocatable citation is treated as reference hallucination.The procedure makes groundedness more directly assessable than assuming cited material reflects the model’s training data.
- Saturated Past Benchmarks: Many factuality benchmarks saturate because they use single-turn prompts with short, indisputable answers that web search can solve easily.SimpleQA reaches 90% accuracy with GPT-4o Search Preview and 95.1% with GPT-5-thinking with web search.
- Saturated Past Benchmarks: Long-form benchmarks often focus on well-documented topics, while hallucination rates rise for entities lacking Wikipedia pages or requiring broader web evidence.This motivates testing niche information that is less comprehensively documented.
- Judge Limitations: Snippet-based judging can misclassify claims when supporting evidence is buried in tables, figures, footnotes, or longer page sections.The limitation is amplified for niche or highly technical statements requiring precise definitions or detailed retrieval.
4. Our Benchmark: HALLUHARD
HALLUHARD is a hard, multi-turn benchmark covering four high-stakes domains, paired with citation-based verification that separately assesses reference and content grounding. Its evaluation uses iterative evidence retrieval and domain-specific judging, including response-level verification for coding tasks.
- Benchmark design: HALLUHARD contains 950 seed questions across legal cases, research questions, medical guidelines, and coding.The first three domains contribute 250 questions each, and coding contributes 200; domain experts review all seed questions.
- Benchmark design: Multi-turn interactions use a user LLM to generate follow-up questions from the conversation history while the target model receives the full dialogue.Initial and follow-up questions include instructions requiring inline citations for factual claims.
- Verification pipeline: The judge verifies each atomic claim by checking both whether its reference is correct and whether the claimed content is grounded in that reference.The pipeline extracts claims, plans iterative web searches, and retrieves full-text sources when needed.
- Evaluation: The hallucination rate is computed as the ratio of hallucinated claims to extracted and verifiable claims.For coding, the evaluation instead reports response-wise hallucination rates.
- Verification pipeline: Coding responses are evaluated as wholes rather than claim by claim because isolated function calls can appear unsupported when definitions are contained in the provided context.The coding judge checks installation steps, imports, and function calls, flagging a response if any of the three hallucination types occurs.
5. Results
HALLUHARD evaluates frontier proprietary and open-weight models across three-turn, citation-grounded conversations. Hallucination varies with turn position, model capacity, reasoning configuration, and grounding quality.
- Evaluation setup: Three-turn conversations average hallucination rates across turns, using sampled claims for legal, research, and medical tasks and response-level evaluation for coding.The legal, research, and medical domains sample up to 1,500 claims each; coding evaluates 600 responses.
- Turn position: Hallucination rates rise across later turns in citation-grounded tasks, with 3–20% of incorrect first-turn references recurring later.The authors attribute this pattern to models conditioning on full conversation histories and repeating earlier erroneous citations.
- Turn position: Coding shows a downward turn-wise hallucination trend because conversations narrow from broad construction requests to focused fixes and edge cases.Narrower tasks leave less room for creative-but-wrong code.
- Model capacity: More capable models hallucinate less: hallucination declines across GPT-5 sizes, improves from GPT-5 to GPT-5.2, and is lowest for Claude Opus.These comparisons are reported across domains and within model families.
- Reasoning: Reasoning generally separates thinking from non-thinking models, but additional reasoning effort does not consistently reduce hallucinations.Reasoning reduces coding hallucination across languages and types apart from Python, while stronger reasoning can produce more claims and references that create additional hallucination risks.
- Grounding: Content-grounding failures are more common than reference failures, and web search reduces reference failures without reliably ensuring that cited sources support generated details.Function-call hallucinations exceed import or installation hallucinations in coding; research-domain verification is further constrained by PDF accessibility.
6. When Models Abstain vs Hallucinate?
A controlled short-form QA experiment distinguishes niche knowledge from fabricated items across five domains. Models tend to hallucinate on niche facts but abstain more on completely fabricated ones, with reasoning effects varying by model family.
- Experimental design: The controlled experiment tests Arts, Geography, History, Research, and Science using fabricated and niche knowledge conditions.It collects 350 questions, with matched templates for the two knowledge types.
- Hallucination versus abstention: Models struggle more with niche facts than fabricated ones, especially in arts and research, where they abstain more on completely fabricated items.The authors report no clear overall domain dependence in hallucinatory behavior.
- Hallucination versus abstention: Niche entities may trigger guessing because traces exist in training data, whereas fabricated items lack a consistent training footprint.The paper presents this as a hypothesis explaining the different hallucination and abstention patterns.
- Reasoning and abstention: Short-form QA elicits substantially more abstention than long open-ended generation, and reasoning changes abstention differently across DeepSeek and GPT families.GPT-5.2-thinking abstains significantly more than GPT-5.2, particularly for niche knowledge.
7. Conclusion
HALLUHARD is introduced as a benchmark for multi-turn, open-ended hallucination evaluation. Its analyses link hallucination to model capability, turn-level error propagation, reasoning, niche knowledge, and citation-supported detail generation.
- Conclusion: HALLUHARD is presented as the first benchmark designed for multi-turn, open-ended generation.
- Conclusion: More capable models tend to hallucinate less, while hallucinations can increase across turns through error propagation.
- Conclusion: Effective thinking can reduce hallucinations, but increasing reasoning effort does not necessarily produce further gains.
- Conclusion: Hallucinations are most prevalent for niche facts and for detailed claims produced with citations.
- Conclusion: Detailed citation-grounding failures are more tractable through additional test-time compute and stronger web verification, whereas continuously emerging niche facts remain intrinsically difficult.
Impact Statement
The work aims to improve hallucination measurement and understanding through a benchmark and analysis, while recognizing risks from narrow or misinterpreted evaluation.
- Impact: HALLUHARD supports more reliable model development by providing a benchmark and analysis for measuring and understanding LLM hallucinations.
- Impact: Benchmark scores may be over-relied upon or misinterpreted, so the authors recommend using HALLUHARD alongside broader evaluations.
A. Model List
The evaluation includes multiple model families with model-specific decoding settings. Most models use temperature 0.0, while GPT and Gemini models use defaults; temperature has negligible influence on hallucination behavior.
- Decoding settings: Most evaluated models use temperature 0.0 to minimize sampling variability.GPT and Gemini models instead use their default temperatures because temperature is not configurable for GPT and default settings are recommended for Gemini.
- Decoding settings: Temperature has negligible influence on hallucination behavior across the tested settings.The authors therefore interpret results primarily as reflecting differences in model grounding and verification capabilities.
- Model coverage: The manuscript provides model details in Table 6.
B.1. Per-turn hallucination rates
Hallucination rates generally rise across later conversation turns in research and medical tasks, while coding shows the reverse trend. Incorrect references can persist across turns, indicating that earlier errors may recur in later responses.
- Per-turn trends: Research questions and medical guidelines show upward hallucination trends across conversation turns, whereas coding shows the reverse trend.The broader per-turn results are plotted in Figure 9.
- Per-turn trends: Figure 9 reports per-turn hallucination rates across all task domains.
- Coding domain: Hallucination rates are lowest in Python across the evaluated programming languages.The authors present these rates by language in Table 8.
C. Templates and prompts
The appendix specifies how multi-turn questions, citations, claim extraction, and grounding judgments are generated and evaluated. Its templates emphasize explicit citation linkage and strict verification of citation metadata.
- C.1. Generation prompts: Generated answers must place inline citations after factual or externally sourced claims and avoid uncited reference lists.The prompts require authorities to be cited at the point of use and every reference-list item to appear inline at least once.
- C.1. Generation prompts: Multi-turn prompts use conversation history to generate natural, engaging follow-up questions grounded in prior dialogue.The target model receives the accumulated conversation and a newly generated question at each later turn.
- C.2. Extractor prompts: The extractor returns atomic legal references with their type, cited proposition, reference name, and attributed holding or description.It skips naked references, unsupported authorities, grouped references, and meta-statements.
- C.2. Extractor prompts: End-of-response citations are extracted only when a clear linkage maps them to a body proposition.Accepted mappings include explicit markers, unambiguous short forms, or statements that the citation supports the preceding proposition.
- C.3. Evaluation prompts: Reference grounding requires an online authority record whose citation metadata matches exactly, with no partial credit for incorrect or unverifiable fields.The checks cover cases, statutes, rules, and secondary sources, including identifiers such as pages, sections, dates, and jurisdictions.
- C.3. Evaluation prompts: The evaluation prompts distinguish reference grounding from content grounding when judging whether a citation exists and supports the stated claim.Reference grounding concerns the authority and its metadata; content grounding concerns whether the described holding or text is supported.
D.1. Research questions
The research-question evaluation compares the proposed judge with humans and other automatic judges using claim extraction and grounding decisions. The judge performs strongly overall, while disagreements remain around evidence availability and strictness of interpretation.
- Claim extraction: The automatic extractor identifies more atomic claims than human judges and made fewer manually checked extraction mistakes.These comparisons are summarized in Table 11.
- Evaluation quality: The proposed judge achieves the highest content-grounding agreement among the automatic judges.The evaluation compares claim-wise judgments with human evaluators in Table 2.
- Evaluation quality: Approximately 88% agreement between human annotators and the proposed judge supports its reliability.The judge achieves the next-highest agreement after the human pair and outperforms the OpenAI web-search judge and SAFE on the reported comparison.
- Sources of disagreement: Human and automatic judges disagree over whether conceptual paraphrases count as content grounded.Some judgments require terminology or wording to appear explicitly, while others accept clearly supported underlying ideas.
- Sources of disagreement: The proposed judge can reject references that humans accept when bibliographic metadata is ambiguous or strict checks expose small mismatches.The paper gives a page-range mismatch as an example of reference failure.
- Judge comparison: Full-text retrieval helps evaluate claims requiring broader context than search snippets provide.The authors attribute half of the cases where their judge is correct and SAFE is not to insufficient contextual evidence available to SAFE.
- Validation: A human annotator independently reviewed 100 machine-judged claims to assess agreement with the proposed judge.The review used internet access and is summarized in Table 12.
- Sources of disagreement: Legal-domain error patterns resemble those in research questions and reflect intrinsic difficulty shared with human annotation.