PaperlayerView exact revision on arXiv

Source-linked AI summary

LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

Ruizhi Zhang, Jinwei Chen, Xiangju Lu, He Yan, Mo Yu, Junmin Zhu, Wei Zhang

arXiv:2608.18082v1cs.CL

TL;DR

Long-context summarization still faces hallucinations, while existing benchmarks and evaluators do not adequately cover very long, varying contexts. LongNovel introduces a bilingual, multi-scale novel benchmark with eight hallucination types, and experiments show that current models struggle as context length increases.

  • Problem

    Existing hallucination evaluators and benchmarks inadequately detect long-context errors or assess robustness across context lengths up to 100k tokens.

  • Method

    LongNovel combines four context lengths from 16k to 100k tokens, eight hallucination types, human-written summaries, and multi-model arbitration with entity-referenced hallucination construction.

  • Results

    LongNovel is a challenging benchmark, with model performance generally declining as context length increases.

  • Takeaways & Limitations

    LongNovel provides a benchmark for developing and evaluating more reliable long-context hallucination detection models.

  • Takeaways & Limitations

    The study evaluates open-source models with up to 32B parameters, and its findings’ applicability beyond novel domains remains unvalidated.

Abstract

from arXiv · show

Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge. Long novels are better suited than news or papers for researching these hallucinations, due to their intrinsic information and detailed descriptions of events and dialogues. However, current research lacks a multi-scale benchmark for hallucination detection in long-context novel summarization and does not fully explore how hallucinations change as the context grows longer. In this study, we propose LongNovel, a multi-scale long-context bilingual (Chinese and English) novel benchmark for hallucination detection. This benchmark is constructed from 29 Chinese novels (ranging from 16k to 100k tokens) and chapter-level data from the BookSum dataset. We design 8 hallucination types and employ a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to ensure both data authenticity and a balanced distribution of hallucination categories. Furthermore, we manually revise the content in the test set to guarantee data reliability. Extensive experimental results demonstrate that LongNovel is a challenging benchmark. We release LongNovel for future research. https://github.com/BDML-lab/LongNovel

1 Introduction

LongNovel addresses the difficulty of detecting hallucinations in long-context novel summarization, where longer context windows do not eliminate hallucinations and existing metrics and benchmarks remain inadequate. It introduces a multi-scale Chinese-English benchmark spanning four context lengths, 29 books, 600 test samples, and eight hallucination types.

  • 100k-token context windows enable long-form processing but do not inherently resolve hallucinations in long-context summarization.
  • Novel summarization is well-suited to hallucination research because it requires inferring implicit information from dialogues and events, unlike the explicit data in news and academic papers.
  • ROUGE and BERTScore measure lexical or semantic similarity, while SummaC and AlignScore often fail on long-context hallucinations because of short input-window limitations.
  • Existing benchmarks face costly manual annotation, limited synthetic-data scale, and insufficient multi-scale designs for studying hallucinations as context length increases.Traditional datasets rely on labor-intensive human labels, synthetic LCHD does not reach 100k tokens, and Fables is largely restricted to one length.
  • LongNovel uses 29 books, four scenarios—S(16k), M(32k), L(64k), and XL(100K)—600 test samples, and eight hallucination types for long-context novel summarization detection.Human-written summaries serve as ground truth to guide LLM generation, and the benchmark is Chinese-English bilingual.

2 RELATED WORK

Prior hallucination-dataset construction uses either LLM generation with manual annotation or automated injection into reference materials. Detection methods span short-text entailment, semantic, and QA approaches, while long-text methods address input-window limits through LLM judgment, CoT, debate, and retrieval augmentation.

  • Hallucination Dataset Construction: Hallucination datasets are constructed by generating samples with LLMs and manually annotating hallucinations, producing patterns aligned with model performance and corresponding reasoning chains.This approach is contrasted with automated hallucination injection based on existing reference materials.
  • Hallucination Dataset Construction: Automated dataset construction injects hallucinations into books or summaries through entity substitution, negation insertion, low-likelihood outputs, or instruction-following LLM synthesis.These methods use existing reference materials and include both rule-based and LLM-based synthesis strategies.
  • Hallucination Detection: Short-text detection decomposes documents into sentence pairs for NLI entailment, uses alignment pre-training or semantic graphs, and verifies faithfulness through source-summary answer consistency.Semantic graph methods model relations between entities and sentences at sentence and passage levels.
  • Hallucination Detection: Long-text detection addresses input-window limitations with binary LLM classification, Chain-of-Thought analysis, indicator-based scoring, debate roles, and RAG-based faithfulness verification.Indicator-based LLM scoring shows high correlation with human judgment, while debate assigns Advocate, Skeptic, and Adjudicator roles.

3 LongNovel Construction

LongNovel is constructed from bilingual literary sources using multi-scale context sampling, human-reviewed summaries, and two hallucination-generation strategies. The resulting benchmark spans four target lengths, eight hallucination types, and balanced Chinese-English test distributions, with high validation agreement.

  • Corpus and summary construction: The Chinese corpus contains 29 coherent-plot books, segmented into 2k–6k-token textual units and summarized through a 16-annotator drafting and revision process.The benchmark uses both Chinese and English literary sources.
  • Multi-scale sampling: Four target context lengths—16k, 32k, 64k, and 100k tokens—are sampled with sliding windows over textual units.Token counts are computed with the Qwen3 tokenizer.
  • Hallucination generation: Eight hallucination types are generated through Multi-Model Arbitration and Entity-Referenced Hallucination Generation to improve authenticity and category balance.The types include entity, numerical, relation, logical, event, temporal, causal, and event-fabrication hallucinations.
  • Quality validation: Two annotators validate 204 test summaries, achieving 0.918 interannotator agreement for binary hallucination classification.Scores of 3–5 are non-hallucinated, while scores of 0–2 are hallucinated.
  • Dataset statistics: The benchmark contains 6,354 samples, including 5,354 training, 400 validation, and 600 test samples.Hallucination categories remain relatively balanced across text lengths, and Chinese-English proportions are evenly distributed across context lengths.

4 Experiments

Experiments evaluate open-source and commercial language models on LongNovel using standardized inference, scoring, and repeated testing procedures. They compare prompting, fine-tuning, retrieval, and ensemble strategies for hallucination detection.

  • Models: The evaluation covers InternLM2.5-20B-chat, GLM-4-9B, Llama3.1-8B, Qwen3 8B/14B/32B, GPT-5.2-chat, Claude-sonnet-4-5-20250929-v1, DeepSeek variants, and Gemini-3-Flash-Preview.Open-source models include InternLM2.5-20B-chat, GLM-4-9B, Llama3.1-8B, and Qwen3; commercial models include GPT-5.2-chat, Claude, DeepSeek, and Gemini.
  • Implementation: Open-source inference uses vLLM, YaRN expands 32k contexts to 128k for 32k, 64k, and 100k tests, commercial APIs use temperature 0, and open-source experiments repeat at least three times.These settings are intended to support longer-context evaluation and consistent, reliable outputs.
  • Evaluation: Consistency scores of 3–5 indicate non-hallucination, scores of 0–2 indicate hallucination, fuzzy matching handles format deviations, and Balanced Accuracy is the primary metric.Fuzzy regular-expression matching permits correct evaluation when models do not strictly follow the required output format.
  • Methods: The compared methods are zero-shot prompting with summary-at-beginning or summary-at-end prompts, Chain-of-Thought prompting, supervised fine-tuning, sliding-window RAG, and voting ensembles.Fine-tuning gradually increases context length, while the ensemble integrates three models using majority voting.
  • Retrieval-Augmented Generation: RAG addresses contextual fragmentation by pairing summary segments with relevant article windows; a summary is non-hallucinated only when every segment scores greater than 2.The source block size is 3,500 characters and the summary block size is 75 characters.

5 Experimental Results and Analysis

Performance generally declines as context length increases, especially for open-source models, although fine-tuning and model-specific prompting strategies can substantially alter this trend. Error analysis attributes long-context failures to incomplete or incorrect processing, ineffective corrections, and limited reasoning over actions and dialogues.

  • Overall Performance: Accuracy generally decreases with longer contexts across commercial and open-source models, with Qwen3-14B falling from 0.588 at 16k to 0.517 at 100k and Claude-4.5-Sonnet from 0.755 to 0.670.Llama3.1-8B-instruct reaches -1.000 at long contexts, reflecting repetitive outputs or confusion between summarization and hallucination detection.
  • Prompting and Adaptation: Chain-of-Thought prompting helps Claude-4.5-Sonnet at 32k, raising accuracy from 0.665 to 0.715, but hurts DeepSeek-v3 at 32k and 100k.DeepSeek-v3 declines from 0.735 to 0.685 at 32k and from 0.670 to 0.610 at 100k.
  • Prompting and Adaptation: Qwen3-32B SFT variants reach 0.720 average accuracy at 100k, versus 0.580 for the base version, showing the effectiveness of length-extrapolation fine-tuning.Across 16k–100k, the SFT variants outperform both base and CoT versions.
  • Retrieval-Augmented Generation: RAG is model-dependent: DeepSeek-v3 falls to 0.530 at 64k, whereas GPT-5.2-chat reaches 0.750 at 100k.The passage attributes this discrepancy to baseline judgment accuracy and the accumulation of errors when summaries are split into smaller chunks.
  • Dataset Construction Ablation: The ablation study compares base models, models fine-tuned on MMA data, and models trained on the complete MMA-plus-ERHG dataset.These configurations evaluate the efficacy of negative samples generated through Entity-Referenced Hallucination Generation.
  • Failure Analysis: Long-context detection fails when models misprocess source-summary pairs, generate unchanged corrections, or cannot map sequences of actions and dialogues through reasoning.The analysis notes that unchanged corrections often correspond to faithful sentences that require no correction.

6 Conclusion

LongNovel is introduced as a multilingual long-context dataset for hallucination detection in novels, built from human-annotated summaries and spanning four subsets from 16k to 100k tokens. Experiments show that current large language models remain insufficient for long-context hallucination detection.

  • LongNovel is a multilingual long-context dataset for hallucination detection in novels, based on human-annotated summaries.
  • Four LongNovel subsets span contexts ranging from 16k to 100k tokens.
  • Current large language models still lack sufficient capability for long-context hallucination detection tasks on LongNovel.

Limitations

The study evaluates only open-source models with up to 32B parameters and has not tested larger models or validated whether its findings generalize beyond novel-domain data.

  • Model and domain scope: The study is limited to open-source models with up to 32B parameters, leaving models such as Llama-3.1-70B unevaluated.The generalization capabilities of larger-scale models remain uninvestigated.
  • Model and domain scope: Because LongNovel is a novel dataset, the applicability of the findings to non-novel domains requires further validation.The authors identify cross-domain applicability as future research.

A LongNovel Dataset … E Experimental Setup Details

LongNovel is a publicly sourced, manually reviewed bilingual benchmark that partitions data for evaluation and defines eight hallucination types. Human annotation uses multi-stage minimal-pair review, while a sliding-window RAG framework aligns summaries with source text and escalates uncertain cases to full-context verification.

  • A LongNovel Dataset: LongNovel uses publicly available literary works for non-commercial academic research, with manual review confirming no sensitive personal information or PII of living individuals.The data is used strictly for training and evaluation.
  • A LongNovel Dataset: The Chinese subset is divided into training, validation, and test sets, while English splits strictly follow BookSum.Chinese corpus composition and book-author details are reported in Tables 6 and 7.
  • B Hallucination Type: The benchmark categorizes novel-summarization hallucinations into eight types: entity, numerical, relation, logical inversion, event, temporal, causal, and event fabrication.These categories cover entity errors, inconsistent numbers or relationships, inverted meanings, contradictory or reordered events, unsupported causality, and fabricated actions or thoughts.
  • B Hallucination Type: Temporal, causal, and event-fabrication hallucinations respectively disrupt event order, introduce unsupported causal links, or add actions, repetitions, and thoughts absent from the source.These categories extend the benchmark beyond entity, numerical, relational, logical, and event contradictions.
  • C.1 Summary Annotation: 16 Chinese Language and Literature annotators trial, review, and scale annotation, with each textual unit drafted by one annotator and revised by two others.The process is designed to improve summary accuracy and faithfulness while ensuring fair compensation.
  • C.2 Hallucination Annotation: Two NLP researchers apply NoCha-style minimal pairs, requiring hallucinated and non-hallucinated summaries to support reliable scoring and cross-verification.Annotators identify hallucinations, hallucinated sentences, and explanations; disagreements are resolved through discussion without strict time limits.
  • D RAG Framework: The RAG framework embeds document and summary blocks with BGE-M3, dynamically matches summary blocks to chronological document anchors, and builds adaptive bidirectional context windows.The window radius is R1 = ⌊N/3⌋ − 1, and the final summary window covers the last document block.
  • D RAG Framework: Each summary segment receives a 0-5 local consistency score, triggering full-context review when the score is ≤2; the summary passes only if every segment scores greater than 2.Source blocks contain 3,500 characters and summary segments contain 75 characters.

E.1 Baseline Evaluation · E.2 Fine-tuning Experiment

The baseline evaluation applies model-specific long-context extrapolation settings, while the fine-tuning experiment uses full-parameter training with sequence parallelism and efficiency-oriented infrastructure. Fine-tuning uses a 5e-6 learning rate, cosine scheduling without warmup, seed 42, and begins with a 100-step procedure for 16K sequences.

  • E.1 Baseline Evaluation: Qwen and InternLM use YaRN interpolation with a 4.0 scaling factor to enhance long-context capabilities.
  • E.1 Baseline Evaluation: Llama-3.1-8B-Instruct uses an 8.0 scaling factor, consistent with its original 8192 position embeddings.
  • E.1 Baseline Evaluation: GLM-4-9B retains its default configuration because it supports a 128K context window.
  • E.2 Fine-tuning Experiment: Fine-tuning uses full-parameter training integrated with Sequence Parallelism and implemented with Ring-Attention.
  • E.2 Fine-tuning Experiment: The experiment uses the 360-llama-factory framework, Flash Attention 2, and DeepSpeed ZeRO-3 Offload to improve computational efficiency and memory usage.
  • E.2 Fine-tuning Experiment: A 5e-6 learning rate, cosine scheduler, zero warmup, and fixed seed 42 are used for all fine-tuning experiments.
  • E.2 Fine-tuning Experiment: For fine-tuning on 16K sequences, the experiment initially performed a 100-step fine-tuning procedure.

F Full Results

Full results show strong differences by model family, prompt placement, and hallucination type. Commercial models and end-positioned summaries perform better, while temporal and causal hallucinations remain especially difficult to detect.

  • Model Biases: Open-source models generally struggle with hallucination detection, whereas Claude-4.5-Sonnet, DeepSeek-v4-flash, and GPT-5.2-chat demonstrate much stronger capabilities.InternLM2.5-20B, Minicheck-7B, and GLM-4-9B-chat show extremely limited detection across almost all context lengths.
  • Prompt Positioning: Placing the target summary at the beginning reduces detection accuracy, while placing it at the end enables superior and more stable performance.The beginning placement is identified as Prompt-B.
  • Hallucination Types: Event and numerical hallucinations are most detectable: Claude-4.5-Sonnet reaches 78.71% recall for events, while GPT-5.2-chat reaches 82.81% for numerical hallucinations.These are the highest recall values reported for the respective models and categories.
  • Hallucination Types: Temporal and causal hallucinations are most difficult, with InternLM2.5-7B and Minicheck-7B at 0.00% recall for both and GPT-5.2-chat at 36.25% for temporal hallucinations.Type-specific recall marks a hit when the model identifies the same factual error or event as the ground truth; one data point may contain multiple hallucination types.

G Ablation Study for RAG … English Version: Case on InternLM2.5-20B-chat with 32k Dataset

The paper evaluates RAG chunking and token efficiency, then illustrates hallucination-detection failures across Chinese and English cases, including comprehension errors, false corrections, and flawed logical reasoning.

  • G Ablation Study for RAG: Dynamic sentence-boundary chunking merges pronoun-led succeeding sentences, while GPT-5.2-chat peaks at chunk sizes of 75 in 64k and 100 in 100k contexts.The chunking mechanism preserves complete sentences rather than enforcing a rigid character limit.
  • G Ablation Study for RAG: Larger chunks dilute localized RAG focus by combining disparate factual claims, so accumulated errors eventually outweigh segmentation benefits and reduce GPT-5.2-chat performance.GPT-5.2-chat is less vulnerable to error accumulation in smaller chunks because of its higher baseline judgment accuracy.
  • G Ablation Study for RAG: Approximately 36k–38k tokens per evaluation call are used in the 64k setting, versus around 59k tokens in the 100k setting.The sliding-window framework evaluates paired summary-article blocks and reduces token expenditure compared with processing the entire article.
  • H.1 Failure in Comprehensively Understanding: The model confuses 40,000 troops sent to Le’an with 70,000 troops sent to Cochin because it fails to cross-reference distinct numerical facts.This case demonstrates incomplete understanding of the article.
  • Case on Deepseek-v4 with 16k Dataset: In the Deepseek-v4 Chinese and English cases, the model is evaluated on passages describing Zhu Zhanji’s negotiations over Annam and military deployment.The supplied cases present corresponding Chinese and English article excerpts for the 16k dataset.
  • H.2 Hallucination Correction for Faithful Sentences: The model falsely labels faithful sentences as hallucinations and claims corrections whose content is literally identical to the original text.This non-hallucination example exposes a severe logical flaw in hallucination correction.
  • H.3 Limitations in Logical Reasoning: The GLM cases show false negatives and false positives caused by missed entity-attribute bindings and contradictions of explicit character-background facts.One case wrongly attributes a character’s talent to another; another invents a hallucination despite textual evidence.
  • English Version: Case on Qwen3-32B with 64k Dataset: The Qwen3-32B English case receives score 5 because its summary that Gao Yang possesses Psychometry and sees objects’ past events by touch perfectly matches the article.The Chinese and English passages describe the same faithful summary and evaluation outcome.

H.4 Repetition Output … English Version: Case 2 on Claude-4.5-Sonnet with 100k Dataset

The reported cases show repetition outputs at 16k and 64k, hallucinated entity and event substitutions, and failures to produce the requested JSON judgment. Other examples include a truncated factual answer and an English synopsis of The Darker.

  • H.4 Repetition Output: Repetition answers may occur when the model processes the 16k and 64k datasets.
  • English Version: Case 1 on GLM-9B with 64k Dataset: The English GLM-9B case contains relation and event hallucinations, including replacing “system” with “dorm supervisor” and describing the “Temple of Talent” as the “universe.”
  • English Version: Case 1 on GLM-9B with 64k Dataset: A Qwen3-8B example repeatedly revises the inheritance account, contrasting Henry’s inheritance with claims involving his son John and grandson Harry.
  • H.5 Failure in JSON Format Generation: Instead of outputting a JSON hallucination-detection judgment, the model described the summary content.
  • English Version: Case 1 on Llama with 32k Dataset: The English Llama case gives a truncated answer stating that Li Lianhua discovered Feng Xiaoqi’s corpse, Feng Qing was imprisoned, and Li Lianhua and Shao Xiaowu.
  • English Version: Case 2 on Claude-4.5-Sonnet with 100k Dataset: The Claude-4.5-Sonnet case presents an English synopsis of The Darker centered on Doudou’s death and the revelation that Lin Xiaopeng orchestrated it.

H.6 Self-contradictory Judgement … English Version: Case on Claude-4.5-Sonnet with 16k Dataset

The cases show that models can identify hallucinations yet issue contradictory non-hallucination scores, while other examples document multilingual judgments, over-correction checks, and consistent summary content across Chinese and English versions.

  • H.6 Self-contradictory Judgement: The model flags an Event Hallucination but assigns score 3, incorrectly judging the response as non-hallucinated.Because scores of 3 or above indicate non-hallucination, the score contradicts the explanation.
  • Case 1 on Qwen3-8B with 16k Dataset: Qwen3-8B identifies that Chen Xun’s intended medical study is presented as completed, while assigning score 3.The reasoning labels this as an Event Hallucination and considers the other listed plot elements consistent with the original.
  • English Version: Case 1 on Qwen3-8B with 16k Dataset: The English Qwen3-8B case likewise identifies Chen Xun’s medical-study claim as an Event Hallucination while assigning score 3.It states that the original expresses only an intention, not that Chen Xun ultimately studied medicine.
  • Case 2 on Qwen3-8B with 64k Dataset: A 64k Qwen3-8B case attributes multiple errors to the summary, including entity, relational, event, and timeline hallucinations.The example confuses Robert Ferrars with Edward Ferrars and reverses the marriage attribution involving Lucy Steele.
  • H.7 Over-Correction: The over-correction example states that blue-highlighted words are semantically equivalent and do not distort the original summary’s meaning.This supports treating the wording changes as acceptable rather than hallucinated alterations.
  • Case on Claude-4.5-Sonnet with 16k Dataset: The Chinese Claude-4.5-Sonnet 16k summary begins with Wu Shiziyang’s double pupils and falling-star omen, followed by Han Diaosi’s proposed rise to power.It also describes Wu informing Han that Yan is weak and Han promising him the State Preceptor title.
  • English Version: Case on Claude-4.5-Sonnet with 16k Dataset: The English Claude-4.5-Sonnet 16k summary presents the same opening plot: Wu Shiziyang’s double pupils, falling stars, Yan’s weakness, and Han Diaosi’s imperial ambition.The passage is an English rendering of the corresponding Chinese summary content.

I All Used Prompts · I. Criteria for No Hallucination

The appendix specifies prompts for generating hallucinations across categories and for evaluating summaries with RAG, zero-shot, and Chain-of-Thought methods. It also defines non-hallucination criteria, hallucination categories, annotation examples, and a 0–5 consistency scale.

  • I.1 Prompt for Hallucination Generation: Table 10 maps different hallucination types to corresponding generation prompts.The table is presented as the prompt reference for hallucination-type generation.
  • I.2 Prompt for Different Methods: RAG, zero-shot, and Chain-of-Thought prompting templates support summary hallucination evaluation.Zero-shot templates place the target summary at either the end or beginning, while Chain-of-Thought prompting elicits step-by-step reasoning.
  • I.2 Prompt for Different Methods: Human annotation asks whether a generated summary contains hallucinations and, when present, requires the hallucinated sentence and judgment reason.Annotators output “No Hallucination” when no hallucination is found and “Hallucination” otherwise.
  • I. Criteria for No Hallucination: Simplification of motives, tone smoothing, reasonable inference, synonym replacement, process simplification, identity blurring, and broad event descriptions are non-hallucinations.The examples treat reasonable inference and broad description as supported when they have a logical or textual basis.
  • I. Criteria for No Hallucination: Hallucinations include entity, numerical, relational, and fabricated-event errors.Examples include incorrect pronouns or role swaps, changed quantities or dates, altered identities or relationships, and invented actions or mental states.
  • I. Criteria for No Hallucination: Generation prompts isolate one hallucination type by modifying exactly one sentence while preserving the remaining summary content.The documented prompts cover entity, numerical, relational, reverse-statement, event, temporal, causal, and fabricated-event hallucinations.
  • I. Criteria for No Hallucination: Consistency scoring ranges from 0 to 5, from fully supported summaries to completely irrelevant or severely false content.A score of 5 means all facts are supported; 4 allows minor over-inference, 3 allows generalization or causal simplification, and scores below 3 indicate increasingly obvious fabrication.
Loading 2608.18082v1…