LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization

Motivation

• Long-context summarization remains vulnerable to hallucinations despite context windows exceeding 100k tokens, creating a need for dedicated evaluation benchmarks.Novel summarization is especially suitable because it requires inferring implicit information from dialogues and events, while existing lexical, semantic, and short-window NLI metrics are inadequate.Source: §1 · p. 1

Challenges

• Existing benchmarks face costly manual annotation, limited 100k-token coverage, and insufficient multi-scale evaluation of hallucination changes as context length increases.Fables largely uses one length, while Clipper is not designed for summary hallucination detection.Source: §1 · pp. 1–2, Table 1 · p. 2

Contribution

• LongNovel introduces a Chinese-English bilingual benchmark spanning 16k, 32k, 64k, and 100K-token scenarios, with 600 test samples and eight hallucination types.The benchmark is constructed from 29 books and targets long-context novel summarization.Source: §1 · p. 2

Benchmark Construction

• Multi-Model Arbitration generates summaries with GLM4-9B-chat, Qwen3-32B, and GPT-4o, then cross-verifies labels to capture authentic hallucinations.Human-written summaries serve as ground truth for guiding generation.Source: §1 · p. 2
• Entity-Referenced Hallucination Construction perturbs human-written summaries using extracted entities and eight prompt templates to balance hallucination-type coverage.The test set is additionally revised by humans to improve reliability.Source: §1 · p. 2

Evaluation

• Experiments show that LongNovel is challenging for state-of-the-art models and methods, exposing persistent difficulties in long-context hallucination detection.Each judgment reports a consistency score, hallucination types, and identification reasons.Source: §1 · p. 2

2.1 Hallucination Detection Benchmarks

• LLM-generated hallucination datasets with manual annotation better reflect real-world model behavior but require costly, resource-intensive labeling (Laban et al., 2023; Karpinska et al., 2024a; Subbiah et al., 2024; Akbar et al., 2024; Tang et al., 2024b; Chen et al., 2024b; Bao et al., 2025; Abdaljalil et al., 2025; Qi et al., 2025).Source: §2.1 · p. 2
• Automated benchmark construction injects hallucinations through entity substitution, negation, low-likelihood outputs, or LLM-based synthesis from books, summaries, and outlines (Kryscinski et al., 2020; Cao and Wang, 2021; Tang et al., 2024a; Liu et al., 2025; Ming et al., 2025; Pham et al., 2025).Pham et al. (2025) induce true-false claim pairs and corresponding reasoning chains from extracted summaries or outlines.Source: §2.1 · pp. 2–3

2.2 Hallucination Detection Methods

• Short-text detection uses NLI entailment, alignment pre-training, Atomic Content Units, semantic graphs, and question answering to assess factual consistency across sentences, passages, entities, and summaries (Laban et al., 2022; Zha et al., 2023; Liu et al., 2023c; Chen et al., 2025; Deutsch et al., 2021; Scialom et al., 2021).Source: §2.2 · p. 3
• Long-text detection addresses input-window constraints with binary LLM classification, Chain-of-Thought analysis, indicator-based scoring, debate among role-specific agents, and RAG-based faithfulness verification (Wei et al., 2022; Liu et al., 2023b; Min et al., 2025; Zhang et al., 2025; Hu et al., 2025; Laban et al., 2024; Min et al., 2023).Indicator-based LLM scoring correlates highly with human judgment, while debate assigns Advocate, Skeptic, and Adjudicator roles to improve data reliability.Source: §2.2 · pp. 3–4

3.1 Data Collection

• The benchmark combines 29 coherent Chinese books with chapter-level English BookSum data, pairing each 2k–6k-token textual unit with a human-written summary.Sixteen annotators draft and revise summaries for the Chinese corpus, while English summaries are adopted from BookSum.Source: §3.1 · p. 4

3.2 Length Extension

• LongNovel evaluates four target context lengths—16k, 32k, 64k, and 100k tokens—using sliding windows that expand unit by unit across textual units for each target range.It uses the Qwen3 tokenizer to compute token counts and discards residual sequences when windows reach a book’s end but fail the lower boundary.Source: §3.2 · p. 4

3.3 Data Synthesis

• The synthesis process covers eight hallucination types using multi-model arbitration and entity-referenced rewriting to improve realism and maintain balanced categories.Arbitration aggregates score-based evaluations and explanations from GPT-4.1 and Claude-sonnet-4-20250514-v1 through Gemini-3-Flash-Preview; entity-based rewriting injects one or three hallucination types with assigned consistency scores of 2 or 1.Source: §3.3 · pp. 4–5

3.4 Human Revision

• Human validation of 204 test summaries achieved 0.918 interannotator agreement for binary hallucination classification, supporting the quality of the final benchmark.Scores 3–5 are classified as non-hallucinated and scores 0–2 as hallucinated.Source: §3.4 · p. 5

3.5 Dataset Statistics

• The 6,354-sample benchmark is split into 5,354 training, 400 validation, and 600 test samples, with hallucination categories and Chinese–English proportions remaining relatively balanced across text lengths.The dataset statistics are reported in Table 2, while the distributional balance across context lengths is illustrated in Fig. 2.Source: §3.5 · p. 5

4.1 Baselines

• The baselines include InternLM2.5-20B-chat, GLM-4-9B, Llama3.1-8B, Qwen3 models, GPT-5.2-chat, Claude-sonnet-4-5-20250929-v1, DeepSeek models, and Gemini-3-Flash-Preview.The open-source models span 8B, 14B, 20B, and 32B configurations, while commercial models include multiple current model families.Source: §4.1 · p. 5

4.2 Experimental Setup

• Open-source models are run with vLLM, YaRN expands 32k contexts to 128k for longer tests, commercial APIs use temperature 0, and open-source experiments repeat at least three times.These settings target consistent inference across context lengths and repeated measurements.Source: §4.2 · p. 5

4.3 Evaluation Metrics

• Consistency scores of 3–5 denote non-hallucination and 0–2 hallucination, with fuzzy matching enabling evaluation despite output-format deviations; Balanced Accuracy is primary and MCC is supplementary.Balanced Accuracy averages per-class recall, while MCC accounts for all confusion-matrix categories and balances false positives against false negatives.Source: §4.3 · p. 5

4.4 Compared Methods

• Compared methods include zero-shot prompts, chain-of-thought prompting, progressively lengthened supervised fine-tuning, sliding-window semantic RAG, and a three-model majority-voting ensemble.RAG classifies an instance as non-hallucinated only when every summary segment scores above 2 in its corresponding window, while the ensemble uses at least two model decisions.Source: §4.4 · p. 6

5.1 Main Results

• Accuracy generally declines as context length grows: Qwen3-14B falls from 0.588 at 16k to 0.517 at 100k, while Claude-4.5-Sonnet drops from 0.755 to 0.670.At 64k and 100k, Llama3.1-8B-instruct reaches an MCC of -1.000, reflecting severe long-context instruction-following degradation.Source: §5.1 · pp. 6–7, Table 4 · p. 7
• SFT improves long-context detection: Qwen3-32B reaches 0.720 accuracy at 100k versus 0.580 for its base version, while CoT effects vary by architecture.CoT raises Claude-4.5-Sonnet’s 32k accuracy from 0.665 to 0.715 but lowers DeepSeek-v3’s from 0.735 to 0.685 and its 100k score from 0.670 to 0.610.Source: §5.1 · p. 7
• RAG is model-dependent, falling to 0.530 accuracy for DeepSeek-v3 at 64k but reaching 0.750 for GPT-5.2-chat at 100k, while DSV4/GPT/C achieves the highest 64k scores.The discrepancy is attributed to baseline judgment accuracy: chunking amplifies DeepSeek-v3’s errors, whereas GPT-5.2-chat benefits from fewer initial mistakes.Source: §5.1 · pp. 7–8

5.2 Analysis and Case Study

• ModelFull consistently achieves the highest Accuracy and MCC across dataset sizes, demonstrating that ERHG adds challenging negative samples beyond MMA-only construction.MMA generally provides incremental gains over base models, but combining MMA and ERHG yields stronger robust alignment.Source: §5.2 · p. 8, Table 5 · p. 8
• Long-context failures arise from misreading inputs, weak action-to-abstraction reasoning, repetitive truncation, incomplete instruction following, logical contradictions, and missed semantic equivalence.Models may also flag faithful summaries as hallucinations when wording changes or peripheral details are omitted.Source: §5.2 · p. 8

• LongNovel is a multilingual long-context dataset for hallucination detection in novels, based on human-annotated summaries.Source: §6 · p. 8
• Its four subsets range from 16k to 100k tokens.Source: §6 · p. 8
• Experiments reveal that current large language models still lack sufficient capability for long-context hallucination detection, motivating future research with LongNovel.Source: §6 · p. 8

• The study is limited to open-source models with up to 32B parameters, so generalization to models such as Llama-3.1-70B remains uninvestigated.This limitation concerns the model scale and scope evaluated in the study.Source: Limitations · p. 9
• Because LongNovel is a novel dataset, whether the findings apply to non-novel domains remains to be validated.The passage identifies cross-domain applicability as an open direction for future research.Source: Limitations · p. 9

A LongNovel Dataset

• LongNovel is built from publicly available literary works, excludes sensitive PII, and partitions Chinese data into train, validation, and test sets while matching BookSum splits for English data.Tables 6 and 7 provide Chinese dataset statistics, book titles, and authors; the English split follows Kryscinski et al. (2022).Source: A LongNovel Dataset · p. 12

B Hallucination Type

• The benchmark categorizes novel-summary hallucinations into eight types: entity, numerical, relation, logical inversion, event, temporal, causal, and event fabrication.These categories cover factual entity errors, numerical inconsistencies, relationship errors, opposite meanings, contradicted events, reordered events, unsupported causality, and fabricated actions or states.Source: B Hallucination Type · p. 12

C.1 Summary Annotation

• Sixteen Chinese Language and Literature annotators produce summaries through trial annotation and iterative drafting and revision, with compensation exceeding the local legal minimum wage.Each textual unit receives an initial summary from one annotator and revisions from two additional annotators, covering both trial and main annotation hours.Source: §C.1 · p. 13

C.2 Hallucination Annotation

• Hallucination annotation uses NoCha-style minimal pairs, requiring both summaries to be correctly classified and resolving disagreements through discussion and consensus.Annotators identify hallucination presence, hallucinated sentences, and explanations without strict time limits; the pairwise criterion filters ambiguous cases.Source: §C.2 · p. 13

D RAG Framework

• The RAG framework partitions documents and summaries into blocks, embeds them with BGE-M3, and aligns summary blocks to document blocks using cosine similarity.Dynamic anchors preserve chronological order, select among top-5 matches, and construct an adaptive bidirectional window with radius R1 = ⌊N/3⌋− 1 while extending the final window to DN−1.Source: D RAG Framework · pp. 13–14
• A two-tier review escalates local consistency scores of ≤2 to full-document review, confirming hallucination only when the full-context score remains ≤2.A full-context score above 2 replaces the local score, while a confirmed hallucination terminates evaluation of the sample’s remaining summary blocks early.Source: D RAG Framework · p. 14
• The implementation uses 3,500-character document blocks and 75-character summary segments, classifying a summary as non-hallucinated only when every segment scores above 2 in its retrieved window.This segment-level rule operationalizes the sliding-window alignment and consistency evaluation.Source: D RAG Framework · p. 14

E.1 Baseline Evaluation

• Qwen and InternLM use YaRN with a 4.0 scaling factor, while Llama-3.1-8B-Instruct uses 8.0 and GLM-4-9B retains its default 128K configuration.Source: §E.1 · p. 14

E.2 Fine-tuning Experiment

• Fine-tuning integrates full-parameter training with Sequence Parallelism through Ring-Attention (Liu et al., 2023a) and the 360-llama-factory framework.Source: §E.2 · p. 14
• Flash Attention 2 (Dao, 2024) and DeepSpeed ZeRO-3 Offload (Ren et al., 2021) reduce computational and memory demands during fine-tuning.Source: §E.2 · p. 14
• Training uses a 5e-6 learning rate, cosine scheduling without warmup, a fixed seed of 42, and two epochs at every stage.Source: §E.2 · p. 14
• For 16K sequences, training begins with 100 steps on 2K data before full 16K fine-tuning; larger scales similarly warm up on 16K data before 32K, 64K, or 100K training.Source: §E.2 · p. 14
• The final model is the best validation checkpoint, with experiments run on 8 NVIDIA H20 GPUs and the fine-tuning prompt matched to the inference prompt.Source: §E.2 · p. 14

• Commercial models detect hallucinations substantially better than open-source base models across context lengths, while InternLM2.5-20B, Minicheck-7B, and GLM-4-9B-chat perform poorly.These comparisons are reported in Table 8 and Table 9.Source: F Full Results · p. 14
• Placing the target summary at the end produces superior and more stable detection than beginning placement, which reduces accuracy across most architectures.Beginning placement is Prompt-B and hinders integration with the extensive long-article context.Source: F Full Results · p. 14
• 82.81% is GPT-5.2-chat’s peak recall for Numerical Hallucination, while Claude-4.5-Sonnet reaches 78.71% for Event Hallucination.Event and numerical hallucinations are the most detectable categories in Fig. 3.Source: F Full Results · pp. 14–15
• Temporal and Causal Hallucinations are hardest to detect, with some models reaching 0.00% recall and GPT-5.2-chat achieving only 36.25% temporal recall.Models handle isolated numerical or factual errors better than altered chronology or causal relationships.Source: F Full Results · p. 15
• Type-specific recall matches a model to the same factual error or event as the ground truth, but one data point may contain multiple hallucination types.Overall accuracy can therefore exceed type-specific recall when a model identifies only one of several errors.Source: F Full Results · p. 15

G Ablation Study for RAG

• RAG chunk-size effects differ by model: DeepSeek-v3 improves from 50 to 150, whereas GPT-5.2-chat peaks at 75 or 100 before declining.Smaller chunks accumulate judgment errors under conjunctive aggregation, while larger chunks dilute localized factual focus by combining disparate claims.Source: G Ablation Study for RAG · p. 15, Figure 4 · p. 16
• Sliding-window RAG reduces average input tokens to approximately 36k–38k at 64k context and around 59k at 100k while preserving factual-alignment continuity.The strategy avoids repeatedly re-processing the full article for each verification step.Source: G Ablation Study for RAG · p. 15, Figure 5 · p. 20

H.1 Failure in Comprehensively Understanding

• A model failed to cross-reference distinct troop counts, confusing 40,000 troops sent to Le’an with 70,000 troops sent to Cochin.This demonstrates failure to comprehensively understand the article’s distributed facts.Source: §H.1 · p. 21

Case on Deepseek-v4 with 16k Dataset

• For the Chinese summary, Deepseek-v4 assigned score 5, stating that it accurately restored the original’s main plot without hallucinations; the source describes 70,000 troops dispatched and 40,000 marching.Source: Case on Deepseek-v4 with 16k Dataset · p. 21

H.2 Hallucination Correction for Faithful Sentences

• A non-hallucination case shows severe false positives: the model labeled faithful sentences as hallucinations and proposed corrections identical to the original text.This illustrates a logical flaw in hallucination correction.Source: §H.2 · p. 23

Case on GLM with 16k Dataset

• GLM assigned score 1 to a faithful 16k summary after repeatedly labeling unchanged sentences as event hallucinations and ultimately identifying an entity hallucination.Its listed corrections preserve the original wording, including actions involving Alec, Tess, and her family.Source: Case on GLM with 16k Dataset · p. 23

H.3 Limitations in Logical Reasoning

• The cases reveal both false negatives and false positives caused by faulty logical reasoning, including missed entity-attribute bindings and contradictions of explicit character facts.These errors lead to incorrect hallucination judgments in opposite directions.Source: §H.3 · p. 24

Case on Qwen3-32B with 64k Dataset; Case on InternLM2.5-20B-chat with 32k Dataset

• Qwen3-32B and InternLM2.5-20B-chat each produced incorrect hallucination judgments by misassigning character information, despite summaries judged faithful or nearly faithful by ground truth.Qwen3-32B wrongly attributed Psychometry to Gao Yang at 64k, while InternLM2.5-20B-chat wrongly treated Pupu and Ding Hao as orphanage children at 32k; the English cases reproduce the same errors.Source: Case on Qwen3-32B with 64k Dataset · p. 24, English Version: Case on Qwen3-32B… · p. 24, Case on InternLM2.5-20B-chat with 32k… · p. 25

H.4 Repetition Output

• Repetition outputs occurred when processing the 16k and 64k datasets.The result is reported for sequences at both context lengths.Source: §H.4 · p. 26

English Version: Case 1 on GLM-9B with 64k Dataset

• GLM-9B’s 64k output repeatedly replaced entities and altered events, including assigning unrelated identities and expanding descriptions beyond the source.Examples include replacing pronouns or roles with Lin Mengjuan and adding unsupported relations or event descriptions.Source: English Version: Case 1 on GLM-9B with… · p. 26, Case 1 on GLM-9B with 64k Dataset · p. 26

Case 2 on Qwen3-8B with 16k Dataset

• Qwen3-8B generated a looping self-correction about inheritance in the 16k case instead of providing a stable hallucination judgment.The output repeatedly reconsidered whether Henry, John, or Harry inherited the estate.Source: Case 2 on Qwen3-8B with 16k Dataset · p. 26

H.5 Failure in JSON Format Generation

• A separate failure occurred when the model described the summary rather than outputting the required JSON hallucination judgment.This demonstrates a format-compliance failure rather than a content judgment.Source: §H.5 · p. 27

English Version: Case 1 on Llama with 32k Dataset

• Llama’s 32k output gave an incomplete Chinese answer about Li Lianhua, Feng Xiaoqi, Feng Qing, and Shao Xiaowu.The English version preserved the same fragmentary answer rather than completing a full judgment.Source: English Version: Case 1 on Llama with… · p. 27

Case 2 on Claude-4.5-Sonnet with 100k Dataset

• The Chinese 100k Claude-4.5-Sonnet case produced a structured synopsis of The Darker, including crimes, character relations, political conflicts, layered truths, and Luo Xue’s final scheme.Its account identifies multiple reversals, the three-hundred-million money-laundering case, and the absence of genuine justice.Source: Case 2 on Claude-4.5-Sonnet with 100k… · p. 27

English Version: Case 2 on Claude-4.5-Sonnet with 100k Dataset

• Claude-4.5-Sonnet produced a detailed English synopsis for the 100k-token case, covering the plot, character relationships, power struggles, reversals, and themes of The Darker.The synopsis describes Doudou’s death, serial murders, political corruption, Luo Xue’s manipulation, and the novel’s thematic conclusion.Source: English Version: Case 2 on… · p. 28
• The English translation of the 100k Claude-4.5-Sonnet synopsis retained the same major plot structure and thematic conclusion.It preserves the murder cases, political power struggle, successive suspects, Luo Xue’s manipulation, and the claim that hidden agendas replace justice.Source: English Version: Case 2 on… · p. 28

H.6 Self-contradictory Judgement

• The model flags an Event Hallucination but assigns score 3, incorrectly judging the response non-hallucinated because scores of 3 or above indicate no hallucination.This directly contradicts the model’s own reasoning and produces an incorrect judgment.Source: §H.6 · p. 29

Case 1 on Qwen3-8B with 16k Dataset

• Qwen3-8B identifies that Chen Xun’s study of medicine was only intended, not confirmed, yet assigns score 3 and labels it an Event Hallucination.The remaining plot elements are judged consistent with the source.Source: Case 1 on Qwen3-8B with 16k Dataset · p. 29

English Version: Case 1 on Qwen3-8B with 16k Dataset

• The English Qwen3-8B case preserves the same diagnosis: Chen Xun’s medical study is unsupported, while other plot elements remain consistent, despite score 3.The listed hallucination type is Event Hallucination.Source: English Version: Case 1 on Qwen3-8B… · p. 29

Case 2 on Qwen3-8B with 64k Dataset

• On the 64k dataset, Qwen3-8B identifies entity, relational, event, and timeline errors involving Robert and Edward Ferrars but still assigns score 3.The summary wrongly attributes Lucy Steele’s marriage and Elinor’s attachment to Edward’s brother, while conflating the event sequence.Source: Case 2 on Qwen3-8B with 64k Dataset · p. 29

H.7 Over-Correction

• The over-correction example shows that semantically equivalent wording in blue does not distort the article’s meaning, so the original summary is not hallucinated.The judgment distinguishes harmless paraphrase from meaning-changing distortion.Source: §H.7 · p. 30

Case on Claude-4.5-Sonnet with 16k Dataset

• For the Chinese 16k case, Claude-4.5-Sonnet assigns score 2, citing event and causal-chain hallucinations in treating “静待即可” as Han Diaosi’s instruction rather than Wu Shiziyang’s own action.The passage clarifies that Jiang Cheng is Jiang Ge’s elder brother.Source: Case on Claude-4.5-Sonnet with 16k… · p. 30

I.1 Prompt for Hallucination Generation

• Table 10 maps hallucination types to their generation prompts.The correspondence is presented as the prompt design for different hallucination categories.Source: Table 10 · p. 32, §I.1 · p. 32

I.2 Prompt for Different Methods

• The evaluation prompts include RAG, zero-shot templates with the target summary at the beginning or end, and Chain-of-Thought prompting.These templates are shown in Figures 24, 26, 28, and 30.Source: §I.2 · p. 32
• Annotators must label summaries as “No Hallucination” or “Hallucination,” identifying the hallucinated sentence and explaining the judgment.The rule is presented as part of the human annotation procedure.Source: Figure 6 · p. 33, §I.2 · p. 34

I. Criteria for No Hallucination

• Simplified motives, smoothed tone, reasonable inference, synonym replacement, process simplification, identity blurring, and broad event descriptions are treated as non-hallucinations.Examples show that logically supported inferences and broad descriptions remain acceptable even when not stated verbatim.Source: §I · pp. 34–35
• Hallucinations include entity, numerical, relational, reverse-statement, event, timeline, causal-chain, and fabricated-event errors.The criteria cover incorrect references or roles, altered numbers or relationships, polarity reversals, changed events or order, fabricated causality, and unsupported actions or mental states.Source: §I · p. 34
• Generation prompts constrain each injected error to one sentence or one targeted modification while preserving the summary’s other facts, style, coherence, and JSON output format.The entity, numerical, relational, reverse-statement, event, temporal, causal, and fabricated-event prompts specify these isolation constraints and category-specific edits.Source: §I · pp. 37, 38, 40, 42, 44, 45
• The consistency checker ignores omissions, accepts supported partial coverage and early narrative facts, and scores summaries from 0 to 5 based on factual consistency and hallucination severity.Scores of 3 or above use structured score-and-reason output, while lower scores additionally select hallucination types.Source: §I · p. 50
• The evaluator compares the original text with the summary to determine whether the summary accurately reflects the source content.Source: §I · pp. 53, 57
• The prompt variants position the target summary at the end or beginning of the input for zero-shot evaluation, as shown in Figures 27–29.Source: Figure 27 · p. 53, Figure 28 · p. 54, Figure 29 · p. 55
• The rubric assigns scores from 0 to 5, ranging from a perfect match with no hallucinations to irrelevant output or extensive serious false facts.Source: §I · pp. 53–54
• A score of 4 indicates high consistency with minor over-inference or semantic shifts, while 3 denotes factual correctness with generalization, altered tone or intensity, or simplified causality.Source: §I · pp. 53–54
• A score of 2 represents significant errors in times, names, numbers, or events, whereas 1 indicates mostly fabricated content with many fictional events.Source: §I · pp. 53–54
• Chain-of-thought prompting is included to elicit step-by-step reasoning during evaluation, as shown in Figures 30 and 31.Source: Figure 30 · p. 56, Figure 31 · p. 57
• Omissions do not reduce the score; evaluation focuses exclusively on hallucinations in the summary.Source: §I · pp. 53, 57