PaperlayerView exact revision on arXiv

Source-linked AI summary

CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation

Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

arXiv:2609.33343v1cs.CLcs.AI

TL;DR

Existing faithfulness metrics can miss how entity, relation, and quantity errors differ, including numerical fabrications amid otherwise accurate summaries. CHI combines three hallucination dimensions, including a tolerance-aware quantity metric, into a composite score. On SummEval, CHI has higher system-level than summary-level correlation with human judgments, while MiniCheck shows the reverse pattern.

  • Problem

    Existing metrics treat hallucination as unidimensional, and a summary can contain a fabricated quantity despite correct entities and relations.

    Source: §1 · pp. 1–2

  • Method

    CHI combines entity, relation, and quantity hallucination dimensions, with QHI using tolerance-aware numerical matching.

    Source: §1 · p. 2, §3.2 · p. 5, §3.4.2 · p. 6

  • Results

    CHI has system-level ρ = 0.66 and summary-level ρ = 0.08 on SummEval, while MiniCheck has summary-level ρ = 0.38 and system-level ρ = 0.41.

    Source: §6.1 · p. 12

  • Takeaways & Limitations

    CHI provides an interpretable, decomposable composite score that unifies the three dimensions through harmonic-mean fusion.

    Source: §8 · p. 15

    The current evaluation is limited to English and four domains; cross-lingual generalization and additional domain coverage remain for future validation.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational inconsistencies, or numerical fabrications, without capturing their co-occurrence or interaction. We introduce CHI (Composite Hallucination Index), the first unified hallucination metric that decomposes faithfulness errors into three orthogonal dimensions: entity hallucination (EHI), relation hallucination (RHI*), and quantity hallucination (QHI). Each dimension employs a shared softmax-normalized architecture over Venn diagram-derived factors representing extractiveness, positive hallucination, over-focus, negative hallucination, and lost focus. The novel QHI component introduces tolerance-aware numerical matching with exact, epsilon, derived, and temporal comparison modes. We fuse the three dimensions via harmonic mean to produce a single composite score that penalizes weakness in any dimension. We validate CHI on 800 source articles spanning four domains (news, medical, legal, financial) with summaries from five generation systems. Empirical results demonstrate that: (i) the three dimensions are statistically orthogonal (mean rho = 0.148), confirming they capture distinct error types; (ii) CHI achieves the highest system-level correlation with human judgments (rho = 0.66, p = 0.006) on SummEval, outperforming ROUGE (rho = 0.53), EHI (rho = 0.58), and all individual components; and (iii) ablation studies confirm that all three dimensions contribute unique variance, with the full composite outperforming any individual component while providing decomposable error diagnostics unavailable from single-score baselines. CHI provides practitioners with a decomposable, interpretable, and efficient faithfulness metric suitable for both offline evaluation and online monitoring of summarization systems.

1 Introduction

Existing faithfulness metrics do not distinguish entity, relation, and numerical errors, a gap that matters especially when a fabricated quantity hides behind otherwise accurate content. CHI introduces a quantity-specific metric and combines it with entity and relation dimensions in a composite score.

1 Introduction

  • A summary can name the correct entities and state the correct relationship yet fabricate a deal value, an error single-type metrics may miss.The example contrasts a reported $2.8 billion deal with a fabricated $4.2 billion value.Source: §1 · p. 1
  • Existing metrics treat hallucination as unidimensional, while high-stakes domains can require detecting distinct entity, relational, and numerical errors.A single aggregate score can obscure actionable diagnostic information.Source: §1 · p. 2
  • QHI introduces a dedicated numerical-hallucination metric with exact, epsilon, derived, and temporal matching modes.The modes are instantiated within a softmax-normalized five-factor architecture.Source: §1 · p. 2
  • The three hallucination dimensions are statistically orthogonal, with mean pairwise ρ < 0.3 and p < 0.001.The paper presents this as evidence that they capture distinct error phenomena.Source: §1 · p. 2
  • CHI unifies EHI, RHI∗, and QHI through harmonic-mean fusion and achieves the highest reported system-level correlation with human judgments, ρ = 0.66.The composite is described as providing decomposable diagnostics at negligible computational cost.Source: §1 · p. 2

2 Related Work

Prior faithfulness metrics often provide scalar scores or focus on one structural dimension, while dedicated numerical evaluation has not addressed summary-source consistency. CHI brings entity, relation, and quantity dimensions together with a shared architecture and harmonic-mean fusion.

2.1 Faithfulness Metrics

  • Single-scalar metrics can assign the same score to summaries with different error profiles, such as three entity errors versus three numerical errors.The paper notes these failure modes may require different interventions.Source: §2.1 · p. 2

2.2 Structured Evaluation Approaches

  • Earlier structural approaches decompose faithfulness along one dimension; CHI integrates entity, relation, and quantity dimensions in one architecture.The cited approaches include dependency, semantic-role, and abstract-meaning representations, alongside prior EHI and RHI work.Source: §2.2 · p. 3

2.3 Numerical Hallucination

  • Prior numerical-evaluation work targets general claims or multi-turn retrieval, and the paper identifies no prior metric isolating numerical hallucination in single-document summarization within a composite framework.Source: §2.3 · p. 3

2.4 Composite and Ensemble Approaches

  • CHI distinguishes itself from ensemble approaches through orthogonal dimensions, a shared architecture, and harmonic-mean fusion that penalizes weakness in any dimension.The passage contrasts this fusion with arithmetic averaging in prior work.Source: §2.4 · p. 3

3 Methodology

CHI uses a shared Venn-factor architecture to score entities, relations, and other units, with softmax-normalized faithfulness scores that can be fused. EHI scores named entities, while RHI∗ evaluates SVO relations with an expanded factor set and semantic matching.

3.1 Unified Architecture

  • CHI extracts entities, relations, and quantities in parallel, derives Venn-diagram factors from source, reference, and generated texts, and normalizes dimension scores to [0, 1].The three sub-indices are fused by harmonic mean into CHI; the pipeline is deterministic and CPU-only, with optional GPU acceleration for Sentence-BERT in RHI∗.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 EHI: Entity Hallucination Index

  • EHI evaluates named-entity faithfulness using five factors and spaCy NER with case-insensitive exact-match deduplication and acronym expansion.Source: §3.2 · p. 5

3.3 RHI∗: Relation Hallucination Index (Softmax Variant)

  • RHI∗ evaluates source, reference, and generated SVO tuples, extending the factor set to six and using semantic-similarity matching for paraphrases.Its distinction from linear RHI is softmax normalization and inclusion of all six factors; extraction uses dependency parsing followed by SVO patterns with passive-voice normalization.Source: §3.3 · pp. 5–6

3.4 QHI: Quantity Hallucination Index

QHI evaluates numerical faithfulness using contextualized quantity extraction and tolerance-aware matching. CHI then combines available dimension scores by harmonic mean, which penalizes weakness in any single dimension.

3.4 QHI: Quantity Hallucination Index

  • QHI introduces a dedicated quantity-hallucination dimension based on quantities extracted from source, reference, and generated summaries.Source: §3.4 · p. 6

3.4.1 Quantity Extraction

  • Quantity extraction combines regex patterns and spaCy named-entity types, then anchors each value to a governing noun phrase or nearby entity.Contextual anchoring distinguishes uses such as an acquisition price from annual revenue.Source: §3.4.1 · p. 6

3.4.2 Tolerance-Aware Matching

  • A quantity matches through exact normalization, a 5% relative tolerance, a source-supported arithmetic derivation, or calendar-aware temporal normalization.When multiple numerical matches exist, context-anchor similarity breaks ties.Source: §3.4.2 · p. 6

3.4.3 QHI Factor Computation

  • QHI computes five quantity factors and omits Lost Hallucination because quantities are atomic rather than partially assembled.Tolerance-aware matching handles legitimate approximation, according to the paper.Source: §3.4.3 · pp. 6–7

3.5 CHI: Composite Hallucination Index

  • CHI uses harmonic-mean fusion to penalize weakness in any dimension and adapts when a dimension is undefined by using the computable dimensions.The paper gives documents without numerical quantities as an example of an undefined QHI dimension.Source: §3.5 · p. 7

4 Experimental Setup

The setup evaluates CHI through a pipeline from data and generated summaries to metric computation and correlation with human judgments. It combines a multi-domain analysis with a primary human-annotated benchmark and evaluates metrics at system and summary levels.

4 Experimental Setup

  • The evaluation pipeline runs from data through generation and metric computation to correlations with human judgments at system and summary levels.Figure 2 depicts this sequence.Source: §4 · p. 7, Figure 2 · p. 8

4.1 Datasets

  • 800 source articles span four domains, chosen to vary the prominence of entity, relation, and quantity challenges.News is entity-dense, medical abstracts have complex relations, legal documents mix entities and numerical clauses, and financial reports are quantity-rich.Source: §4.1 · p. 8

4.2 Evaluation Design: Two Complementary Tracks

  • The primary SummEval track uses 100 CNN/DailyMail articles summarized by 16 systems, yielding 1,600 human-scored summary-article pairs for correlation analysis against existing human scores.The supplementary track applies five systems to an 800-article multi-domain corpus, with a design spanning the full faithfulness range for discrimination testing.Source: §4.2 · pp. 8–9

4.3 Baselines

  • The reported metric set spans ROUGE-L longest-common-subsequence overlap, BERTScore [23] contextual embedding similarity, and AlignScore [22] and SummaC [10] NLI-based consistency measures.FActScore [15] and MiniCheck [19] are referenced but not recomputed because they require retrieval pipelines and specialized model infrastructure beyond scope; GPT-4-as-judge is not recomputed because API access costs $0.15/document.Source: §4.3 · p. 9

4.4 Evaluation Protocol

  • System-level correlation compares system-averaged metric and human scores across 16 systems; summary-level correlation compares scores for 1,600 individual summaries.Three annotators rate faithfulness dimensions and overall faithfulness; summary-level analysis faces annotation noise and a consistency ceiling effect, with 81.6% scoring 5/5.Source: §4.4 · pp. 9–10

5 Results

CHI’s system-level correlations with human judgments are strongest on SummEval, while its dimensions show low intercorrelation and distinct principal components. The results also report validation of RHI∗ and QHI, and an ablation in which the full composite outperforms subsets.

5.1 RHI∗Validation

  • Table 2 compares RHI∗ with published linear RHI [6]; its caption reports high inter-variant correlation as evidence of architectural consistency.The validation tests whether softmax RHI∗ maintains fidelity with the linear variant.Source: Table 2 · p. 10, §5.1 · p. 10

5.2 QHI Standalone Validation

  • QHI is validated against human quantity-faithfulness judgments, with Table 3 also identifying a subset restricted to sentences containing numerical content.Source: Table 3 · p. 10, §5.2 · p. 10

5.3 Orthogonality Analysis

  • The CHI dimensions have low pairwise correlations (mean ρ < 0.3), and PCA places them on three distinct components with eigenvalues > 0.8.These results support the claim that the dimensions capture distinct, nonredundant phenomena.Source: Table 4 · p. 11, §5.3 · p. 11

5.4 CHI vs. Baselines

  • On SummEval’s 16 systems, CHI achieves the highest system-level Average correlation (ρ = 0.66) and Relevance correlation (ρ = 0.70), while RHI∗ leads Consistency (ρ = 0.51).Table 5 reports CHI’s system-level correlation as ρ = 0.66 (p = 0.006).Source: Figure 3 · p. 11, Table 5 · p. 12

5.6 Per-Domain Analysis

  • CHI’s system-level advantage contrasts with modest summary-level correlations; its three sub-indices identify the dominant hallucination type when the composite is low.The reported summary-level correlations include AlignScore ρ = 0.14 and SummaC ρ = 0.13; 81.6% of SummEval summaries score 5/5 on consistency.Source: §5.6 · p. 11, Table 5 · p. 12

5.5 Ablation Study

  • Each CHI dimension contributes unique variance: the full three-way composite significantly outperforms all subsets (p < 0.05, Williams’ test).The ablation compares versions formed by removing dimensions individually and in pairs.Source: §5.5 · p. 11, Table 6 · p. 12

6 Analysis

CHI’s dimensions distinguish model-level ranking from single-summary entailment checks, while its case studies show how it diagnoses isolated and combined errors. The analysis also identifies omission and extraction-related failure cases that constrain its use.

6.1 Why MiniCheck Wins at Summary-Level but Loses at System-Level

  • MiniCheck’s ceiling limits system discrimination, while CHI’s dimensional variation accompanies stronger system-level correlation (ρ = 0.66 vs. ρ = 0.41).MiniCheck scores seven systems perfectly; CHI is preferred for model ranking and error diagnosis.Source: §6.1 · pp. 12–13, Table 7 · p. 13
  • MiniCheck can score an incomplete summary perfectly because entailment checks do not detect omitted source information, including a material loan-loss disclosure.CHI penalizes the omission through its Lost Focus factor; the passage identifies regulated domains as especially sensitive to material omissions.Source: §6.1 · p. 13

6.2 Case Studies

  • Across three examples, CHI isolates entity-only, quantity-only, and simultaneous entity, relation, and quantity errors.In the quantity-only example, omitting QHI raises the EHI+RHI harmonic mean to 0.90 despite a fabricated percentage.Source: §6.2 · pp. 13–14

6.3 Error Type Distribution

  • Financial texts show disproportionate quantity errors, while medical texts show high relational error rates.Source: Table 9 · p. 14

6.4 Failure Cases

  • CHI may miss implicit quantities, multi-hop relations, complex coreference, and domain-specific notation; proposed directions include domain-adapted extraction and discourse-level relation modeling.Source: §6.4 · p. 14

7 Domain-Enhanced CHI: Knowledge Base Integration

Domain-enhanced CHI uses knowledge-base integration to address false penalties from surface-form entity matching. In a medical-text evaluation, UMLS normalization improved EHI scores, and the authors propose extending domain grounding to other dimensions.

7.1 UMLS-Normalized EHI

  • Surface-form matching can penalize faithful synonyms as negative hallucinations; UMLS maps extracted entities to canonical concepts before set operations.For example, “aspirin,” “ASA,” and “acetylsalicylic acid” resolve to the same concept.Source: §7 · p. 14, §7.1 · p. 15

7.2 Quantitative Impact on Medical Text

  • 88% of evaluated articles improved with UMLS normalization; mean EHI improvement was +0.007, reaching +0.11 on synonym-heavy text.Source: §7.2 · p. 15
  • With medical synonyms, NH drops from ∼1.0 to ∼0.2, while EF rises from 0 to 0.72.Source: §7.2 · p. 15, Table 10 · p. 15

7.3 Implications for CHI

  • Domain grounding can improve specialized-text accuracy without changing CHI’s core mathematical framework; only the extraction layer changes.The authors identify extensions to relation-predicate synonyms and quantity-unit normalization, while leaving full integration for future work.Source: §7.3 · p. 15

8 Conclusion

CHI unifies entity, relation, and quantity hallucination dimensions, with a harmonic-mean composite and decomposable diagnostics. On SummEval, it achieves the highest reported system-level correlation with human judgments among evaluated metrics.

8 Conclusion

  • CHI combines EHI, RHI∗, and QHI in a unified softmax architecture with harmonic-mean fusion, yielding a decomposable composite score.The conclusion identifies empirical orthogonality among the three dimensions as evidence that they capture distinct error phenomena.Source: §8 · p. 15
  • CHI achieves the highest system-level correlation with human judgments on SummEval (ρ = 0.66, p = 0.006), exceeding ROUGE-1 (ρ = 0.53), ROUGE-2 (ρ = 0.55), EHI (ρ = 0.58), RHI∗ (ρ = 0.56), and QHI (ρ = 0.39).Source: §8 · p. 15
  • Future work includes domain-adaptive fusion weights, cross-lingual CHI, and using decomposed dimensions as shaped rewards in RLHF.Source: §8 · p. 16

Limitations

CHI’s extraction components constrain what it can detect, and its orthogonality assumption may not generalize beyond the four evaluated domains. The current evaluation also covers only English and four domains.

Limitations

  • Relation extraction from SVO tuples captures only binary predicates, while quantity extraction may miss domain-specific numerical notation.Entity extraction also depends on NER quality, which varies across domains and languages.Source: Limitations · p. 16
  • The empirically validated orthogonality assumption may not hold universally, especially in technical domains where entities can be quantities.The paper gives chemical compounds with molecular weights as an example of a setting that could show higher inter-dimensional correlation.Source: Limitations · p. 16
  • Evaluation is limited to English and four domains; cross-lingual generalization and broader domain coverage remain for future validation.Source: Limitations · p. 16

Ethics Statement

CHI is an evaluation metric, not a system for generating or deploying text. The authors position it as a complement to human evaluation and use existing SummEval annotations rather than collecting new human data.

Ethics Statement

Related papers

Loading 2609.33343v1…