PaperlayerView exact revision on arXiv

Source-linked AI summary

LOOMSUM:Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarization

Meng Zhou, Wenhao You, Wei Yuan

arXiv:2609.00241v1cs.CL

TL;DR

Long text–table documents distribute evidence across narrative passages and tables, and supported quantities and analyses can still be incorrectly associated. LOOMSUM links source-grounded evidence and plans its discourse, while TGF separately scores grounding, analysis, and relations. Experiments report improved analytical faithfulness with strong summary quality, and human evaluation supports TGF’s component judgments.

  • Problem

    Long text–table summaries may combine individually supported quantities and analyses into an unsupported relation, making faithful integration across distant evidence challenging.

    Source: §1 · p. 1

  • Method

    LOOMSUM extracts source-grounded claims, links table-derived facts to narrative interpretations, and plans discourse before generating summaries; TGF separately evaluates grounding, analysis support, and relation consistency.

    Source: §6 · p. 9

  • Results

    Experiments on FINDSum and USTT show improved analytical faithfulness without compromising reference-based quality, while human evaluation supports TGF and finds stronger human-judgment alignment for Relation Consistency than generic factuality metrics.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Faithful long text–table summarization requires preserving relations between quantitative facts and narrative interpretations, not only grounding individual facts.

    Source: §6 · p. 9

    LOOMSUM is an initial, non-exhaustive instantiation of relation-aware summarization, and its modular pipeline may propagate errors across stages.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long documents often distribute important information across extensive narrative passages and multiple tables, making faithful summarization particularly challenging. Existing methods may generate individually supported quantitative facts and analytical statements yet associate them incorrectly, producing quantitatively plausible yet analytically unfaithful summaries. In this work, we propose LOOMSUM, a training-free framework that extracts source-grounded atomic evidence, explicitly links table-derived facts with supporting narrative analyses, and plans the discourse structure before generation. We also introduce Table-Grounded Faithfulness (TGF), a claim-level metric that separately evaluates Numeric Grounding, Analysis Support, and Relation Consistency. Experiments on the text--table summarization benchmarks FINDSum and USTT show that LOOMSUM improves analytical faithfulness while maintaining strong summarization quality. Human evaluation finds positive component-level associations with the corresponding human judgments. Our Relation Consistency metric further shows stronger agreement with human relation judgments than generic factuality metrics, indicating that explicit cross-modal linking helps reduce errors in which supported quantities are paired with incorrect narrative interpretations. Together, these findings show that faithful long text--table summarization requires not only grounding individual facts, but also preserving the relations between them.

1 Introduction

Long text–table summaries can combine individually supported quantities and analyses into an unsupported relation. LOOMSUM links these evidence types before generation, and TGF evaluates both their grounding and their relation.

1 Introduction

  • Quantitative–narrative misalignment occurs when supported quantities and analyses are combined through an unsupported relation.A revenue increase, for example, can be attributed to an explanation concerning a different metric, entity, or period.Source: §1 · p. 1, PROPOSED METHOD (LINK PRESERVED) · p. 3

DIRECT SUMMARY (LINK OMITTED)

PROPOSED METHOD (LINK PRESERVED)

  • Experiments on FINDSum and USTT show improved analytical faithfulness while maintaining competitive summary quality and content coverage; human evaluation supports TGF, particularly for Numeric Grounding and Relation Consistency.The Relation Consistency component aligns more closely with human judgments than generic factuality metrics.Source: PROPOSED METHOD (LINK PRESERVED) · pp. 2–3
  • LOOMSUM constructs source-grounded atomic claims, explicitly links quantitative facts with narrative interpretations, and organizes them through discourse planning before generation.The framework is training-free and preserves provenance for the linked evidence.Source: PROPOSED METHOD (LINK PRESERVED) · pp. 2–3
  • TGF separately evaluates Numeric Grounding, Analysis Support, and Relation Consistency, targeting relation-level errors that existing factuality and numerical metrics do not specifically assess.The metric is reference-summary-free and source-grounded.Source: PROPOSED METHOD (LINK PRESERVED) · pp. 2–3

2 Method

LOOMSUM turns narrative and table evidence into traceable claims, links quantitative facts to their interpretations, then plans and generates a summary from the organized evidence.

2.1 Overview

  • LOOMSUM constructs source-grounded text and table claims, aligns quantitative facts with narrative interpretations, consolidates evidence, plans its organization, and generates the summary from that structure.The framework links and organizes textual and tabular evidence before generation.Source: §2.1 · p. 3

2.2 Claim Construction and Cross-Modal Alignment

  • Text claims preserve entities, quantities, units, periods, and relations, while table claims retain their table-record pointers; alignment links quantitative claims to text that explains, compares, qualifies, attributes, or contextualizes them.Each linked claim retains both components, their asserted relation, and the provenance of both.Source: §2.2 · pp. 3–4

2.3 Claim Consolidation and Planning

  • Claims are merged only when proposition, entity, metric, period, direction, and value agree; conflicting claims remain separate, and a planner orders the resulting evidence and specifies relations to preserve.Provenance from consolidated instances remains traceable.Source: §2.3 · p. 4

2.4 Summary Generation

  • The generator uses demonstrations selected by structural properties to realize the discourse plan, and each generated sentence carries identifiers for its supporting claims.The demonstrations show how to compress and order evidence while preserving quantitative–narrative relations.Source: §2.4 · p. 4

3 Table-Grounded Faithfulness Score

Table-Grounded Faithfulness (TGF) scores generated claims against source evidence along three dimensions, then aggregates each claim according to its applicable evidence requirements.

3 Table-Grounded Faithfulness Score

  • For each claim and applicable dimension, an LLM judge rates support against retrieved source evidence on a five-point rubric, with 1 meaning unsupported or contradicted and 5 meaning fully supported.Repeated judgments are used because the judge does not provide calibrated probabilities over the ratings.Source: §3 · p. 4
  • Numeric Grounding checks quantitative content and context, Analysis Support checks whether analyses are licensed by evidence, and Relation Consistency checks the association between quantitative and narrative components.A correct value tied to the wrong metric, entity, unit, or period does not receive full Numeric Grounding credit.Source: §3 · p. 5
  • A cross-modal claim receives the minimum of its Numeric Grounding, Analysis Support, and Relation Consistency scores, so stronger components cannot compensate for a weak required link.TGF then averages claim scores for a summary and macroaverages document scores across the test set.Source: §3 · p. 5

4 Experiments

The experiments ask whether LOOMSUM improves summary quality, numeric coverage, and table-grounded faithfulness, how its components contribute, and whether TGF aligns with human judgments. They use English and Chinese financial text–table benchmarks and compare methods across input modalities.

4 Experiments

  • The experiments address overall effectiveness, component contributions, and whether TGF aligns with human judgments while capturing relation-level errors overlooked by generic factuality metrics.The overall question covers summary quality, numeric content coverage, and table-grounded faithfulness.Source: §4 · p. 5

4.1 Settings

  • FINDSum includes English long-document financial summarization subsets for revenue, expenses, and profitability (ROO) and for liquidity, cash flows, debt, and capital resources; USTT is a Chinese benchmark with a table, associated narrative, and human-written summary.All datasets use their official test split.Source: §4.1 · p. 5
  • Baselines cover text-only, table-only, and joint text–table inputs, including direct long-context generation and retrieval-based summarization.Unless otherwise stated, LLM-based methods use GPT-5-nano with the same decoding configuration as LOOMSUM.Source: §4.1 · p. 6
  • Evaluation combines reference-overlap and semantic-similarity metrics, numerical precision, coverage, and selection, plus AlignScore, TGF, and Grounded Claim Coverage.Grounded Claim Coverage complements precision-oriented TGF by measuring reference claims covered by generated claims.Source: §4.1 · p. 6

5 Results

Across FINDSum and USTT, LOOMSUM leads key coverage and faithfulness measures. Human evaluation, ablations, and a case study examine relation judgments and evidence linking.

5.1 Results on FINDSum (RQ1); 5.2 Results on USTT (RQ1)

  • LOOMSUM leads most reported FINDSum metrics, including NS and TGF; on USTT it leads R-1, R-L, and NS, while TextRank leads TGF.Table-only methods perform substantially worse than text-based and multimodal counterparts on USTT.Source: §5.1 · p. 6, §5.2 · p. 7, Table 1 · p. 8

5.1 Results on FINDSum (RQ1)

  • 74 NG, 72 AS, and 69 RC on FINDSum-ROO, and 82 NG, 80 AS, and 77 RC on Liquidity, each outperform the strongest baseline.The reported component gains are progressively larger from Numeric Grounding to Relation Consistency.Source: §5.1 · p. 6

5.2 Results on USTT (RQ1)

  • TextRank leads USTT TGF and NG, whereas LOOMSUM leads AS and RC; LOOMSUM also achieves the highest RC score across all three evaluation sets.TextRank’s higher faithfulness score despite lower ROUGE and NS reflects that selecting fewer directly supported claims can omit salient information.Source: §5.2 · p. 7

5.3 Results on Human Evaluation (RQ3)

  • TGF-RC correlates most strongly with human RC judgments (ρ = 0.598, τb = 0.571, p < 0.05); TGF agrees more strongly than AlignScore and SummaC across dimensions and correlation tests.TGF-AS correlates positively with human judgments, but the association is not statistically significant; Figure 4 compares rank correlations across the three dimensions.Source: §5.3 · pp. 7–8, Figure 4 · p. 8

5.4 Ablations (RQ2)

  • Adding cross-modal alignment improves all metrics on both FINDSum subsets; planning and demonstrations further improve NS and TGF, although Liquidity R-L falls from 16.44 to 16.35.Performance is broadly consistent across different base models and tasks.Source: §5.4 · pp. 8–9, Table 3 · p. 8

5.5 Case Study

  • LOOMSUM links the $10.8 million provision benefit to $62.2 million post-provision income and retains its mortgage-portfolio attribution, unlike baselines that omit or distort the relation.The comparison illustrates the preserved driver–outcome link represented in Table 4.Source: §5.5 · p. 9, Table 4 · p. 9

6 Conclusion

The conclusion presents LOOMSUM and TGF as responses to quantitative–narrative misalignment in text–table summarization. Results support the importance of preserving relations between facts, not only grounding facts individually.

6 Conclusion

  • Quantitative–narrative misalignment occurs when individually supported facts and analyses are combined into unsupported relations.LOOMSUM links table-derived facts with narrative interpretations and organizes them through discourse planning; TGF evaluates Numeric Grounding, Analysis Support, and Relation Consistency.Source: §6 · p. 9
  • Experiments on FINDSum and USTT show improved analytical faithfulness without compromising reference-based quality; human evaluation finds TGF-RC aligns more closely with judgments than generic factuality metrics.The conclusion also notes future work on richer relation-aware models and broader domains.Source: §6 · p. 9
  • Faithful text–table summarization requires preserving relations between facts as well as grounding individual facts.Source: §6 · p. 9

Limitations

The paper frames LOOMSUM as an initial, lightweight investigation rather than a complete solution to quantitative–narrative misalignment. Its evaluation and TGF metric also have scope and validation limits.

Limitations

  • LOOMSUM is one possible, training-free instantiation of relation-aware summarization, and errors may propagate across its modular pipeline.The stages named are evidence extraction, cross-modal alignment, discourse planning, and generation.Source: Limitations · p. 9
  • The evaluation covers financial text–table inputs, leaving the prevalence and characteristics of the failure mode in other domains unestablished.Broader domains, languages, table structures, and document lengths are identified as future evaluation directions.Source: Limitations · pp. 9–10
  • TGF relies on automated claim extraction and LLM judgments, has limited human validation with weaker Analysis Support association, and evaluates expressed claims rather than salient-content omissions.Future work could combine relation-aware faithfulness with source-level coverage evaluation and larger, more diverse human annotations.Source: Limitations · p. 10

A Datasets Statistics

The test sets span long financial reports in FINDSum and shorter table–text inputs in USTT. Sensitivity analysis used 500 samples from each dataset because of resource limits.

A Datasets Statistics

  • FINDSum-Liquidity has 4,204 test instances, 34,630 average source tokens, and 538 average reference tokens; FINDSum-ROO has 6,201, 36,447, and 444, respectively.The reported source lengths for FINDSum include narrative content and associated tabular information.Source: A Datasets Statistics · p. 12, Table 5 · p. 12
  • USTT has 1,390 test instances, with an average source length of 1,680 tokens and an average summary length of 85 tokens.Its inputs are considerably shorter than FINDSum's and provide a complementary table–text setting.Source: A Datasets Statistics · p. 12, Table 5 · p. 12

B Sensitive Analysis of Base Model

C Additional Quantitative Results

LOOMSUM performs strongly across FINDSum and USTT, while the additional analyses show why faithfulness and reference coverage should be assessed together. Results also distinguish cross-modal methods from single-modality approaches and generic factuality metrics.

Grounded Claim Coverage

  • G-Cov credits reference claims only when semantic coverage and source grounding are strong, measuring grounded recall of salient reference content rather than exhaustive source coverage.It complements TGF, which averages faithfulness across claims that the generated summary expresses.Source: C Additional Quantitative Results · pp. 12–13

Overall performance

  • LOOMSUM leads 8/11 reported metrics on FINDSum-ROO and achieves the best R-2, R-L, BERTScore, NC, NS, G-Cov, and TGF on Liquidity.The authors report that these results indicate improved analytical faithfulness without a corresponding loss in reference-based summary quality.Source: C Additional Quantitative Results · p. 13

Numerical content selection

  • LOOMSUM has the highest NC on both FINDSum subsets, reaching 54.17 on ROO and 56.76 on Liquidity, while its NS is 27.03 on ROO and 25.98 on Liquidity.The ROO NS is nearly the best score, 27.08 from RAG-Sum; broader numerical coverage may include source-supported quantities absent from the single reference.Source: C Additional Quantitative Results · p. 13

Effect of input modality

  • Table-only methods generally score lowest on TGF and G-Cov, while methods using both tables and narrative generally perform strongest jointly.Tables provide exact quantities; narrative passages explain drivers, qualifications, and implications, and joint input alone is insufficient without correctly associating and organizing related evidence.Source: C Additional Quantitative Results · p. 13

Faithfulness–coverage trade-off

  • RAG-Sum’s TGF is 69.47 on ROO and 76.34 on Liquidity, while its G-Cov is 13.10 and 14.84, respectively.TGF measures faithfulness over generated claims, whereas G-Cov measures grounded coverage of reference claims; LOOMSUM leads both on both FINDSum subsets.Source: C Additional Quantitative Results · pp. 12–14

Comparison with generic factuality metrics

  • On Liquidity, text-only BART leads AlignScore and SummaC but scores only 63.70 TGF and 10.56 G-Cov, illustrating disagreement among metrics.Generic factuality scores may not reflect coverage of salient reference claims or whether quantities are associated with the correct narrative interpretations.Source: C Additional Quantitative Results · p. 14

Results on USTT

  • On USTT, LOOMSUM leads R-1, R-L, NP, NC, NS, AlignScore, and G-Cov; NC rises from 30.36 to 49.79 and NS from 21.44 to 27.36.TextRank has the highest TGF, while LOOMSUM is strongest on both TGF and G-Cov among joint-input methods.Source: C Additional Quantitative Results · pp. 14–15

D Additional Case Studies

The case studies illustrate two distinct errors: incorrect composition of supported facts and explanations, and grounded outputs that omit reference-relevant claims. The evaluation prompts decompose claims and assess their evidence support, including the association between quantitative and analytical content.

D Additional Case Studies

  • Direct-LC recovers major indicators in the FINDSum example but misconnects revenue growth to low rates; GCG and RAG-Sum attach valid drivers to the wrong outcomes.LOOMSUM preserves the revenue decomposition and driver chains, distinguishing loan growth’s positive effect from lower rates’ offsetting effect.Source: D Additional Case Studies · p. 15
  • In the USTT example, several methods select supported information from non-target sections yet omit the four FX claims emphasized by the reference.RAG-Sum captures the upward trend but omits exact values and cross-rate comparisons, while LOOMSUM preserves directions, values, and pairwise relations for all four claims.Source: D Additional Case Studies · p. 15, Table 10 · p. 19

E Evaluation Prompts

  • The decomposition prompt extracts exhaustive, minimal factual claims and keeps a quantitative proposition linked to its interpretation only when the summary explicitly asserts that relation.For cross-source analytic claims, it records the quantitative component, analytical component, and relation type.Source: E Evaluation Prompts · p. 16
  • The TGF prompt judges applicable numeric grounding, analysis support, and relation consistency against source evidence, using a five-point support rubric.Relation consistency applies only to explicit quantitative–analytical links; individually supported components do not by themselves establish that their association is supported.Source: E Evaluation Prompts · p. 16

Related papers

Loading 2609.00241v1…