Source-linked AI summary
A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers
Nadhem Zmandar, Mo El-Haj, Paul Rayson
TL;DR
Financial annual reports are long, and the study examines how summarization methods and evaluation metrics perform on UK financial narratives. It benchmarks transformer-based and other approaches, proposes BRUGEscore, and reports that it outperformed ROUGE variants. Its conclusions are bounded by reliance on extracted reference sections and an English-language dataset.
Problem
Method
Results
Takeaways & Limitations
The authors conclude that their BRUGEscore and model-based evaluation metrics provide alternatives to ROUGE for assessing financial summaries.
Source: §4 · p. 6, §7 · p. 8The study uses extracted annual-report sections rather than domain-expert human summaries, and its results are limited to an English dataset.
Source: §8 · p. 9
Abstract
from arXiv · showhide
There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year. UK annual reports are very lengthy documents with around 80 pages on average. In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques. In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries. We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore. Finally, we perform a statistical significance test on our results to verify whether they are statistically robust, alongside an adversarial analysis task with three different corruption methods.
1 Introduction
Financial narratives are expanding as regulated companies communicate results to stakeholders, who use these disclosures to assess value creation. This study benchmarks summarization methods for UK financial narratives and examines evaluation metrics, statistical testing, and BRUGEscore.
1 Introduction
- Regulated financial markets require listed companies to report regularly, and stakeholders use financial communications to assess how well firms create value.Relevant stakeholders include investors, shareholders, customers, employees, analysts, regulators, lenders, rating agencies, and suppliers.Source: §1 · p. 1
- The study benchmarks summarization methods for UK financial narratives and examines long-document methods and their interactions with evaluation metrics, including ROUGE..Source: §1 · p. 2
- It also introduces a statistical testing method for generated financial summaries and proposes the BRUGEscore metric.Source: §1 · p. 2
2 Background
The background reviews summary-evaluation metrics and financial summarization work, then positions the study around transformer choices, metric correlations, and a block-based approach for long reports.
2 Background
- ROUGE measures overlap with human references, while METEOR aligns candidate and reference unigrams using stemming, synonyms, and paraphrastic matches.Source: §2 · p. 2
- BERTScore aligns contextualized token embeddings, BARTScore offers generative-quality perspectives, and Bleurt compares candidate and reference meaning.Source: §2 · p. 2
- BRUGEscore is the harmonic mean of ROUGE-2 and BERTScore, combining word overlap with embedding cosine similarity.Source: §2 · p. 2
2.1 Related work
- Financial summarization work ranges from sentence-linkage and query-based systems to statistical heuristics and the FNS tasks, whose 2020 edition reported large-scale results on UK annual reports.Source: §2.1 · pp. 2–3
- Prior UK-report approaches include T5, Pointer Networks, Determinantal Point Processes, financial embeddings with knowledge features, and end-to-end training.Source: §2.1 · p. 3
- This study compares transformer architectures and pretraining tasks and data, and examines automated-metric correlations for long, unstructured, technical, and subjective UK reports.Source: §2.1 · p. 3
- Block-based summarization predicts the summary's start and locates the matching report passage, adapting sequence-to-sequence transformers to long documents with continuously extracted references.Source: §2.1 · pp. 3–4
- The methods span transformer, reinforcement-learning, LSA, BERT-extractive, and SBERT-extractive approaches, compared with toplines, baselines, and Lead-1000.Source: §2.1 · p. 4
3 Dataset
The study uses English UK annual reports from the FNS 2021 shared task, with multiple extracted report sections serving as gold summaries. The dataset contains thousands of reports and summaries.
3 Dataset
- The FNS 2021 dataset contains 3,863 LSE-company annual reports from 2002–2017, averaging 52,000 tokens, and 9,873 gold summaries.Source: §3 · p. 4
- The reports are in English, and the dataset is randomly split into training (75%), testing, and validation (25%).Source: §3 · p. 4
- Each report has one to five gold summaries, drawn from Financial Highlights, the Letter to Shareholders, Financial Statements, and the Auditor’s Report.Multiple references were used because the study lacked a human-generated reference summary as a gold standard.Source: §3 · p. 4
4 Experimental Work
The experiments compare transformer summarizers and training strategies on financial reports, evaluate them with multiple metrics, and examine how those metrics relate. The reported results favor T5 and Longformer Encoder-Decoder among text-to-text models, while the metric correlations suggest that some measures overlap.
4 Experimental Work
- The experiments compare T5, LongFormer Encoder-Decoder, BART, Pegasus, and BERT, and train T5, Pegasus, and BART using alternative reference-summary strategies.After multi-reference training did not significantly improve ROUGE and proved computationally expensive, the authors selected one reference per report by its ROUGE-2 score.Source: §4 · p. 5
- Hyperparameters were selected to maximize model input and target lengths rather than through a comprehensive grid search.The authors cite the computational power and time required for grid search as the reason for using a simpler strategy.Source: §4 · p. 5, Table 3 · p. 5
- The evaluation compares model versions and baselines using multiple metrics, including ROUGE variants, BERTScore, BARTScore, BLEURT, METEOR, and BRUGEscore.The reported F-measures are ranked by ROUGE-2 F1.Source: §4 · p. 5, Table 4 · p. 7
- T5 performs best among text-to-text models, alongside Longformer Encoder-Decoder; LED base is memory-efficient, while LED Large is constrained by GPU memory.Lead-1000 is also reported as a strong benchmark, with transformer summarization outperforming deep-learning and reinforcement-learning methods.Source: §4 · p. 6, Table 4 · p. 7
- ROUGE variants are highly correlated, BERTScore and BARTScore are highly correlated with each other, and neither correlates with the ROUGE variants.The displayed matrix uses T5 summaries; the authors report similar patterns for other transformers.Source: §4 · pp. 6–7, Figure 1 · p. 6
5 Statistical significance
The paper tests whether observed system-performance differences are statistically significant, rather than treating a higher metric score alone as proof of superiority. It uses bootstrap testing and interprets p-values against a 0.1 significance level.
5 Statistical significance
- A higher evaluation score alone is insufficient to establish that one system outperforms another; the difference must also be statistically significant.The paper frames this as a test between a no-difference null hypothesis and a difference alternative hypothesis.Source: §5 · p. 7
- The authors use a bootstrap test and report p-values for ROUGE-2, ROUGE-L, BERTScore, and BLEURT.The p-values are obtained from differences between series of results generated by each system.Source: §5 · p. 7
- The reported p-values are compared with a 0.1 significance level; uncolored cells indicate statistically insignificant differences.For those cells, the paper says one system cannot be claimed to outperform the other using that metric.Source: §5 · p. 7
6 Adversarial analysis
The adversarial analysis tests evaluation metrics on corrupted T5-small summaries using mask-filling, word dropping, and adjacent-token permutation. BERTScore and BARTScore achieve 60% accuracy across the three tasks, while metric performance varies with corruption type.
6 Adversarial analysis
- The analysis corrupts T5-small summaries using BERT mask-filling, word dropping, and adjacent-token permutation, with corruption distributed across chunks of 4, 6, 8, or 10.The reported Table 5 comparison uses chunk length 10.Source: §6 · p. 8
- Under strict comparison, ROUGE-1 scored 0.00 accuracy for word permutation because it is insensitive to syntax.Accuracy measures the percentage of uncorrupted summaries assigned better scores than their corrupted counterparts; ties do not count as better.Source: §6 · p. 8
- 60% accuracy was achieved by both BERTScore and BARTScore across the three adversarial tasks.The paper suggests ROUGE is better suited to extractive summarization and model-based metrics to abstractive summarization.Source: §6 · p. 8
- ROUGE-2 performed best on word permutation and BERT mask-filling, while ROUGE-3 performed best on word dropping.The authors report that a single-token corruption disrupts n-gram sequences for ROUGE-n when n is greater than 1.Source: §6 · p. 8
7 Conclusion and Future Work
The paper evaluates transformer-based extractive summarization and metrics for UK annual reports, including BRUGEscore, and identifies future evaluation and reference-summary work.
7 Conclusion and Future Work
- BRUGEscore and other proposed model-based metrics outperformed ROUGE metric variants in the financial annual-report summarization task.The paper also analyzed generated summaries with adversarial analysis to assess metric robustness.Source: §7 · p. 8
- Adversarial evaluation compared scores for original and corrupted T5-small summaries across three corruption tasks, with standard deviations below 0.2%.The experiment used the FNS dataset and introduced a corruption every 10 chunks.Source: Table 5 · p. 9
- Future work includes human evaluation, measuring correlation with existing metrics, and improving the quality of reference summaries.The models are hosted in a private Hugging Face repository and are planned for release after paper acceptance.Source: §7 · p. 9
8 Limitations
The study identifies reference-summary quality and English-only data as limitations, alongside computational demands and financial-domain language needs.
8 Limitations
- Extracted annual-report sections serve as gold summaries because domain-expert human summaries are lacking.The paper identifies the lack of expert-generated gold standards as a major technical challenge for financial summarization research.Source: §8 · p. 9
- Results are limited to an English dataset, so metric performance on other languages is not guaranteed, especially for English-pretrained language models.Source: §8 · p. 9
- Large financial datasets require substantial computational capacity, and the paper identifies a need for pretrained models tailored to financial jargon.Source: §8 · p. 9
Appendix A
Appendix A presents bootstrap-test p-values for BERTScore, BLEURT, ROUGE-2, and ROUGE-L across system comparisons.
Appendix A
- Table 6 reports bootstrap-test p-values for BERTScore, with each column covering one system and the remaining systems.Source: Table 6 · p. 11
- Table 7 reports bootstrap-test p-values for BLEURT, with each column covering one system and the remaining systems.Source: Table 7 · p. 12
- Table 8 reports bootstrap-test p-values for ROUGE-2, with each column covering one system and the remaining systems.Source: Table 8 · p. 12
- Table 9 reports bootstrap-test p-values for ROUGE-L, with each column covering one system and the remaining systems.Source: Table 9 · p. 12