Source-linked AI summary
A Citation-Grounded Benchmark for Trustworthy Earnings Call Transcript Analysis with Large Language Models
Yingzhu Zhao, Vlad Pandelea, Han Yuan, Bo Hu, Wuqiong Luo, Li Zhang, Zheng Ma
TL;DR
Evaluating citation-grounded financial analysis is costly to annotate, and long earnings-call question-answer contexts add complexity. The paper benchmarks answers using numeric evidence evaluation and an automated ECTs-100 dataset; results show strong groundedness but persistent correctness challenges, including when evidence is insufficient.
Problem
Evaluating claim-citation pairs in earnings call analysis requires expert annotation that is costly to scale, while real-world long context-question-answer triplets make the task more complex.
Source: §1 · pp. 1–2Method
The paper introduces numeric evidence evaluation and an automated dataset construction pipeline producing ECTs-100 to benchmark groundedness and correctness on earnings call transcripts.
Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 3Results
Takeaways & Limitations
Trustworthy ECT analysis must assess both whether claims are grounded and whether answers are correct when evidence is insufficient.
Source: §1 · p. 2, §4 · p. 7The study evaluates earnings call transcripts averaging 9,373.84 words, while Form 10-Q and 10-K filings typically exceed 25,000 words.
Source: A Limitations · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations. Empirical results show that LLMs perform well in groundedness but face notable limitations in correctness, with informational insufficiency presenting an additional challenge.
1 Introduction
Earnings call transcripts are dense financial documents, and evaluating citation-grounded answers is costly because expert annotation does not scale easily. The paper proposes numeric evidence evaluation and ECTs-100 to benchmark groundedness and correctness, finding that grounded answers may still be inaccurate when evidence is insufficient.
1 Introduction
- Expert annotation is costly and difficult to scale, while model-based alternatives can show model-dependent preferences and long context-question-answer triplets add complexity.ECTs are jargon-heavy and content-dense, making analysis of their unstructured content challenging.Source: §1 · pp. 1–2
- The paper proposes numeric evidence evaluation, which compares financially salient numbers across claims, citations, and source documents as an annotation-free groundedness proxy.It benchmarks long questions over ECTs and introduces an automated pipeline and ECTs-100 for evaluating both correctness and groundedness.Source: §1 · p. 2
- LLMs show strong groundedness but persistent correctness limitations, and informational insufficiency presents an additional challenge.A grounded claim-citation pair can still fail to answer the question accurately.Source: §1 · p. 2
2 Methods
The methods distinguish evidence support from answer correctness, then use numeric evidence evaluation and an automated figure-level dataset pipeline to benchmark both in long-context settings.
2.1 Preliminaries
- Groundedness means a claim or citation is explicitly supported by context, whereas correctness also requires identifying the information the question asks for.A correct answer should be grounded, but a grounded answer need not be correct.Source: §2.1 · p. 2
2.2 Numeric evidence evaluation
- Numeric evidence evaluation compares salient numbers across generated claims, citations, ground-truth answers, and source contexts using normalized exact matches.It treats ranges as atomic items and excludes identifier-like numbers such as dates and section labels; numeric items are represented by a text span, value, type, and optional unit.Source: §2.2 · p. 3
2.3 Automated dataset construction
- For each salient context figure, the dataset pipeline generates a sub-question, uses the figure as the ground-truth answer, and cites its source sentence.Each figure-level question-answer pair is evaluated against the full source context, retaining the long-context setting.Source: §2.3 · p. 3
2.4.1 Groundedness
- Groundedness is evaluated through claim–citation, claim–context, and citation–context relations, measured by proportions of normalized exact numeric matches.These complementary relations assess support between claims, citations, and source context.Source: §2.4.1 · p. 4
2.4.2 Correctness
- Correctness metrics compare numeric figures in generated answers with ground-truth answers and compare generated citation identifiers with reference evidence.Claim precision, recall, and F1 distinguish whether outputs provide the required information from whether they are grounded.Source: §2.4.2 · p. 4
3 Experiments
The experiments benchmark citation-grounded analysis across six generation baselines, assess correctness using ECTs-100, and test whether models abstain when evidence is insufficient.
3.1 Baseline methods
- Six GPT-4.1 zero-shot baselines combine Vanilla, PostCite, or LLMR generation with index- or text-based citations.Vanilla jointly generates claims and citations; PostCite generates claims first; LLMR selects citations first and generates claims from them.Source: §3.1 · p. 4
3.2.1 Benchmark setup
- The benchmark uses fourth-quarter 2024 calls from the top 100 S&P 500 constituents, includes full transcripts, and poses five long questions across financial themes.The Cigna Group replaces Berkshire Hathaway because Berkshire does not hold regular earnings calls.Source: §3.2.1 · p. 5
3.2.2 Benchmark results
- Claim lengths across the six methods cluster around 100–300 words, alongside long transcripts and expert-designed questions.Figure 1 presents the claim-word distributions.Source: §3.2.2 · p. 5, Figure 1 · p. 5
- All six baselines show high claim–context and citation–context groundedness; LLMR-text leads claim–citation groundedness, with LLMR-index comparable.LLMR-index scores 0.977 on claim–context, 0.993 on citation–context, and 0.927 on claim–citation groundedness.Source: §3.2.2 · p. 5
3.3.1 ECTs-100
- ECTs-100 pairs transcript context and questions with ground-truth claims and citations, enabling comparison against both source documents and reference answers.Its questions target salient financial figures already cited in the groundedness benchmark.Source: §3.3 · p. 5, §3.3.1 · p. 6
3.3.2 Benchmark results
- 0.9721 (0.0032) citation precision contrasts with 0.7139 (0.0087) citation recall on ECTs-100, while normalized claim F1 is 0.6468 (0.0080).Values are mean (SE); the authors identify citation coverage, rather than citation accuracy, as the main limitation.Source: §3.3.2 · p. 6
3.4 Conscious incompetence
- 0.4834 (0.0149) is the average hallucination rate when target-figure citations are removed, despite the remaining context being unchanged.Outputs count as hallucinated if they include salient-number claims or citations when the question is informationally insufficient.Source: §3.4 · p. 6, Figure 3 · p. 6
4 Conclusion
The conclusion summarizes the benchmark, its evaluation and dataset contributions, and the main challenges observed.
4 Conclusion
- LLMs are generally reliable in groundedness but face challenges in correctness over long context-question-answer triplets and under informational insufficiency.The paper introduces numeric evidence evaluation and an automated pipeline for ECTs-100 to benchmark these dimensions.Source: §4 · p. 7
5 Disclaimer
The disclaimer states that the paper should not be taken as representing or supporting American Express’s views or business practices.
5 Disclaimer
- The paper is for academic informational purposes and does not represent or support American Express or its affiliates’ views, policies, positions, or practices.It also says the paper should not be cited or relied upon as evidence of those business views or practices.Source: §5 · p. 7
A Limitations
The authors describe boundaries of the study’s data, evaluation, and automated references, and note ECTs-100’s research-oriented scope.
A Limitations
- NEE is a scalable numeric-alignment proxy, not a complete entailment measure: identical figures may match across entities or periods, and nonnumeric claim support is not evaluated.The authors also did not validate NEE scores against human or expert groundedness judgments; results use GPT-4.1 baselines and automatically constructed references.Source: A Limitations · p. 10
B Ethical considerations
- ECTs-100 uses public transcripts and objective financial-figure questions intended to reduce subjective interpretation, and is designated for research use.Source: B Ethical considerations · p. 11
C.1 ECTs analysis
ECTs have supported a range of financial and non-financial NLP tasks. ECTs-100 distinguishes itself through detailed claim-and-citation QA pairs, long-form triplets, and coverage across 100 companies.
C.1 ECTs analysis
- ECTs-100 pairs specific QA claims with citations, unlike summary datasets that present comprehensive lists of points.Its long-form question-answer triplets also differ from work centered on short queries and limited company coverage.Source: §C.1 · p. 11
- Prior ECT research also uses transcripts for stock-price, earnings-surprise, risk-type, strategy-maturity, subjectivity, argument-relation, speech-recognition, and vagueness tasks.The listed tasks output continuous or categorical values, or address non-financial analysis.Source: §C.1 · p. 11
C.2 Grounded analysis
Grounded responses pair claims with source citations, supporting independent validation, but assessing whether citations support claims remains a challenge. The literature has moved toward models that select evidence within retrieval-augmented generation and can serve directly as retrievers.
C.2 Grounded analysis
- Grounded answers pair claims with verifiable citations, enabling independent validation and increasing user trust.The passage notes that financial analysis can be affected by factual inaccuracies in otherwise coherent, persuasive model outputs.Source: §C.2 · p. 11
- Human assessment found only 51.5% of generated claims supported by accompanying citations in a late-2023 generative-search evaluation.The passage identifies claim-citation evaluation as challenging.Source: §C.2 · p. 11
- Recent approaches have LLMs selectively incorporate retrieved evidence in RAG and, in newer work, act directly as evidence retrievers.One described method jointly generates claims and citations, then refines citations based on initial outputs.Source: §C.2 · p. 11
D.1 Groundedness
The evaluation separates groundedness into three numeric-evidence relationships, then assesses correctness against reference answers and citations so grounded but irrelevant outputs are not counted as sufficient.
D.1 Groundedness
- Groundedness measures numeric support across claim–citation, claim–context, and citation–context relationships, providing a more hierarchical assessment than work evaluating only one or two.The respective ratios track numeric claims supported by citations, claims matched to source context, and citations matched to source context; the latter ratios quantify numeric hallucination in claims and citations.Source: §D.1 · p. 12
D.2 Correctness
- Correctness compares generated answers with ground-truth answers to check whether they recover the figures required by each question.Claim precision, recall, and F1 measure generated figures matching ground-truth figures, using normalized numeric exact match.Source: §D.2 · p. 12
- Citation correctness uses identifier matching rather than exact citation-text matching to assess recovery of ground-truth evidence.The resulting metrics assess whether outputs provide required answers and corresponding citations, not merely numerically grounded but irrelevant content.Source: §D.2 · pp. 12–13
E Questions
The appendix presents questions covering five ECT analysis themes. The paper checklist answers yes that the abstract and introduction accurately reflect the paper’s contributions and scope.
E Questions
- The five ECT question themes cover discussion focus and outlook, risk assessment, strategic growth and long-term planning, macroeconomic impact, and financial and capital allocation management.Figures 4–8 present the detailed questions corresponding to the five themes.Source: E Questions · p. 13, Figure 4 · p. 13, Figure 5 · p. 13
NeurIPS Paper Checklist
- The checklist answers yes that the abstract and introduction accurately reflect the paper’s contributions and scope.Its guidance asks that stated claims match the results and reflect expected generalizability.Source: NeurIPS Paper Checklist · p. 14
2. Limitations
The study’s stated boundaries include public-dataset coverage, no statistical analysis, and an empirical focus rather than formal theoretical analysis.
2. Limitations
- Formal theoretical analysis is outside the study’s scope, which focuses on experimental evaluation of LLMs.The authors cite detailed benchmark-construction descriptions as their approach to rigor.Source: §2 · p. 15
- No statistical analysis is performed, although the authors say the substantial number of test ECTs improves result stability.Source: §2 · p. 17
- The evaluation is confined to publicly available datasets and to tasks using content within those datasets.The paper states that the tasks do not involve private or sensitive information.Source: §2 · p. 18