PaperlayerView exact revision on arXiv

Source-linked AI summary

From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls

Tomas Goldsack, Yang Wang, Chenghua Lin, Chung-Chi Chen

arXiv:2410.01039v1cs.CL

TL;DR

Analytical Earnings Call reports require multifaceted analysis, a task the paper explores with a multi-agent LLM framework and specialized feedback. The study reports improved insightfulness with all agents, while noting a tendency for LLMs to favor generated reports over human-authored ones.

  • Problem

    Although journalistic Earnings Call reports have been studied, analytical-report generation remains unexplored despite its potential benefits and need for in-depth analysis across varied aspects.

    Source: §1 · p. 1

  • Method

    The paper investigates an LLM-based multi-agent framework for generating analytical Earnings Call reports, characterizing outputs and assessing LLM-based report evaluation.

    Source: §1 · pp. 1–2

  • Results

    LLMs show a detrimental tendency to favor generated over human-authored reports, while including all agents increases characteristics classified as reported and insightful across instances.

    Source: §3.3 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    All-agent feedback improves reporting of management expectations and possible future events, which the paper identifies as relevant to investment decision-making.

    Source: §3.3 · p. 5

    Open research lacks suitable reference reports because analytical reports typically originate in corporate settings.

    Source: Limitations · p. 9

Abstract

from arXiv · show

This paper explores the use of Large Language Models (LLMs) in the generation and evaluation of analytical reports derived from Earnings Calls (ECs). Addressing a current gap in research, we explore the generation of analytical reports with LLMs in a multi-agent framework, designing specialized agents that introduce diverse viewpoints and desirable topics of analysis into the report generation process. Through multiple analyses, we examine the alignment between generated and human-written reports and the impact of both individual and collective agents. Our findings suggest that the introduction of additional agents results in more insightful reports, although reports generated by human experts remain preferred in the majority of cases. Finally, we address the challenging issue of report evaluation, we examine the limitations and strengths of LLMs in assessing the quality of generated reports in different settings, revealing a significant correlation with human experts across multiple dimensions.

1 Introduction

Prior research has studied Earnings Calls for prediction and journalistic summarization, but analytical reports—more extensive, multifaceted analyses—remain unexplored. The paper investigates LLM-based multi-agent generation and report evaluation.

1 Introduction

  • Analytical-report generation for Earnings Calls remains unexplored, despite its potential benefits and the need to analyze varied technical aspects.These reports analyze meeting events, financial performance, and investment-strategy implications; relevant aspects include future operations and managers’ attitudes.Source: §1 · p. 1
  • Multi-agent LLM frameworks may contribute role specialization and diverse viewpoints to the complex task of generating analytical reports.The paper presents these features as reasons multi-agent frameworks may suit the task.Source: §1 · p. 1
  • The study develops a multi-agent framework, characterizes generated reports, and assesses LLMs’ capabilities and limitations in evaluating them.The analyses include comparisons with human-authored reports and feedback-agent contributions.Source: §1 · p. 2
  • The paper investigates generated-versus-human report differences, whether multi-agent generation yields more insightful reports, and LLM-based report evaluation.These are the three research questions addressed in the study.Source: §1 · p. 2

2 Multi-Agent Report Generation

The report-generation framework uses a Writer to draft and revise analytical reports through collaborative conversation and feedback from agents with distinct roles. Its process combines a client brief, an initial draft, and iterative feedback and revision.

2 Multi-Agent Report Generation

  • The framework uses collaborative multi-agent conversation, with a Writer drafting and revising reports based on other agents’ feedback.It was developed using AutoGen and used gpt-4-1106-preview as the underlying model.Source: §2 · p. 2
  • The process defines a Client and feedback agents, has the Writer draft from the Client’s brief, then iterates feedback and revision until completion or the preset limit.The study includes three feedback agents alongside the Writer and Client.Source: §2 · pp. 2–3
  • The Analyst uses historical financial data, including prior-quarter earnings performance, to provide comparative feedback on current performance.The agent gathers data through the AlphaVantage API and considers it alongside preceding conversation context.Source: §2 · p. 2
  • An audio-analysis agent uses phonetic statistics from management’s earnings-call utterances to inform feedback about vocal confidence and attitudes toward financial performance.The study derives PRAAT features from management-team utterances.Source: §2 · p. 2
  • A feedback agent checks report suitability for the intended audience and whether important information is retained through revisions.Source: §2 · p. 3

3 Generated Report Characterization

The characterization compares generated reports with analyst-authored reports and examines how feedback agents alter their content and style. Human ratings indicate more insightful coverage with all agents, while case examples illustrate added investor insights and financial context.

3 Generated Report Characterization

  • The study generated reports for a sample of 60 EC transcripts across multiple framework configurations and analyzed similarities and differences against 26 human-authored J.P. Morgan reports from Bloomberg Terminal.Generated reports were produced in a zero-shot setting.Source: §3 · p. 3, §3.1 · p. 3

3.1 Generated vs. Human-Authored Reports

  • Growth appears in 95% of analytical reports generated with all agents, compared with 65% of human-authored JP Morgan reports; humans emphasize financial statistics, while generated reports emphasize performance, future, earnings, and market.Source: §3.1 · pp. 3–4
  • Expert-written readability scores range from 7-10, versus 12-20 for generated reports, although the report types are generally similar in length and abstractiveness.The paper associates the generated reports’ higher scores with longer, more complex sentences.Source: §3.1 · p. 4

3.2 Impact of Feedback Agents

  • Adding the Editor or Analyst increases discussion of outlook, market, management, and future, alongside greater abstractiveness and readability scores.The paper describes the resulting discussion as broader and more speculative than transcript-focused reporting.Source: §3.2 · p. 4
  • The Psychologist shifts coverage from financial figures toward management attitude and confidence, while raising readability and abstractiveness metrics more than other agents.These were the areas the agent was designed to focus on.Source: §3.2 · p. 4

3.3 Insightfulness of Generated Reports

  • Across both systems, positive labels outweighed negative ones, and financial takeaways were most often included while management attitudes were most often absent.There were almost no “reported but not useful” labels.Source: §3.3 · p. 5, Figure 3 · p. 5
  • All-agent reports had fewer characteristics labeled “not reported” and more labeled “reported and insightful”; case examples show added investor insight, management confidence, and historical context.The clearest rating improvements concerned management expectations and possible future events.Source: §3.3 · pp. 5–6, Figure 4 · p. 6

4 Evaluating Analytical Reports with LLMs

The section examines reference-based and reference-free approaches to evaluating analytical reports, using LLMs to assess fine-grained report characteristics and rank reports against human-authored references. LLMs align reasonably well with experts on fine-grained criteria, but ranking generated against human reports exposes important reliability limits.

4.1 Challenges of Reference-based Evaluation

  • Reference-based metrics face scarce, restricted corporate reports, mismatches in content and style, and older 2012–2016 calls that may not reflect later reporting practices.The collected 26 reports cannot be redistributed publicly, and novel insights may not be captured by reference comparison.Source: §4.1 · p. 6

4.2 Reference-free Evaluation with LLMs

  • The study tests GPT-4, Gemini-pro, and Mistral-medium as financial-expert evaluators in characteristic-based and preference-based settings.The characteristic-based setting uses expert-defined report criteria; the preference setting asks evaluators to compare generated and reference reports.Source: §4.2 · pp. 6–7
  • GPT-4 and Mistral correlate well with human experts; every model exceeds 0.5 on at least one characteristic, while GPT-4 exceeds 0.3 across all characteristics.The reported correlations are for fine-grained report characteristics.Source: §4.2 · p. 7, Table 3 · p. 7
  • Human evaluators preferred JP Morgan reference reports over generated reports 83.33% of the time on average.Interviews attributed this preference to references’ detailed, forward-looking evaluations, risk assessments, and financial forecasts.Source: §4.2 · p. 7
  • The earlier earnings calls used for preference evaluation had no available audio, so the Psychologist agent was omitted from report generation for those cases.Source: §4.2 · p. 7
  • GPT-4 and Gemini-pro strongly favored generated reports regardless of candidate order, while LLM rankings were inconsistent across runs.Only Mistral showed the strong first-position bias noted in prior work.Source: §4.2 · p. 8

5 Related Work

Prior multi-agent work motivates role-based collaboration, while this paper adapts that approach to a specialized financial-reporting task. Related earnings-call research has focused on journalistic reports and prediction rather than analytical-report generation.

5.1 LLMs as Task-solving Agents

  • Prior multi-agent research spans collective reasoning and complex task-solving; this work aligns more closely with complex tasks and develops domain-expert agents that use external data.Autogen is identified as a flexible framework for customizable agent-based approaches.Source: §5.1 · p. 8

5.2 Earnings Call Processing

  • ECTSumm benchmarks journalistic earnings-call reports, whereas this study addresses the more challenging generation of analytical reports using a multi-agent framework.Source: §5.2 · p. 8
  • Prior earnings-call NLP emphasizes prediction tasks; Qin and Yang (2019) also influenced the Psychologist agent through findings on audio features for financial-risk prediction.Related prediction tasks include volatility, analyst decisions, financial risk, and earnings surprises.Source: §5.2 · p. 8

6 Conclusion

The conclusion frames analytical-report generation for earnings calls as a new task and reports findings on generated reports and LLM-based evaluation. It also identifies further work on generating novel insights and incorporating real-time financial information.

6 Conclusion

  • The study introduces analytical-report generation for earnings calls and investigates it with an LLM-based multi-agent framework.Source: §6 · p. 8
  • Generated reports diverge from human-authored reports but can include useful agent-introduced insights; LLMs tend to favor generated reports, yet largely align with experts on fine-grained evaluation criteria.Source: §6 · p. 8
  • The authors identify further opportunities to generate novel insights and incorporate real-time financial data, news, and market trends.Source: §6 · p. 8

Limitations

The work is an initial exploration that prioritizes report characterization and LLM-based evaluation over exploring the full design space. A further boundary is the lack of suitable reference reports for open research.

Limitations

  • As a first exploration, the study concentrates on report characterization and LLM-based evaluation rather than investigating all alternative agent designs and interactions.Source: Limitations · p. 9
  • Suitable reference reports are scarce in open research because analytical reports typically originate within corporations.Source: Limitations · p. 9
  • The authors explore reference-free LLM evaluation in response to the lack of suitable reports and suggest future use of real-time financial data, news, and market trends.Source: Limitations · p. 9

A Journalistic vs. Analytical Reports

Analytical reports are more structurally complex and cover a broader range of topics than journalistic reports, with more forward-looking analysis. The differences reflect their distinct audiences and purposes.

Style

  • Analytical reports score higher on two of three readability metrics, indicating longer, more intricate sentences than journalistic reports.Their greater abstractiveness also indicates content less directly derived from the source transcript.Source: A Journalistic vs. Analytical Reports · pp. 10–11

Content

  • Figure 5 shows the earnings topic in 42% of analytical reports versus 83% of journalistic reports, while longer analytical reports cover a much broader range of topics than shorter journalistic reports.Management, outlook, guidance, and increasing represent in-depth discussion of management attitudes, financial-performance trends, and future performance.Source: A Journalistic vs. Analytical Reports · p. 11

Implications

  • Analysts lean toward forward-looking analysis, while journalists focus on key points; the broader topic range and complexity challenge a single model using only the transcript.The authors argue that specialist agents and external data suit this task’s complex analysis needs.Source: A Journalistic vs. Analytical Reports · p. 11

B Prompts

The system uses initialization prompts to establish agent roles, while specialized agents receive external data to shape their feedback. LLM evaluators receive the same instructions as human evaluators.

Agent prompts

  • Initialization prompts dictate agent roles; for the Writer, Editor, and Client, they are the only behavior-controlling attribute, with interaction handled by AutoGen.The supplied excerpt points to Table 6 for the initialization prompts.Source: B Prompts · p. 11

Specialized agents

  • The Analyst and Psychologist collect relevant external data before responding and use it in prompts designed to elicit targeted feedback.The excerpt points to Tables 8 and 9 for data examples and prompt formats.Source: B Prompts · p. 11

Evaluation prompts

  • LLM evaluators receive the same instructions as human evaluators for characteristic-based and preference-based evaluations.Their specific prompts are listed in Table 7.Source: B Prompts · p. 11

C Additional Experimental Details and Results

The supplementary details describe metric calculation, restricted access to reference reports, and the human evaluation setup. Annotator agreement is weak overall, but stronger when two subjective labels are combined.

Evaluation details

  • Readability metrics were computed using the textstat package, and Table 2 provides the characteristic descriptions for human evaluation.The characteristics cover five report topics, with labels for presence and insightfulness.Source: C Additional Experimental Details and… · p. 11

Reference reports

  • The J.P. Morgan reference reports are not publicly available and were accessed under restricted permission, limiting the likelihood of LLM training-data contamination.The paper notes that the reports are unlikely to have appeared in model training.Source: C Additional Experimental Details and… · p. 11

Annotator agreement

  • Combining “Reported and insightful” with “Reported and reasonable” raises pairwise Cohen’s κ to 0.476; results retain all evaluator votes rather than aggregating them.The authors attribute many disagreements to the subjective usefulness distinction and therefore present statistics based on all votes.Source: C Additional Experimental Details and… · pp. 11–12

Correlation results

  • Table 12 breaks down the correlation statistics reported in Table 3, showing correlations between LLMs and individual annotators to provide further insight into agreement with expert evaluators.Source: C Additional Experimental Details and… · p. 12

D Examples

The supplementary material provides examples of the multi-agent feedback process and generated reports, alongside tables describing reference calls, evaluation labels, prompts, and external data.

Generated reports

Supplementary tables

  • The supplementary tables cover the evaluator prompt, external-data examples and prompt formats, reference earnings-call meetings, annotation labels, and LLM–human correlations.Table 7 concerns evaluation prompts; Tables 8 and 9 cover specialized-agent data and formats; Tables 10–12 cover calls, labels, and correlations.Source: Table 7 · p. 13, Table 8 · p. 14, Table 9 · p. 14

Feedback example

  • Figure 6 depicts a typical run in which the Client gives the task, feedback agents respond in role, and the Client ends composition when the report is acceptable.The example illustrates an iterative feedback process.Source: D Examples · p. 12, Figure 6 · p. 15

Related papers

Loading 2410.01039v1…