PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting

Dong Shu, Yanguang Liu, Huopu Zhang, Saisai Hu, Haiyan Zhao, Hekun Huang, Mengnan Du

arXiv:2609.38523v1cs.LGcs.AI

TL;DR

Earnings-call datasets have often omitted acoustic and visual signals, despite their presence in real calls. MM-FinEval evaluates models on 2,045 calls using transcripts, slides, and audio across 12 financial tasks. The authors report a performance advantage from integrating all three modalities, with audio and visual signals described as non-redundant for replicating expert financial reasoning.

  • Problem

    Existing earnings-call datasets rely on unimodal architectures and omit acoustic and visual signals present in actual conference calls.

    Source: §2.1 · p. 2

  • Method

    MM-FinEval benchmarks 2,045 earnings calls from 2019–2022, pairing transcripts, slides, and audio with 12 financial tasks and evaluating 19 models.

    Source: §5.1 · p. 4, §7 · p. 10

  • Results

    Integrating all three modalities provides a performance advantage, with visual and audio information described as non-redundant for replicating expert financial reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Financial analysis involves synthesizing what executives disclose and how they present it, as reflected in MM-FinEval’s multimodal framing.

    Source: §7 · p. 10

    The authors conclude that forecasting long-term variance is inherently unfeasible for current LLM architectures.

    Source: §6.5 · p. 9

Abstract

from arXiv · show

Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals. However, existing financial benchmarks are often limited to unimodal inputs or single-task settings, making it difficult to evaluate whether multimodal large language models (LLMs) can support real-world financial analysis. In this paper, we introduce MM-FinEval, a novel benchmark designed to evaluate multimodal LLMs across multiple financial tasks. MM-FinEval spans a diverse timeline from 2019 to 2022. The entire proposed dataset contains 2,045 S\&P 500 conference earning calls as inputs and 12 financial task labels as outputs. Each input contains three modalities: a word-to-word text transcript of the earning call, the corresponding presentation slides used during the call, and the entire audio recording. To establish a rigorous evaluation framework, we analyze 19 baseline models across three distinct model categories: Image-Text, Audio-Text, and Any-to-Any configurations. We observe that small-size Any-to-Any models processing all three modalities achieve strong performance, even when compared against larger proprietary models restricted to two-modality inputs. This indicates that our tri-modal dataset design introduces useful, non-redundant information. These results validate that text, audio, and visual data serve as important, complementary signals that mimic the decision-making process of expert human analysts.

1 Introduction

MM-FinEval addresses the limits of unimodal, task-specific financial benchmarks with earnings-call data combining transcripts, audio, and slides across 12 financial tasks. Its experiments highlight the value of tri-modal inputs and differences in forecasting performance across tasks and horizons.

1 Introduction

  • The introduction identifies unimodal models and task-specific datasets as limits on learning contextual information for complex real-world financial analysis.It notes that even multimodal datasets beyond text are typically restricted to a specific task.Source: §1 · p. 1
  • MM-FinEval contains 2,045 S&P 500 earnings calls from 2019–2022, each with a transcript, raw audio, slides, and 12 financial-task labels.The annual counts are 450 in 2019, 273 in 2020, 499 in 2021, and 823 in 2022.Source: §1 · pp. 1–2
  • Current LLMs exhibit a severe performance gap between market-return and volatility forecasting.Source: §1 · p. 2
  • Integrating text, audio, and slides provides a performance advantage, with visual and audio information contributing non-redundant signals for financial reasoning.The introduction presents tri-modal integration as important for replicating expert financial reasoning.Source: §1 · p. 2
  • As the forecasting horizon expands from 3 to 30 days, earnings signals are rapidly diluted by unpredictable market factors, consistent with information half-life theory.Source: §1 · p. 2

2 Related Work

Prior earnings-call datasets address specific analysis tasks, but existing resources often omit acoustic and visual signals or remain limited to a single financial task, motivating multi-label multimodal benchmarks.

2.1 Earnings Conference Call Datasets

  • Call datasets support tasks including transcript summarization with ECTSum (Mukherjee et al., 2022), logical deduction with FinNLI (Magomere et al., 2025), and Q&A subjectivity with SubjECTive-QA.Text datasets STRUX (Lu et al., 2024) and DEC (Yu et al., 2025) were created for LLM-driven investment decision-making.Source: §2.1 · p. 2
  • Existing call datasets often rely on unimodal architectures that omit acoustic and visual signals, while multimodal datasets remain task-specific, motivating multi-label multimodal benchmarks.FinCall-Surprise combines call transcripts, audio, and slides but is optimized exclusively for earnings-surprise prediction.Source: §2.1 · pp. 2–3
  • EC combined transcripts with raw audio to predict financial risk, and MAEC scaled call data to predict stock volatility.Source: §2.1 · p. 2
  • Earnings-21 (Del Rio et al., 2021) and ConEC focus on acoustic challenges, primarily benchmarking automatic speech recognition rather than broader financial reasoning.Source: §2.1 · p. 2

3 MM-FinEval Construction

MM-FinEval is constructed through screening, multimodal collection and synchronization, and annotation of financial outcomes. Its 12 labels cover market-adjusted stock reactions, stock volatility, and corporate bond returns around quarterly earnings calls.

3 MM-FinEval Construction

  • The construction pipeline first screens U.S. public firms for market capitalization above $1 billion and average daily trading volume above $50 million, yielding over 4,000 companies.Source: §3 · p. 3
  • Transcripts, audio, and slides are collected from official company websites and manually aligned by reporting dates and cross-checked by call title to ensure they match the same event.Source: §3 · p. 3

3.1 Multi-Label Preliminaries

  • The benchmark’s 12 labels predict cumulative abnormal return, stock volatility, and corporate bond returns from quarterly earnings calls.Source: §3.1 · p. 3
  • CAR measures market-adjusted price reactions, with post-call windows used to assess short- and medium-term directional market responses.Source: §3.1 · p. 3
  • Volatility captures price instability regardless of return direction; labels measure pre- and post-call volatility over 3-, 7-, 15-, and 30-day windows.Source: §3.1 · p. 3
  • Corporate bond labels use monthly returns based on dirty prices adjusted for coupon payments, with RET_EOM, RET_LDM, and RET_L5M differing by month-end price definition.Source: §3.1 · pp. 3–4

4 Data Analysis

MM-FinEval comprises 2,045 multimodal earnings calls spanning 2019–2022, with calls structured around Operators, Executives, and Analysts.

  • 2,045 calls from 2019–2022 average approximately 9,000 transcript words, 27 slide pages, and 3,700 seconds of audio.The dataset includes a text transcript, presentation slides, and an audio recording for each call.Source: §4 · p. 4
  • Calls feature three primary roles: Operators manage the event, Executives present financial results and outlook, and Analysts ask about company performance.Source: §4 · p. 4

5 Benchmark Settings

The benchmark compares 19 models across modality groups using tailored multimodal inputs, regression metrics, and standardized implementation settings.

5.1 Baseline LLMs

  • 19 open- and closed-source baselines span image-text LVLMs, text-audio models, and tri-modal Any-to-Any models, including Gemma 4 E2B and E4B.The model groups are defined by their input capabilities.Source: §5.1 · p. 4

5.2 Benchmark Input Design

  • Image-text inputs used raw transcripts and five randomly sampled substantive slide images; audio-text added audio, while tri-modal inputs combined those same images and audio with the transcript.Presentations with five or fewer substantive slides used all available slides; full PDFs are included in the dataset release.Source: §5.2 · pp. 4–5

5.3 Benchmark Metrics

  • Continuous stock-return or volatility labels make prediction a regression task, evaluated by MAE, Pearson correlation for cross-company reaction magnitudes, and directional accuracy for sign correctness.Table 2 reports averaged MAE and DA across 12 financial tasks spanning 2019–2022.Source: §5.3 · p. 5, Table 2 · p. 6

5.4 Implementation Details

  • Experiments used two 80GB NVIDIA A100 GPUs, official repository configurations, and bfloat16 precision, with no other modifications to default model parameters.These settings were used to support computational efficiency and a fair, reproducible comparison.Source: §5.4 · p. 6

6 Benchmark Results

MM-FinEval results show a gap between volatility and return prediction, while small Any-to-Any models outperform larger baselines on several return measures. Across tasks, results also reveal inconsistent scaling, conservative-versus-aggressive Audio-Text strategies, and temporal degradation in predictive performance.

6.1 The Gap Between Returns and Volatility

  • Return-task DA is usually 40%–60%, whereas most models exceed 90% on volatility tasks, despite small models sometimes failing the non-negativity constraint.For car, Claude and Gemma-4 achieve MAEs of 0.041–0.045, about 77% of the ground-truth standard deviation; for fut_15d, top-model MAEs are 0.0085–0.0097 against a ground-truth mean of 0.019874 and standard deviation of 0.011657.Source: §6.1 · p. 6

6.2 Inconsistent Scaling Laws and the Proprietary Advantage

  • Qwen 3.5 and 3.6 models show rising DA as parameter counts grow, but MAE does not improve consistently with scale.For car, Qwen 3.5 27B has MAE 0.0534 versus 0.0498 for 4B and 0.0499 for 9B; proprietary GPT-5.5 and Claude-4.8 reach about 72% DA on ret_eom and 67% on ret_l5m and ret_ldm, with 100% DA on volatility tasks.Source: §6.2 · p. 7

6.3 The Advantage of Any-to-Any Multimodal Integration

  • Gemma 4 E2B and E4B exceed 60% DA across ret_eom, ret_l5m, and ret_ldm, outperforming larger Qwen 3.6 and GPT 5.4 baselines.On car, their MAEs are 0.0418 and 0.0437, respectively, versus 0.0458 for Claude-4.8; the paper attributes their performance to complementary text, visual, and audio signals.Source: §6.3 · pp. 7–8

6.4 Conservative Versus Aggressive Prediction in Audio-Text Models

  • Voxtral achieves lower MAE across Audio-Text tasks, while Moss performs better on DA.Voxtral’s near-zero predictions reduce absolute-error penalties but hurt directional accuracy; Moss’s larger predictions are more likely to capture the correct sign.Source: §6.4 · p. 8

6.5 Pearson Correlation

  • Proprietary models show stronger Pearson correlations on volatility tasks than open-source models, but both groups have near-zero or negative correlations on car.GPT-5.4 reaches 0.44 on vol_fut_7d, Claude 4.8 reaches 0.42, and GPT-5.5 reaches 0.60 on ret_eom; correlations on volatility tasks decline as prediction horizons lengthen, with Claude 4.8 an exception across post-event horizons in 2021.Source: Figure 3 · p. 7, §6.5 · pp. 8–9, Figure 4 · p. 8

6.6 Temporal Degradation on Direction Accuracy

  • Return-task DA declines from 70%–90% among several top models in 2019 to 30%–40% in 2022, indicating weak temporal generalization.The paper suggests post-pandemic market dynamics increased complexity and reduced predictive capability.Source: Figure 5 · p. 9, §6.6 · p. 10

7 Conclusion

MM-FinEval evaluates multimodal LLMs on 12 financial tasks using 2,045 S&P 500 earnings calls from 2019–2022. Its three modalities reflect how expert analysts synthesize both what executives disclose and how they present it.

  • MM-FinEval covers 12 financial tasks with 2,045 S&P 500 earnings conference calls spanning 2019–2022.Source: §7 · p. 10
  • Its text, slides, and audio reflect how expert financial analysts synthesize both executive disclosures and their presentation.Source: §7 · p. 10
  • Baseline evaluations across Image-Text, Audio-Text, and Any-to-Any configurations expose where language modeling must transition to multisensory, calibrated quantitative reasoning.The authors hope the benchmark supports future multimodal financial analysis for economic forecasting.Source: §7 · p. 10

A Data Analysis

Return labels are near-zero, high-variance targets, whereas volatility labels are positive and more tightly clustered. The paper links these distributional differences to volatility tasks being easier for models to predict quantitatively.

A Data Analysis

  • Return-task means range from 0.001620 to 0.003443, while standard deviations range from 0.024981 to 0.054573, indicating low signal-to-noise ratios.The four return tasks are car, ret_eom, ret_l5m, and ret_ldm; car ranges from -0.261699 to 0.312476.Source: A Data Analysis · p. 14
  • Across eight volatility labels, minima are strictly positive, means range from 0.017588 to 0.020292, and standard deviations range from 0.009554 to 0.015334.The lowest absolute minimum is 0.000288 for past_3d; the paper describes volatility as nonnegative, tightly clustered, and characterized by volatility clustering.Source: A Data Analysis · p. 14
  • The contrasting distributions explain benchmark performance divergence: noisy, symmetric returns resist quantitative regression, while structured, bounded, clustered volatility is easier to map.The paper characterizes volatility as a more autoregressive and stable phenomenon, consistent with established financial econometrics.Source: A Data Analysis · p. 14

B Prompt Used in Our Experiments

The experiment prompts ask models to estimate equity or bond returns and volatility from earnings-call materials, including transcripts paired with slides, audio, or both. They specify prediction targets and time windows and require a brief rationale alongside each decimal prediction.

B.1 CAR Task

  • CAR prompts predict cumulative abnormal return from transcript-and-slide, transcript-and-audio, or tri-modal inputs, with a brief driver rationale and decimal output.CAR is defined as the sum of differences between actual and expected stock returns over the event window.Source: §B.1 · p. 14

B.2 ret_eom; B.3 ret_l5m; B.4 ret_ldm

  • The bond-return prompts cover end-of-month, final-five-trading-day, and final-trading-day returns, using transcript-and-slide, transcript-and-audio, or tri-modal inputs.Each asks for a brief key-driver rationale and a decimal prediction; the end-of-month target is a one-month percentage gain or loss.Source: §B.2 · p. 15, §B.3 · p. 15, §B.4 · p. 16

B.5 vol_fut_3d; B.6 vol_fut_7d

  • Prompts target post-event volatility as vol_fut_3d over the next three trading days and vol_fut_7d, using earnings-call transcripts with presentation slides; separate transcript-plus-MP3 variants are specified for both targets.For vol_fut_3d, the output requires a 1–2 sentence key-driver rationale and a decimal estimate.Source: §B.5 · p. 16, §B.6 · p. 17

B.7 vol_fut_15d; B.8 vol_fut_30d

  • Prompts specify predicting post-event 15-day (vol_fut_15d) and 30-day (vol_fut_30d) volatility using earnings-call transcripts with presentation slides, audio, or both audio and slides.Source: §B.7 · p. 17, §B.8 · p. 18

B.9 vol_past_3d

  • The pre-event volatility prompt estimates realized volatility over the 3 trading days immediately before the call from transcript-and-slide or transcript-and-audio inputs.It requests a brief key-driver rationale and a decimal estimate.Source: §B.9 · p. 18

B.9–B.10 output format

  • For vol_past_3d and vol_past_7d, estimate realized volatility over the 3 or 7 trading days immediately preceding the earnings call, explain the key driver in 1–2 sentences, and express each estimate as a decimal.Source: §B.9 · p. 18, §B.10 · p. 19

B.9 vol_past_3d

  • The vol_past_3d prompt estimates realized volatility for the three trading days before the call using the transcript, audio, and presentation slides.It requests a 1–2 sentence rationale and a decimal prediction.Source: §B.9 · p. 18

B.10 vol_past_7d

  • The vol_past_7d prompts estimate realized volatility for the seven trading days before the call, using transcript-and-slides, transcript-and-audio, or all three modalities.The prompt variants retain the same 1–2 sentence rationale and decimal prediction format.Source: §B.10 · p. 19

B.11 vol_past_15d

  • The vol_past_15d prompts estimate realized volatility for the 15 trading days before the call, using transcript-and-slides, transcript-and-audio, or all three modalities.The prompt variants request a brief rationale and a decimal prediction.Source: §B.11 · p. 19

B.11–B.12 output format

  • For vol_past_15d and vol_past_30d, estimate realized asset volatility over the respective 15 or 30 trading days immediately preceding the earnings call, give the estimate as a decimal, and explain the key driver in 1-2 sentences.Source: §B.11 · p. 19, §B.12 · p. 20

B.12 vol_past_30d

  • The vol_past_30d prompts estimate realized volatility for the 30 trading days before the call, using transcript-and-slides, transcript-and-audio, or all three modalities.The prompt variants request a brief rationale and a decimal prediction.Source: §B.12 · p. 20

C Pearson Correlation Temporal Observations

Across 2019–2022, proprietary models increasingly captured linear relationships on ret_eom, while open-source models remained near zero. Figure 6 illustrates this contrast.

C Pearson Correlation Temporal Observations

  • On ret_eom, GPT-5.5’s Pearson correlation rose from 0.309 in 2019 to 0.581 in 2022, while Claude 4.8’s rose from 0.144 to 0.437.Both models show an upward trajectory across the same task and years.Source: C Pearson Correlation Temporal… · p. 20
  • Qwen 3.5 and Mistral 3 remained near-zero Pearson correlations on ret_eom across the four-year span.The passage contrasts their lack of temporal adaptation with the improving proprietary models.Source: C Pearson Correlation Temporal… · p. 20

D Examples of Model Responses

The examples show model rationales linking company performance, expectations, and guidance to predicted market reactions, with both positive and negative CAR predictions.

GPT 5.4 … Moss

The model examples predict both positive and negative market reactions, citing reported performance alongside forward guidance, costs, and other company developments. Their rationales range from specific financial catalysts to broad outlook signals.

GPT 5.4

  • A 0.012 prediction for Kimberly-Clark reflects strong 2020 execution tempered by cautious 2021 growth guidance and commodity-driven margin pressure.The rationale identifies strong EPS, margin expansion, cash flow, market-share gains, and cost savings, but expects the outlook to limit the positive reaction.Source: GPT 5.4 · p. 23

GPT 5.5

  • A -0.065 prediction links investor concerns about a Q4 EPS miss, compensation and wage inflation, weaker trading, and investment spending to negative profitability revisions.The rationale notes record full-year results but says capital headwinds and limited near-term buybacks could reinforce concerns.Source: GPT 5.5 · p. 23

Claude 4.6

  • A 0.038 prediction treats PMI’s Q2 adjusted diluted EPS of $1.57 (+17.8% organic), raised full-year guidance, buyback program, and imminent IQOS ILUMA launch as positive catalysts.The rationale gives the raised EPS guidance as $5.97-$6.07, implying +15-17% in dollar terms, and identifies a $5-7B share buyback program.Source: Claude 4.6 · p. 23

Claude 4.8

  • A 0.045 prediction for WBA reflects a Q1 adjusted EPS beat at $1.68, up 53%, and raised full-year guidance, despite shrink, labor, and script headwinds.The rationale attributes the beat to COVID vaccinations and testing and robust retail comps.Source: Claude 4.8 · p. 23

Voxtral 24B

  • A 0.035 prediction links a 32% increase in comparable basis EPS excluding canopy equity earnings and a $2 billion repurchase program to a positive CAR.The rationale also cites a commitment to return $5 billion in value to shareholders through fiscal 23.Source: Voxtral 24B · p. 23
Loading 2609.38523v1…