Source-linked AI summary
A Shaky Voice Is Not Always a Dodge: Benchmarking Textual and Vocal Evasion Detection in Earnings Calls
Mirae Kim, Seonghun Jeong, Youngjun Kwak
TL;DR
Existing evasion detection treats earnings-call communication as textual, although vocal delivery provides an independent dimension. The paper introduces DualEvasion, a dual-label text-and-audio benchmark, and finds that models perform far worse on vocal confidence than textual evasion, with speaker calibration offering only modest improvement. The benchmark’s vocal-confidence operationalization captures one aspect of vocal cues, and its exploratory stock-volatility analysis uses a small sample.
Problem
Existing earnings-call evasion work focuses on textual transcripts, overlooking independent vocal information such as confidence and uncertainty.
Method
The paper introduces DualEvasion, containing 505 annotated question-answer pairs from 60 calls with independent textual-evasion and vocal-confidence labels.
Results
Models approach human-level textual-evasion performance but fall far short on vocal-confidence detection; speaker-level calibration yields modest improvements without closing the human-performance gap.
Takeaways & Limitations
Speaker-aware audio understanding remains an open challenge for current models, because they rely on absolute vocal features rather than speaker-relative deviations.
Takeaways & Limitations
The vocal dimension captures only one aspect of vocal cues, and the stock-volatility analysis is exploratory because it uses a small sample.
Abstract
from arXiv · showhide
Existing approaches to evasion detection in earnings calls focus on textual transcripts, treating evasion as a single-dimensional phenomenon. We argue that evasion in spoken communication is inherently multidimensional: beyond what executives say, how they say it carries independent and complementary information. To study these dimensions jointly, we introduce DualEvasion, a benchmark for evasion detection across text and audio in earnings call Q&A. The benchmark contains 505 annotated question-answer pairs from 60 earnings calls, each with two independent labels: textual evasion (direct vs. evasive) and vocal cues operationalized as speaker confidence (confident vs. unconfident). Our experiments show that state-of-the-art multimodal models struggle to detect vocal confidence, particularly on unconfident responses. Our analysis suggests these models interpret acoustic cues in isolation rather than relative to each speaker's baseline. Providing speaker-level references yields modest improvements, but a substantial gap with human performance remains.
1 Introduction
Existing evasion research focuses on textual transcripts, but earnings-call responses also convey independent vocal information. DualEvasion benchmarks both dimensions and highlights speaker calibration as a central challenge.
- Existing earnings-call evasion studies focus on what executives say, overlooking how they say it.
- 32.1% of instances show cross-modal disagreement between textual directness and vocal confidence.Textually complete answers can sound hesitant, while skillful deflections can sound composed.
- Vocal confidence provides an annotatable entry point into vocal evasion cues but does not exhaust that dimension.It is grounded in observable prosodic cues that listeners can reliably perceive.
- DualEvasion contains 505 annotated question-answer pairs from 60 earnings calls with independent textual and vocal labels.The vocal label operationalizes confidence as confident versus unconfident.
- Frontier models approach human-level textual-evasion performance but struggle with vocal-confidence detection.Speaker-level calibration yields only modest improvements and leaves a substantial human-performance gap.
2 Related Work
Prior work develops textual evasion taxonomies and classification tasks, while vocal signals in financial communication remain largely unexplored. Research on vocal confidence supplies evidence that prosodic cues can support uncertainty judgments.
- Prior evasion research spans directness taxonomies and finer-grained strategies such as omission, vagueness, and agenda shifting.
- Earnings-call evasion datasets and related-domain studies rely exclusively on textual features.
- Vocal signals in financial communication remain largely unexplored despite established research on vocal confidence.
- Filled pauses, response delays, and prosodic cues are associated with or support listeners’ perception and classification of speaker uncertainty.
3 DualEvasion
DualEvasion is a dual-label earnings-call Q&A benchmark constructed from recordings and transcripts, with independent textual-evasion and vocal-confidence annotations. Its tables document zero-shot evaluation, label disagreement, and annotation agreement.
- Data Collection: DualEvasion is built from earnings-call recordings and transcripts aligned into speaker-attributed question-answer pairs.The collection pipeline uses WhisperX timestamps and transcript speaker metadata.
- Evaluation: Zero-shot evaluation tests language models on transcript text and audio-capable models on response audio.Table 2 reports textual-evasion and vocal-confidence performance, with commercial audio results averaged over three runs.
- Data Collection: The dataset is filtered for single-executive responses, sufficient per-speaker reference material, and responses under 300 seconds.The filtering sequence selects 60 calls across 49 unique tickers.
- Annotation and Validation: The final benchmark contains 505 question-answer pairs, each annotated independently for textual evasion and vocal confidence.Two financial domain experts annotate all pairs, followed by a separate reliability check on a subset.
- Annotation and Validation: Textual labels indicate whether responses substantively address questions, while vocal labels use prosodic cues and speaker-specific baselines.Vocal annotation is performed separately from transcript content and textual labels.
- Annotation and Validation: Validation of vocal-confidence labels on 52 responses yields Fleiss’ κ = 0.713, indicating substantial agreement.
4 Experiment
The experiments evaluate textual evasion and vocal confidence detection, then examine whether speaker-relative acoustic analysis and explicit speaker calibration improve vocal-confidence predictions. Models perform well on textual evasion but struggle particularly with unconfident responses, while calibration provides only modest gains.
- 4.1 A Modality Gap in Model Performance: GPT-5 and Gemini-3-Flash reach 87.4 and 84.7 macro F1 on textual evasion, while the best audio model achieves only 58.5 macro F1 for vocal confidence.GPT-5’s textual agreement approaches human–human levels, whereas Gemini-3-Flash’s vocal-confidence agreement remains substantially lower.
- 4.1 A Modality Gap in Model Performance: Unconfident-response F1 ranges from 30.8 to 39.2, far below the confident class at up to ∼79 F1.The strongest models exceed the 44.0 macro-F1 all-confident majority baseline, while weaker models overpredict unconfidence with recall as high as 95.4.
- 4.2 Speaker Calibration: Speaker-centering reduces Gemini-2.5-Flash’s mean absolute correlation with unconfident predictions from .195 to .032 (p < .01).The analysis compares absolute acoustic values with speaker-relative deviations, measured as each response minus the speaker’s mean, using cluster bootstrap significance over 60 calls.
- 4.2 Speaker Calibration: The strongest models show reduced acoustic correlations after speaker-centering, indicating reliance on raw vocal values rather than each speaker’s baseline.Human annotator correlations remain stable or increase slightly, while weaker models have raw correlations already near zero.
- 4.2 Speaker Calibration: Speaker-level calibration with three same-speaker reference utterances modestly improves unconfident F1, including a +4.7 gain for Gemini-2.5-Flash.Providing same-speaker examples alone yields no consistent improvement; the calibrated prompt first infers habitual pitch, tempo, and prosody before judging deviation.
- 4.3 Market-Relevant Signal: Across coverage settings, audio-based unconfidence generally explains more post-earnings stock-volatility variance than textual evasion.The analysis regresses call-level evasive and unconfident ratios over 3- to 30-day windows while progressively retaining calls with more complete Q&A coverage.
5 Conclusion
The paper concludes that DualEvasion benchmarks evasion across independent textual and vocal labels, exposing a modality gap in current models. Models approach human-level textual-evasion performance but remain far weaker on vocal confidence, and speaker calibration does not close that gap.
- DualEvasion contains 505 annotated question-answer pairs with independent textual-evasion and vocal-confidence labels for evasion detection across text and audio.
- Models approach human-level performance on textual evasion but fall far short on vocal-confidence detection.
- Models rely on absolute vocal features rather than speaker-relative deviations, while speaker-level calibration yields only modest improvements.This reliance can conflate habitual speaker traits with uncertainty cues.
- Speaker-aware audio understanding remains an open challenge for current models.
6 Limitations
The benchmark is constrained by its limited scale, the partial scope of its vocal-confidence operationalization, and the exploratory nature of its stock-volatility analysis.
- The benchmark contains 505 question-answer pairs from 60 calls, with dual annotation and per-speaker vocal calibration making annotation time-intensive.This annotation burden limits the scale achieved.
- Vocal confidence captures one aspect of vocal cues but does not exhaust them.
- The stock-volatility analysis is based on a small sample and should be interpreted as exploratory.
A Dataset Statistics
The dataset spans diverse sectors and market groups, with response lengths varying widely, and uses separate annotation procedures for textual evasion and vocal confidence. Vocal labels incorporate speaker calibration and validation, while the annotation interface supports binary judgments and comments for ambiguity.
- Dataset composition: The benchmark spans 11 sectors and three market groups, with response durations ranging from under 1 second to nearly 5 minutes.
- Annotation procedure: The vocal-confidence interface lets annotators listen to each response, choose one of two labels, and optionally comment on ambiguous cases.
- Validation: Nine optional comments were recorded, all for unconfident judgments, citing large sighs, slower-than-usual speech, or pauses.Textual evasion achieved 94.7% raw agreement in the main annotation, so additional validation focused on vocal confidence.
- Validation: Pairwise Cohen’s κ is reported for vocal-confidence validation among five annotators and for inter-model agreement on a validation subset of N = 52.
C.1 Baseline and Threshold Analysis
Audio models outperform the all-confident majority baseline in some cases, but vocal confidence detection remains weak, especially for unconfident responses. Threshold tuning provides only marginal gains.
- 44.0 macro F1 is achieved by the all-confident baseline, with confident F1 88.0, unconfident F1 0.0, and accuracy 78.6.
- The strongest audio models exceed the majority baseline, while weaker models overpredict the unconfident class, showing that low scores are not explained by majority-class behavior alone.
- 36.7 and 35.5 are the best unconfident F1 scores after threshold tuning, only slightly above untuned scores of 36.2 and 30.8.
C.2 Supervised Fine-Tuning
Fine-tuning speech models with five-fold cross-validation improves overall macro F1 for some systems but does not reliably improve detection of the unconfident class.
- Five-fold cross-validation evaluates fine-tuned speech models on held-out folds of the 505-pair dataset.
- +20.6 macro F1 is achieved by Qwen2-Audio after fine-tuning, but its unconfident F1 decreases from 36.2 to 32.4.
- 34.7 unconfident F1 remains low for Audio-Flamingo-3 after a modest +1.8 macro F1 gain.
- 9.9 unconfident F1 is reached by VibeVoice-ASR despite a macro F1 of 48.8, and all fine-tuned models remain at or below 34.7 unconfident F1.
E Few-Shot Speaker Calibration
The benchmark tests speaker calibration by giving models same-speaker reference clips before the target response. References yield no consistent improvement and can reduce unconfident detection.
- Same-speaker reference audio clips are prepended to target responses at n ∈ {1, 2, 3} to establish a speaker-specific baseline.
- No consistent improvement results from reference audio, although Gemini-2.5-Flash and Gemini-3-Flash gain +1.6 and +5.4 macro-F1 at n=1.
- At n=3, unconfident F1 drops from 38.5 to 28.2 for Gemini-2.5-Flash and from 38.2 to 23.8 for GPT-Audio.
- A possible explanation is that audio LLMs struggle to form a stable speaker-level baseline from in-context examples.
F Stock Volatility Analysis
The study relates textual evasion and vocal unconfidence to post-earnings stock volatility using call-level regressions across multiple coverage thresholds. Audio-based unconfidence generally explains more variance, while combining both dimensions yields the highest R2.
- Return volatility computation: Daily returns are computed from closing prices, and volatility is calculated over k-day windows using the return series.
- Trading-day alignment: Trading Day 0 is assigned according to whether the call occurs before market open, after market close, or during market hours.
- Coverage thresholds: Higher Q&A coverage thresholds provide more complete call-level estimates but reduce the number of calls available for regression analysis.