Source-linked AI summary
EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A
Shijian Ma, Yan Lin, Yi Yang
TL;DR
EvasionBench addresses the lack of large-scale benchmarks for detecting whether corporate earnings-call responses answer analyst questions. It builds a filtered earnings-call benchmark with consensus-based annotation and reports Eva-4B at 84.9% Macro-F1, while cross-domain generalization remains unvalidated.
Problem
NLP lacks a large-scale benchmark for detecting discourse-level evasion, despite its relevance to high-stakes corporate earnings-call Q&A.
Method
The paper constructs EvasionBench from earnings-call transcripts and uses Multi-Model Consensus with dual LLM labeling and three-judge arbitration.
Results
84.9% Macro-F1: Eva-4B outperforms Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash on this task.
Takeaways & Limitations
EvasionBench provides a large-scale benchmark, consensus annotation framework, and human-validated evaluation resources for managerial evasion detection.
Takeaways & Limitations
The dataset and models are trained exclusively on earnings-call transcripts, so generalization to political interviews and legal depositions requires further validation.
Abstract
from arXiv · showhide
We present EvasionBench, a comprehensive benchmark for detecting evasive responses in corporate earnings call question-and-answer sessions. Drawing from 22.7 million Q&A pairs extracted from S&P Capital IQ transcripts, we construct a rigorously filtered dataset and introduce a three-level evasion taxonomy: direct, intermediate, and fully evasive. Our annotation pipeline employs a Multi-Model Consensus (MMC) framework, combining dual frontier LLM annotation with a three-judge majority voting mechanism for ambiguous cases, achieving a Cohen's Kappa of 0.835 on human inter-annotator agreement. We release: (1) a balanced 84K training set, (2) a 1K gold-standard evaluation set with expert human labels, and (3) [Eva-4B], a 4-billion parameter classifier fine-tuned from Qwen3-4B that achieves 84.9% Macro-F1, outperforming Claude 4.5, GPT-5.2, and Gemini 3 Flash. Our ablation studies demonstrate the effectiveness of multi-model consensus labeling over single-model annotation. EvasionBench fills a critical gap in financial NLP by providing the first large-scale benchmark specifically targeting managerial communication evasion.
1 Introduction
Evasion is widespread and consequential, but NLP lacks a large-scale benchmark for detecting whether responses actually answer questions. EvasionBench addresses this gap with a domain-grounded dataset, three-level taxonomy, consensus annotation framework, and validated evaluation resources.
- Evasive communication spans political interviews, legal depositions, and corporate earnings calls, where it predicts subsequent earnings misses and stock underperformance.
- NLP lacks a canonical large-scale evasion benchmark because evasiveness is subjective and expert annotation is expensive.
- Earnings calls provide high-stakes, adversarial Q&A, abundant transcripts, and question types that support operationalizing response directness.
- EvasionBench derives 84K balanced training samples and 1K human-validated evaluation samples from 22.7M raw Q&A pairs.
- MMC combines dual frontier-LLM labeling with three-judge arbitration and achieves Cohen’s Kappa of 0.835 against human annotators.
- Eva-4B achieves 84.9% Macro-F1 and outperforms Claude Opus 4.5, GPT-5.2, and Gemini 3 Flash on evasion detection.
3 Task Definition and Taxonomy
EvasionBench formulates evasion detection as three-level classification of analyst questions and management responses. Its ordinal taxonomy distinguishes complete answers, partial sidestepping, and total non-responsiveness across several question types.
- The task classifies each management response to an analyst question as direct, intermediate, or fully evasive.
- The three-level scale preserves the distinction between partial answers and complete deflection, which binary labels lose.
- Direct responses explicitly and completely address the question core using specific figures, clear stances, or direct explanations.
- Intermediate responses provide related context while sidestepping the specific ask through hedging, conditional framing, or adjacent topics.
- Fully evasive responses ignore, refuse, or entirely depart from the question, including information flooding or silent pivots.
- Question cores comprise quantitative, temporal, binary, and causal or directional types.
4 Data Collection and Filtering
The dataset is drawn from S&P Capital IQ earnings-call transcripts and filtered through structural, textual-quality, and substantive-content criteria. This process retains 11.27M high-quality Q&A pairs for benchmark construction.
- The source contains 22.7M Q&A pairs from 1.38M earnings-call transcripts covering 420K unique speakers.
- Pair extraction retains sequential analyst questions and management answers while removing operator instructions and pleasantries.
- Quality filtering requires a question mark, answers longer than 30 characters, and no transcription markers.
- Substantial-content selection requires the combined question and answer length to be at least 500 characters.
- After three-stage filtering, 11.27M Q&A pairs, or 49.6% of the source, remain.
5 Multi-Model Consensus Annotation
MMC uses heterogeneous frontier LLMs to label earnings-call responses, routes disagreements to majority-vote arbitration, and mitigates systematic single-model tendencies. The resulting training data spans broad corporate communication patterns.
- Claude Opus 4.5 and Gemini 3 Flash independently annotate samples, with agreement forming the consensus set.
- 3,645 disagreement cases, representing 16.1% of samples, are resolved by three judges through majority voting.
- Opus favors direct labels at 53.3%, Gemini assigns fully evasive labels at 23.5%, and GPT-5.2 favors intermediate labels at 56.7%.
- MMC randomizes prediction order to mitigate position bias in arbitration prompts.
- The training data covers 2002–2022 and 8,081 unique companies, providing broad corporate-communication coverage.
- MMC first distills raw transcripts, then applies dual LLM annotation, three-judge arbitration, and final dataset balancing.
6 Inter-Annotator Agreement
Human validation supports high annotation reliability for the three-level evasion labels, while the training pipeline combines consensus data with judge-refined boundary cases.
- Inter-Annotator Agreement: 0.835 Cohen’s Kappa indicates almost perfect agreement on a balanced Gold 1K subset.Ten of 11 disagreements involved the intermediate class, identifying it as the most ambiguous category.
- Two-Stage Training Pipeline: 60K consensus samples provide Stage 1 training data for Eva-4B (Consensus).Consensus samples are those on which both frontier LLMs agreed.
- Two-Stage Training Pipeline: 24K disagreement cases support Stage 2 training with either three-judge majority labels or Opus-only labels.The judge-refined stage incorporates harder boundary cases resolved through three-judge voting.
- Two-Stage Training Pipeline: Three ablation variants compare consensus-only training, Opus-only refinement, and full majority-vote refinement.Eva-4B (Consensus) uses Stage 1 only; Opus Only and Full add Stage 2 with different labels.
8 Experiments
Experiments evaluate Eva-4B variants alongside closed- and open-source models on the Gold 1K benchmark, with consensus-based refinement producing the strongest ablation result.
- Evaluation Setup: 12 models are evaluated on the Gold 1K evaluation set, including frontier, open-source, Eva-4B, and base-model systems.The comparison includes Claude Opus 4.5, GPT-5.2, Gemini 3 Flash, five open-source models, three Eva-4B variants, and Qwen3-4B.
- Ablation Results: 84.9% Macro-F1 is achieved by full fine-tuning with three-judge consensus, improving 50.6 percentage points over the base model.This is the reported ablation comparison for Eva-4B variants.
- Ablation Results: +4.3 pp Macro-F1 separates Eva-4B (Full) from Eva-4B (Opus Only).The result supports three-judge majority voting over single-model labeling in the ablation.
- Training Dynamics: 0.007 versus 0.56 final training loss distinguishes Eva-4B (Full) from Eva-4B (Opus Only).The paper describes this as an 80× difference and associates the higher loss with noisier single-model annotations.
- Error Patterns: 95.4% of Eva-4B (Full)’s errors are between adjacent classes, while fully evasive reaches 92.4% F1.The confusion matrix reports the adjacent-class concentration and identifies fully evasive as easiest to detect.
9 Discussion
Discussion centers on ambiguity at class boundaries, shared model difficulty, judge position bias, and practical uses for monitoring evasive corporate communication.
- Ambiguous Classes: Intermediate evasion is difficult because executives can appear responsive while sidestepping the question’s core ask.The same ambiguity affects human annotation and model predictions.
- Qualitative Error Analysis: Quantitative questions expose a directness mismatch: specific qualitative explanations may be predicted direct but human-labeled intermediate for omitting numbers.The model and annotators differ when specificity substitutes for the requested numeric answer.
- Shared Difficulty: 33.6% of error samples are misclassified by 5–6 top models, while only 10.5% of Eva-4B errors are unique to it.The paper interprets these shared errors as genuinely ambiguous boundary cases.
- Position Bias: Randomization raises Opus’s win rate from 63.5% to 68.6% (+5.1%), demonstrating position bias in LLM judging.The experiment uses 5,541 samples and motivates randomized presentation order.
- Practical Applications: EvasionBench supports automated investor-relations screening, regulatory monitoring of disclosure quality, and research on strategic communication.These applications connect the benchmark to practical and academic uses of evasion detection.
10 Conclusion
The paper introduces EvasionBench and MMC for managerial-evasion detection in earnings calls, and concludes that a compact Eva-4B model can perform strongly with consensus data.
- Conclusion: EvasionBench is presented as the first large-scale benchmark for detecting managerial evasion in earnings calls.The conclusion frames the benchmark around corporate earnings-call communication.
- Conclusion: Eva-4B demonstrates that a 4B-parameter model can match or exceed frontier LLMs when trained with high-quality consensus data.The conclusion attributes this result to the training data quality rather than model scale alone.
- Conclusion: MMC provides a scalable annotation approach that outperforms single-model labeling.The paper’s anti-bias methodology includes randomized presentation order for LLM judges.
- Conclusion: The authors release data, models, and code to facilitate future research.The conclusion states the release as part of the benchmark contribution.
Limitations
The benchmark is constrained by its domain, annotation scale, temporal coverage, and language and company scope. The authors also caution that model outputs should not serve as sole evidence for legal or regulatory actions.
- Generalization beyond earnings calls remains unvalidated for political interviews and legal depositions.
- The Gold 1K set is validated, but large-scale training labels rely on single-annotator labeling with IAA verification on 100 samples.The authors state that larger-scale human annotation would strengthen validity.
- The dataset covers 2002–2022, so evolving language patterns and evasion strategies may require periodic updates.
- The benchmark includes only English-language transcripts from U.S.-listed companies.
- Model outputs are intended for research and analytical purposes, not as sole evidence for legal or regulatory actions.
A Training Hyperparameters
The paper identifies the training hyperparameters in Table 6 and reports evaluation results for 12 models in Table 7.
- Table 6 presents the training hyperparameters for two-stage fine-tuning.
- Table 7 presents complete evaluation results for all 12 models on the Gold 1K evaluation set.
- Table 7 groups models into Eva, Closed, Open, and Base categories and reports all metrics as percentages.