Source-linked AI summary
Does Financial Speech Need Continual Learning Regularization? A Study of EWC and mini-KFAC on Earnings Call ASR
Thai Quoc On, Homayoon Beigi
TL;DR
Whether continual adaptation of financial-speech ASR causes forgetting is underexplored. The paper compares unregularized fine-tuning, diagonal EWC, and mini-KFAC EWC while adapting Whisper-small to Earnings22; all methods improve held-out financial-speech performance during adaptation.
Problem
Method
The study adapts Whisper-small from a TEDLIUM-derived corpus to Earnings22 and compares unregularized fine-tuning, diagonal EWC, and mini-KFAC EWC.
Source: §1 · p. 1Results
Takeaways & Limitations
Assess task complementarity before applying regularization; when catastrophic forgetting is observed, mini-KFAC EWC is preferable to diagonal EWC, with careful strength tuning.
Source: §5 · pp. 4–5Evidence is limited to two Earnings22 partitions and five seeds; pairwise regularization comparisons are underpowered, and behavior under larger domain shifts is not established.
Source: §7 · p. 5
Abstract
from arXiv · showhide
Earnings call transcription presents a realistic continual learning scenario where Automatic Speech Recognition (ASR) systems are updated with new recordings but risk catastrophic forgetting of previously learned financial speech patterns. Adapting Whisper-small from a general-domain speech corpus to the Earnings22 benchmark reveals the opposite: all evaluated methods exhibit improvement on held-out financial speech from the same domain during adaptation rather than forgetting. This task complementarity makes regularization counterproductive. Nevertheless, when regularization is warranted, a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation seems to achieve lower word error rates (WERs) and more stable optimization than standard diagonal Elastic Weight Consolidation (EWC). The results suggest that task complementarity should be assessed before applying continual learning constraints, and prefer mini-KFAC EWC over diagonal EWC when needed.
1 Introduction
Financial-speech ASR raises a continual-learning question: whether sequential adaptation causes forgetting or benefits from task complementarity. This study compares EWC approaches and finds held-out financial speech improves during adaptation.
1 Introduction
- Financial-speech ASR adaptation risks catastrophic forgetting, but whether it occurs in this domain remains underexplored.When tasks share acoustic and linguistic structure, adaptation to one partition may improve performance on another—a phenomenon termed task complementarity here.Source: §1 · p. 1
- Diagonal Fisher approximations ignore parameter correlations; Kronecker-factored EWC had reduced forgetting versus diagonal EWC elsewhere, but not yet been examined in financial speech.The prior comparison was reported by McInerney et al. (2021).Source: §1 · p. 1
- All evaluated methods improve performance on held-out financial speech during adaptation, showing negative forgetting in this single-domain setting.The study adapts Whisper-small to Earnings22 and reports this as its central finding.Source: §1 · p. 1
- The study adapts Whisper-small from a TEDLIUM-derived corpus to Earnings22, comparing unregularized fine-tuning, diagonal EWC, and mini-KFAC EWC.Mini-KFAC EWC is the memory-efficient Kronecker-factored variant.Source: §1 · p. 1
2 Related Work
The related work frames financial-speech ASR as an underexplored continual-learning setting and motivates testing Kronecker-factored regularization alongside diagonal EWC.
2 Related Work
- Continual-learning research seeks to learn sequential tasks without forgetting; EWC penalizes parameter drift using Fisher information, alongside replay and architectural expansion alternatives.EWC is attributed to Kirkpatrick et al. (2017); alternatives include Rebuffi et al. (2017), Rusu et al. (2016), Zenke et al. (2017), and Serra et al. (2018).Source: §2 · p. 1
- ASR continual learning is less studied than vision, and financial-speech benchmarks such as Earnings22 are underexplored in this context.The passage cites Vander Eeckt and Van Hamme (2022), Plantinga et al. (2023), and Rio et al. (2022).Source: §2 · p. 1
- KFAC approximates Fisher information layer-wise to capture parameter dependencies; prior examples report reduced forgetting for BERT medical entity linking and lower WER for children’s ASR.McInerney et al. (2021) reported 51% versus 27% forgetting reduction for Kronecker-factored versus diagonal EWC; Ahadzi et al. (2025) reported a 5.21% relative WER reduction over unregularized fine-tuning. Neither examined financial speech or task complementarity.Source: §2 · p. 2
3 Experimental Setup
The setup adapts Whisper-small from TEDLIUM-derived speech to Earnings22, estimates Fisher statistics before financial adaptation, and evaluates retention on a disjoint Earnings22 partition.
3 Experimental Setup
- Earnings22 is divided into Task B for fine-tuning and held-out Task A for retention evaluation, with MD5-based file partitioning ensuring non-overlapping speaker groups.Both partitions are evaluated alongside an unregularized baseline, diagonal EWC, and mini-KFAC EWC.Source: §3 · p. 2
- The three-stage pipeline adapts Whisper-small on TEDLIUM-derived speech, estimates diagonal and mini-KFAC Fisher statistics, then fine-tunes on Earnings22.Figure 1 presents the experimental pipeline architecture.Source: §3 · p. 2, Figure 1 · p. 2
- The comparison includes unregularized fine-tuning, diagonal EWC, and mini-KFAC EWC, with stronger regularization assigned to encoder than decoder layers.Mini-KFAC uses restricted linear layers, row subsampling, and CPU-side accumulation; regularization ramps from zero over the first 50 steps.Source: §3 · pp. 2–3
- Evaluation reports best Task B WER, final Task A WER, and the change in Task A WER before and after Stage B fine-tuning.Results are means ± standard deviations across five seeds; negative forgetting means Task A performance improved during Stage B rather than degraded.Source: §3 · p. 3
4 Results
Task A WER improved significantly under all three methods during Stage B fine-tuning, while unregularized fine-tuning achieved the lowest Task B WER. Mini-KFAC had lower WER than diagonal EWC on both tasks, but the differences were not statistically significant.
4.1 Finding 1: Task Complementarity Makes Regularization Counterproductive
- Task A WER improved significantly under every method; forgetting was −4.87% for baseline, −4.02% for mini-KFAC, and −3.66% for diagonal EWC.All three one-sample tests found improvement significant (p < 0.05).Source: §4.1 · p. 3, Figure 2 · p. 3, Table 1 · p. 3
4.2 Finding 2: Unregularized Fine-tuning Remains Superior
- Task B WER was 31.94% for baseline versus 35.23% for EWC and 34.26% for mini-KFAC; pairwise differences were not significant at n = 5.The reported effect sizes were large, and Figure 3 shows baseline with the lowest WER throughout training.Source: §4.2 · p. 3, Figure 3 · p. 4
4.3 Finding 3: Mini-KFAC Shows Consistent but Statistically Modest Advantages over Diagonal EWC
- Mini-KFAC reduced Task B WER by 0.97% (34.26% vs. 35.23%) and Task A WER by 0.36% (21.52% vs. 21.88%) relative to diagonal EWC.These directional differences were not statistically significant at n = 5 seeds.Source: Table 1 · p. 3, §4.3 · p. 4
- Task B variance was lower for mini-KFAC (std: 3.57 vs. 4.91), and it converged faster than diagonal EWC on both tasks across seeds.The authors note that larger-scale experiments are needed to confirm the modest advantage statistically.Source: Figure 2 · p. 3, §4.3 · p. 4, Figure 3 · p. 4
5 Discussion
In this within-domain setting, improvement on Task A suggests positive transfer dominates, so task relevance should be assessed before applying regularization. The discussion also relates this result to shared speech characteristics and contrasts mini-KFAC’s layer-wise structure with diagonal EWC.
5 Discussion
- Task A’s consistent improvement during Task B training suggests positive transfer dominates in this setting, making task relevance important to assess before applying regularization.The discussion says strong transfer can render such constraints unnecessary.Source: §5 · p. 4
- Task A and Task B are partitions of the same Earnings22 distribution, so the negative-forgetting result does not establish resistance to forgetting under larger domain shifts.Here, the signed WER change reflects generalization across speakers within one domain, not the classical retention–adaptation trade-off under larger shifts.Source: §5 · p. 4
- Non-overlapping speaker populations share prepared or semi-prepared professional English speech and controlled recording conditions, a plausible account of task complementarity.The discussion presents these shared properties as consistent with Stage A encoder representations generalizing constructively to Earnings22.Source: §5 · p. 4
- Mini-KFAC captures layer-wise structure and parameter dependencies, whereas diagonal EWC’s independent-parameter constraints can be overly restrictive in high dimensions.Its layer-wise scaling also addresses Fisher-magnitude imbalance across the network, which diagonal EWC cannot address.Source: §5 · p. 4
- Practitioners should evaluate forgetting before adding regularization, prefer mini-KFAC EWC when forgetting occurs, and tune strength to avoid constraints that limit adaptation or fail to preserve prior knowledge.Stable or negative forgetting indicates task complementarity and reduces the need for additional constraints.Source: §5 · pp. 4–5
6 Conclusion
The conclusion compares Fisher-based continual-learning methods for Whisper-small on financial speech. All methods show negative forgetting, and the authors recommend assessing task similarity before regularizing and considering mini-KFAC EWC when regularization is warranted.
6 Conclusion
- All evaluated methods show negative forgetting, with Task A performance improving throughout Stage B fine-tuning.The authors interpret this within-domain result as evidence of strong task complementarity in financial speech.Source: §6 · p. 5
- The comparative evaluation uses Fisher-based continual-learning methods for financial speech recognition with Whisper-small.Source: §6 · p. 5
- The authors recommend assessing task similarity before applying regularization and suggest mini-KFAC EWC when regularization is warranted.They suggest layer-wise parameter interactions may make Kronecker-factored approximations more effective than diagonal Fisher approximations in this setting.Source: §6 · p. 5
7 Limitations
The study uses two partitions of Earnings22 and five random seeds, limiting the strength and scope of its conclusions. Pairwise comparisons are underpowered, and the within-domain result should not be generalized to larger domain shifts.
7 Limitations
- The evaluation uses a controlled two-partition Earnings22 setup and five random seeds, with both tasks drawn from the same corpus.Source: §7 · p. 5
- At five seeds, pairwise comparisons between regularization strategies remain underpowered, and the normality assumption is difficult to verify.The reported effect sizes for mini-KFAC versus diagonal EWC are d ≤0.23; the authors identify larger seed counts and longer task sequences as future directions.Source: §7 · p. 5
- The observed negative forgetting is a within-domain result and does not establish that regularization is unnecessary under larger domain shifts.Source: §7 · p. 5
A Task Construction Rationale
The study uses disjoint Earnings22 partitions to measure retention within the same financial-speech domain, reflecting incremental updates from successive quarterly recordings. Its hyperparameter choices balance memory-efficient curvature estimation with a non-dominating regularization constraint.
A Task Construction Rationale
- Task A and Task B are disjoint Earnings22 partitions with different speakers and recordings, isolating retention within the same financial-speech domain.This controlled design is framed as a lower-bound study; stronger domain shifts may produce different regularization dynamics.Source: A Task Construction Rationale · p. 6
B Hyperparameter Details
- Stage B uses Task B for fine-tuning, while Task A is held out exclusively to measure retention.The partitions are deterministically assigned by hashing file identifiers.Source: B Hyperparameter Details · pp. 6–7
- The global EWC scales are intentionally different because diagonal penalties sum scalar Fisher terms, whereas mini-KFAC uses layer-wise matrix quadratic forms.Both scales were set so the regularization-gradient-to-ASR-gradient ratio remains below 1, guiding without overwhelming task learning.Source: B Hyperparameter Details · p. 7
- Mini-KFAC approximates the Fisher layer-wise with Kronecker products of activation and gradient covariances, capturing parameter correlations that diagonal methods ignore.The memory-efficient procedure uses subsampled statistics and CPU-side accumulation to fit the single-GPU budget.Source: B Hyperparameter Details · p. 7
C Training Dynamics Analysis
During Stage B, the EWC penalties rise as scaling ramps up and parameter drift grows, with different trajectories for mini-KFAC and diagonal EWC. The loss breakdown attributes the initial total-loss rise to the growing regularization term, which constrains ASR improvement relative to baseline.
C Training Dynamics Analysis
- The initial rise in total training loss during Stage B reflects the increasing EWC contribution as scaling ramps up and parameter drift grows.The total loss is the sum of ASR loss and EWC loss.Source: Figure 5 · p. 7, C Training Dynamics Analysis · p. 8
- Mini-KFAC’s penalty rises faster and peaks earlier, then decreases, while diagonal EWC sustains a higher, flatter penalty trajectory.The 50-step ramp-up and growing drift both contribute to the initial penalty increase.Source: C Training Dynamics Analysis · pp. 7–8
- Mini-KFAC stabilizes at a lower constraint level, allowing greater ASR progress while preserving retention and explaining its WER gains over diagonal EWC.The loss-component breakdown covers total loss, ASR loss, and EWC regularization loss.Source: Figure 6 · p. 7, C Training Dynamics Analysis · p. 8, Figure 7 · p. 8
D Statistical Significance
Across five seeds, all three methods show statistically significant negative forgetting, while pairwise method comparisons are not significant. The tests rely on approximate independence and normality assumptions, with limited power for small pairwise differences.
D Statistical Significance
- The t-tests assume approximately independent and normally distributed per-seed WER values, but normality cannot be verified empirically at n = 5.Pairwise comparisons involve smaller differences and are treated as suggestive rather than conclusive; bootstrap alternatives are proposed.Source: D Statistical Significance · p. 8
- No pairwise method comparison reaches significance at n = 5; mini-KFAC versus diagonal EWC differences are small (d ≤ 0.23).The baseline’s Task B WER effect sizes versus EWC and mini-KFAC are large (d ≈ 0.83–0.89), though those comparisons are not significant.Source: Table 2 · p. 9, D Statistical Significance · p. 9
- All three methods show statistically significant negative forgetting (p < 0.05, one-tailed), indicating Task A performance improves during Stage B.The one-sample tests compare forgetting against a zero-forgetting null.Source: Table 3 · p. 9, D Statistical Significance · p. 9
E Gradient Flow Analysis
Gradient ratios track how ASR and EWC objectives interact during Stage B: regularization remains active without vanishing or dominating. The findings are scoped to closely related financial-speech tasks, with several directions proposed for testing broader settings.
E Gradient Flow Analysis
- After the 50-step ramp-up, regularized methods maintain stable, non-zero gradient ratios, while smooth curves indicate stable optimization dynamics.The ratios indicate that EWC contributes consistently to the total loss rather than vanishing or dominating.Source: E Gradient Flow Analysis · p. 9
F When These Findings May Not Apply
- These findings concern closely related financial-speech tasks with negative forgetting; regularization may help under task mismatch, strong forgetting, or limited Task B data.The authors identify these settings as future evaluation directions.Source: F When These Findings May Not Apply · p. 9
G Future Directions
- Quantitative task-relatedness measures, including representation similarity, Fisher spectrum alignment, and loss overlap, could help predict when regularization is beneficial.Source: G Future Directions · p. 9
- Dynamic λ weighting could decrease when Task A performance improves during Stage B, allowing more flexible transfer in complementary domains.Source: G Future Directions · p. 9
- Future work could constrain only highly task-specific layers, using gradient attribution or attention sensitivity instead of global Fisher penalties.Source: G Future Directions · p. 9
- Evaluation on longer, multi-stage financial streams could test whether task complementarity persists across quarters, investor presentations, and regulatory transcripts.Source: G Future Directions · p. 10
- Comparisons with replay, adapter-based fine-tuning, and progressive architectures could situate mini-KFAC EWC among broader continual-learning baselines for financial ASR.Source: G Future Directions · p. 10