PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

Souranil Kahali, Rituparna Bose, Abner Hernandez, Tomas Arias-Vergara, Andreas Maier, Ning Ma, Paula Andrea Perez-Toro

arXiv:2608.18825v1cs.CLcs.AIcs.LGcs.SD

TL;DR

Medical and multilingual adaptation improves Whisper-based medical ASR, but its effects on internal representations remain insufficiently understood beyond WER. The paper compares adaptation strategies across model sizes and analyzes encoder layers, finding that English medical fine-tuning drives the largest representational shift while multilingual continuation largely preserves it.

  • Problem

    How medical and multilingual fine-tuning reorganise Whisper’s internal encoder representations remains unclear beyond recognition accuracy.

  • Method

    The study compares Whisper adaptation strategies across four model sizes and applies systematic layer-wise analysis of encoder representations.

  • Results

    Fine-tuning improves MedASR, with English medical adaptation causing the largest encoder drift and multilingual continuation largely preserving adapted representations.

  • Takeaways & Limitations

    Domain and language information remains recoverable across layers, while linearly recoverable error cues weaken as decoding improves.

  • Takeaways & Limitations

    The small corpora, limited probing samples, and analysis restricted to Whisper-Small constrain fine-grained, cross-speaker, and cross-scale conclusions.

Abstract

from arXiv · show

Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.

I. INTRODUCTION · II. RELATED WORK

Medical ASR adaptation must address specialised terminology, variable acoustic conditions, limited multilingual clinical data, and interpretability beyond WER. This work combines multi-size Whisper evaluation with layer-wise analysis to examine how monolingual and multilingual medical fine-tuning reshape encoder representations.

  • I. INTRODUCTION: Medical speech poses adaptation challenges because it contains rare terminology, variable acoustic conditions, and domain-specific language demands.General-purpose ASR performance does not necessarily transfer reliably to healthcare applications.
  • I. INTRODUCTION: Fine-tuning pretrained ASR models on clinical corpora substantially improves medical transcription performance.Prior work also combines domain-specific fine-tuning with language-model postprocessing, while multilingual benchmarks demonstrate adaptation feasibility across clinical languages.
  • II. RELATED WORK: Prior interpretability studies show that Transformer speech encoders organise acoustic, phonetic, phonological, and linguistic information differently across layer depths.This literature motivates probing frozen representations with lightweight classifiers and examining layer selection or accent information.
  • I. INTRODUCTION: The study systematically compares zero-shot, monolingual, continued EN→EN+DE, and direct EN+DE fine-tuning across multiple Whisper sizes.The two-stage EN→EN+DE setting means English medical fine-tuning followed by continued training on combined English–German medical speech from the English checkpoint.
  • I. INTRODUCTION: Adapted models are evaluated with WER, while Whisper-Small encoder representations are analysed through drift and probing tasks.The analysis targets how medical and multilingual adaptation changes internal representations rather than relying only on transcription accuracy.
  • II. RELATED WORK: Whisper and Wav2Vec2 provide strong pretrained foundations, but their general-domain strengths do not eliminate the need for targeted medical adaptation.Whisper offers zero-shot and cross-lingual capabilities, whereas Wav2Vec2 relies on self-supervised contrastive pretraining and supervised fine-tuning.
  • II. RELATED WORK: No prior work jointly examines two-stage multilingual medical ASR adaptation and layer-wise probing of domain, language, and error-predictability signals.The paper positions this combination as its central research gap.
  • II. RELATED WORK: Unlike diagnostic-label pipelines, this work studies encoder layer-wise behaviour for interpretability without relying on diagnostic labels.This distinguishes the approach from prior clinical speech analysis focused on supervised classification tasks.

III. DATA AND PRE-PROCESSING

The study uses speaker-disjoint English medical data, privacy-restricted German procedure recordings, and a merged multilingual dataset for Whisper adaptation. Audio is standardized to 16 kHz mono with Whisper log-Mel features, while transcripts retain original casing and punctuation.

  • English Medical Corpus: 8.5 hours of English medical speech are split by speaker identifiers into 997 training, 136 validation, and 104 test samples.The corpus contains human-written transcriptions and is resampled to 16 kHz mono before fine-tuning.
  • German Medical Corpus: 86 German training samples come from one operating physician, so German-only results indicate within-corpus adaptation rather than cross-speaker generalisation.Validation and test sets contain 37 and 38 file-disjoint samples from a separate operation.
  • Multilingual Dataset: 1,083 training, 173 validation, and 142 test samples form the multilingual dataset by merging English and German splits with language identifiers.Evaluation uses language-specific forced decoder identifiers, including 104 English and 38 German test utterances.
  • Pre-processing and normalisation: 16 kHz mono audio and Whisper log-Mel features are used, with 80 bins for Base/Small/Medium and 128 bins for Large-v3.Transcripts retain original casing and punctuation; no text normalisation, denoising, voice-activity detection, or augmentation is applied.
  • Domain contrast and reproducibility: 100 LibriSpeech test-clean samples provide a same-language English contrast set so domain probing reflects encoder domain content rather than language discrimination.The study also plans to release English manifests, preprocessing scripts, evaluation code, and configurations, while German audio remains restricted by privacy regulations.

IV. FINE-TUNING METHODOLOGY · A. Training Configuration · B. Monolingual Medical Fine-Tuning

The study evaluates four Whisper fine-tuning settings under a shared training and decoding framework. English adaptation uses the common medical training setup, while German-only adaptation is explicitly treated as within-corpus diagnostic evidence.

  • A. Training Configuration: Four settings are evaluated: English-only, German-only diagnostic, two-stage EN→EN+DE continuation, and direct EN+DE fine-tuning from pretrained weights.Experiments use Hugging Face Transformers and PyTorch on a single NVIDIA A100 GPU.
  • A. Training Configuration: Training uses cross-entropy loss, greedy validation decoding, a 225-token generation limit, effective batch size 32, mixed precision, and gradient checkpointing.Early stopping uses patience 5 based on validation WER; checkpoints are saved every 200 steps and selected by lowest validation WER.
  • A. Training Configuration: Language-specific forced decoder identifiers control Whisper generation, with monolingual checkpoints evaluated separately and EN+DE outputs pooled after language filtering.Wav2Vec2 uses CTC argmax decoding, while WER and CER use jiwer without case-folding or punctuation removal.
  • B. Monolingual Medical Fine-Tuning: Each Whisper variant is fine-tuned on the English medical training set with the processor language fixed to English and best-checkpoint selection by validation WER.The shared configuration is used throughout English-only fine-tuning.
  • B. Monolingual Medical Fine-Tuning: 11.98% English test WER is achieved by the Whisper-Small EN-FT checkpoint at step 400 used for layer-wise analysis.Whisper-Small was selected to balance adapted ASR performance with the computational cost of repeated encoder hidden-state extraction.
  • B. Monolingual Medical Fine-Tuning: The German-only diagnostic applies the same configuration to 86 German training utterances with best-checkpoint selection by validation WER.Only German test performance is reported because the data come from a small single-speaker training set.
  • B. Monolingual Medical Fine-Tuning: German-only results are interpreted as within-corpus adaptation evidence rather than broad generalisation.This interpretation follows from the small, single-speaker German training set.

C. EN+DE Adaptation · V. LAYER-WISE ANALYSIS · A. Hidden-State Extraction

The study compares two multilingual EN+DE adaptation strategies and analyses how medical and multilingual fine-tuning reshape Whisper-Small encoder representations. Hidden states are extracted layer by layer and reduced to mean-pooled vectors for checkpoint comparison.

  • C. EN+DE Adaptation: Two-stage adaptation initializes combined EN+DE training from an English medical fine-tuned checkpoint.This tests whether English medical adaptation can be preserved while introducing German medical speech.
  • C. EN+DE Adaptation: Direct EN+DE fine-tuning instead starts from pretrained Whisper weights and trains on the same combined English–German dataset in one stage.Both multilingual settings exclude validation and test samples from gradient updates.
  • V. LAYER-WISE ANALYSIS: The layer-wise analysis examines Whisper-Small encoder representations to assess effects of medical and multilingual fine-tuning on the internal representation space.It combines hidden-state extraction, representation-drift measurement, and probing classifiers.
  • V. LAYER-WISE ANALYSIS: The analysis probes recoverable domain, language, and error-related information across encoder representations.These probing classifiers are part of the combined layer-wise analysis framework.
  • A. Hidden-State Extraction: Whisper-small provides 13 extracted hidden states, comprising the input embedding output and 12 encoder Transformer blocks.The states are indexed L0–L12.
  • A. Hidden-State Extraction: Each hidden state has shape T × 384, with T denoting sequence length.The representation dimensionality is therefore 384 before temporal pooling.
  • A. Hidden-State Extraction: Mean-pooling over time produces one 384-dimensional vector per audio sample and layer.This procedure is applied to hidden states extracted independently from the Pretrained, EN-FT, and multilingual ML-FT checkpoints.

B. Representation Drift · C. Probing Classifiers.

The paper measures layer-wise representation drift between adaptation checkpoints and probes domain, language, and WER information from frozen encoder features. These analyses use controlled speech contrasts, rank-based WER labels, and cross-validated linear and nonlinear classifiers while noting corpus-related confounds.

  • B. Representation Drift: Representation drift compares Pretrained →EN-FT, EN-FT →ML-FT, and ML-FT English checkpoints on the same 100 English audio samples.Per-layer drift is assessed without task labels using cosine similarity and Linear CKA.
  • B. Representation Drift: Cosine similarity measures directional alignment between mean representations, whereas Linear CKA measures changes in representational geometry.
  • C. Probing Classifiers.: The domain probe separates English medical speech from LibriSpeech general English speech, controlling language while retaining possible corpus-specific differences.Potential differences include accent, speaker population, recording conditions, and background noise.
  • C. Probing Classifiers.: Table I reports ASR WER (%) for English and German medical speech test sets across adapted Base, Small, Medium, and Large Whisper versions.
  • C. Probing Classifiers.: The language probe contrasts English medical with German medical speech, but corpus differences may confound speaker and acoustic-condition effects.
  • C. Probing Classifiers.: The WER probe labels 100 English test clips by checkpoint-specific greedy-decoding WER tertiles, contrasting bottom-third low-WER and top-third high-WER utterances.The middle third is excluded, leaving n=33 per class and applying the same procedure across checkpoints.
  • C. Probing Classifiers.: Near-ceiling probe F1 scores should be interpreted cautiously because domain and language contrasts contain corpus-related confounds.
  • C. Probing Classifiers.: Variable-length encoder sequences are mean-pooled, MinMax-scaled, and reduced to 50 principal components before probing.Logistic Regression, Linear SVC, and a one-hidden-layer MLP use frozen features with stratified 5-fold cross-validation; panel-mean macro-F1 is primary.

VI. EXPERIMENTS & RESULTS

The experiments evaluate ASR performance by comparing zero-shot and fine-tuned Whisper models with Wav2Vec2 baselines. They also analyse Whisper-Small encoder representations to examine how medical and multilingual adaptation affect layer-wise information.

  • The section compares zero-shot and fine-tuned Whisper models with Wav2Vec2 baselines for ASR performance.
  • The representation analysis focuses on Whisper-Small encoder layers.
  • The layer-wise analysis examines how medical and multilingual adaptation affect information encoded in Whisper-Small representations.

A. ASR Performance: Zero-Shot and Fine-Tuned

Fine-tuning substantially improves medical ASR performance and changes model rankings across English, German, and combined evaluation. Direct multilingual adaptation achieves the best combined WER, while the layer-wise findings apply specifically to the two-stage Whisper-Small trajectory.

  • Zero-shot and fine-tuned performance: 16.50% English WER, 56.51% German WER, and 35.52% combined WER make zero-shot Whisper-Large-v3 the strongest zero-shot Whisper model across all subsets.Wav2Vec2 CTC baselines transfer poorly out of domain, with English WER above 92% and German XLSR-53 at 76.00%.
  • Zero-shot and fine-tuned performance: 7.72% WER makes Whisper-Medium the best English model after English-only fine-tuning, while German transfer without German training is inconsistent.Whisper-Small degrades on German, whereas Whisper-Medium improves relative to its zero-shot baseline.
  • Zero-shot and fine-tuned performance: 44.96% German WER makes German-only Whisper-Large-v3 the strongest diagnostic model, with Whisper-Medium reaching 45.94%.This within-corpus diagnostic uses only 86 German training utterances, limiting its interpretation as robust generalisation.
  • Multilingual adaptation: 26.30% combined WER makes direct EN+DE fine-tuned Whisper-Medium the best overall combined model, while two-stage Whisper-Small reaches 31.33%.Two-stage EN→EN+DE adaptation improves over zero-shot for all Whisper sizes.
  • Scope of representation analysis: Layer-wise representation findings describe only the two-stage Whisper-Small path; comparable analyses of direct EN+DE and German-only encoders remain future work.The two-stage Whisper-Small trajectory is the model used for the representation-level analysis below.
  • Qualitative error audit: The qualitative audit compares zero-shot Whisper-Small, two-stage Whisper-Small, and German-only Whisper-Large-v3, with DE-Large included only for German rows.DE-Large is identified as the model with the best German WER.

B. Representation Drift

English medical fine-tuning causes the dominant encoder reorganisation, especially in upper layers, while multilingual continuation largely preserves the adapted representation space. Cross-lingual representations show centroid alignment but retain language-sensitive geometry.

  • B. Representation Drift: Pretrained→EN-FT produces the largest representational shift, concentrated in upper layers L8–L12.This indicates that English medical fine-tuning drives the main encoder reorganisation.
  • B. Representation Drift: EN-FT→ML-FT shows substantially smaller drift, largely preserving the English-adapted representation space.The same pattern appears across encoder layers in the five-seed drift analysis.
  • B. Representation Drift: 0.980–0.994 cosine similarity contrasts with 0.040–0.107 Linear CKA between ML-FT English and German representations.The pattern indicates centroid-level cross-lingual alignment while the full representation geometry remains language-sensitive.

C. Probing Results

Domain and language information remains highly recoverable across encoder layers and adaptation checkpoints, while linearly recoverable error-predictive information decreases as adaptation improves decoding. Layer-wise variation is significant for domain and language probes, but not for adapted WER probes after correction.

  • Domain Probe: Domain information remains highly separable across checkpoints and layers, with panel-mean macro-F1 at or near ceiling (≥0.984).Medical versus general speech remains consistently recoverable from frozen encoder representations across adaptation stages.
  • Language Probe: Language identity is almost perfectly separable across checkpoints and layers, with bootstrap mean F1 ≥0.990 from L1 onward.These language-discriminative features are preserved through both fine-tuning stages.
  • WER/Error-Predictability Probe: 0.721 ± 0.028 at L2 falls to 0.619 ± 0.039 at L6 and 0.556 ± 0.033 at L11 in best-layer WER-probe macro-F1.Mean macro-F1 likewise decreases from 0.666 to 0.569 and 0.509, while decoder WER on the same 100 English clips decreases from 19.87% to 13.61% and 12.48%.
  • Statistical Tests: All adjusted q < 10−4 for layer-wise domain and language variation after Bonferroni and FDR-BH correction.For the WER probe, only the pretrained checkpoint remains significant after correction, with p = 1.47 × 10−3, Bonferroni q = 1.32 × 10−2, and FDR-BH q = 1.89 × 10−3.

VII. DISCUSSION & CONCLUSION

Fine-tuning improves medical ASR, but the strongest adapted model depends on the language setting and data regime rather than model scale alone. Layer-wise analysis shows that English medical adaptation drives the largest representational change, multilingual continuation largely preserves it, and probe findings motivate targeted future adaptation strategies while requiring cautious interpretation.

  • Model performance: Fine-tuning substantially improves MedASR over zero-shot decoding, while the best adapted model varies by language setting and data regime.Whisper-Medium is strongest under English-only and direct EN+DE fine-tuning; Whisper-Large-v3 leads only in the German-only diagnostic.
  • Model performance: Whisper-Large-v3 leads the German-only diagnostic, but this result reflects within-corpus adaptation on 86 single-speaker training utterances and requires caution.The German subset is small, so the diagnostic does not establish robust generalisation.
  • Layer-wise analysis: English medical fine-tuning produces the largest encoder drift, concentrated in upper layers, whereas EN-FT→ML-FT drift is substantially smaller.The domain and language probe contrasts remain highly separable across layers, although near-ceiling separability partly reflects corpus, speaker, and acoustic-condition confounds.
  • Implications: Persistent domain and language information motivates parameter-efficient or selective fine-tuning, while weakening WER-predictive signals support difficulty-aware data selection from pretrained encoder representations.The strong English–German medical contrast also motivates further study of shared multilingual encoder architectures for clinical ASR.
  • Limitations and future work: The small medical corpora, limited probing sample sizes, and Whisper-Small-only layer-wise analysis constrain fine-grained, cross-speaker, and cross-scale conclusions.Future work should validate the findings on larger multilingual clinical corpora and extend analysis to larger Whisper variants, direct EN+DE, and German-only settings.
Loading 2608.18825v1…