Pseudo2Real: Task Arithmetic for Pseudo-Label Correction in Automatic Speech Recognition

1 Introduction

• Target-domain transcription can be costly, time-consuming, infeasible, or unavailable, motivating adaptation methods that do not require target ground truth.Privacy or legal constraints can also impede transcription collection.Source: §1 · p. 1
• Pseudo-labels can inherit systematic biases, including rare-word omissions, accent-driven substitutions, and domain-specific mis-segmentations.When used for adaptation, these errors can accumulate and degrade real-world performance.Source: §1 · p. 1
• Filtering and agreement checks suppress noise, while iterative pseudo-labeling approaches require multiple passes and tuning yet may still propagate recurring teacher mistakes.The introduction frames the need as correcting structured errors without target-domain ground truth.Source: §1 · pp. 1–2
• Pseudo2Real derives a source-domain correction vector from two models fine-tuned from the same backbone, then adds a scaled version to a pseudo-label-adapted target model.The vector is intended to capture systematic discrepancies introduced by pseudo-labeling.Source: §1 · p. 2

2.1 Pseudo-labeling for ASR unsupervised domain adaptation

• Prior pseudo-labeling approaches include repeated teacher updates, confidence filtering or agreement checks, and teacher refinement through an exponential moving average.These approaches improve or control label quality through different parts of the adaptation pipeline.Source: §2.1 · p. 2
• Pseudo2Real learns a reusable parameter-space correction from real-versus-pseudo training on auxiliary domains, then adds it to a new-domain model; it can complement filtering or iterative self-training.This contrasts with prior approaches that filter or iteratively refine noisy labels.Source: §2.1 · p. 2

2.2 Task arithmetic in speech

• Task-arithmetic research uses model-difference vectors for capability transfer, zero-shot ASR adaptation, language composition, and multilingual transfer.SYN2REAL uses a real-versus-synthetic speech vector to address acoustic distribution differences, typically for new text domains.Source: §2.2 · p. 2
• Unlike SYN2REAL’s real-versus-synthetic audio vector for acoustic gaps, Pseudo2Real targets real-versus-pseudo-label biases in new acoustic domains such as accented speech.Pseudo2Real-SC uses automatic speaker clustering rather than source-domain labels to form multiple correction vectors.Source: §2.2 · p. 3

3.1 Problem Formulation

• The problem setting uses paired source speech and text but only unlabeled target speech, where systematic teacher errors can pass to the student and resist confidence filtering.The goal is to mitigate these errors without target-domain ground-truth annotations.Source: §3.1 · p. 3

3.2 Pseudo2Real

• Pseudo2Real relies on fine-tuned models sharing a pretrained initialization so their parameter difference can act as a correction direction for pseudo-label bias.The paper motivates this interpretation through linear mode connectivity and task arithmetic.Source: §3.2 · p. 3
• The single-vector variant subtracts the source pseudo-label model from the real-label model, then applies the correction to a pseudo-label-fine-tuned target model.A scaling factor controls correction strength.Source: §3.2 · pp. 3–4
• Pseudo2Real-SC clusters source speakers and computes subgroup-specific correction vectors, then averages them for target application to preserve shared and localized bias corrections.The motivation is that pseudo-label errors can vary with accent, pronunciation style, or recording conditions.Source: §3.2 · p. 4

3.3 Baselines

• The baselines include pretrained and source-ground-truth references, direct target pseudo-label fine-tuning, confidence filtering, T5 error correction, EMA, and a target-ground-truth topline.Confidence filtering acts on labels, while EMA and Pseudo2Real modify model weights using different signals.Source: §3.3 · p. 4

4 Experimental Setup

• The evaluation selects ten AFRISPEECH-200 accents with complete train, development, and test splits while preserving official splits to avoid data leakage.Accent labels identify speakers’ native-language backgrounds, not the language of the English transcripts.Source: §4 · pp. 4–5
• Cross-fold validation alternates two accent folds as source and target, using paired source transcripts to derive corrections and unlabeled target speech for adaptation.The folds contain five accents each and are designed to have comparable utterance and speaker counts.Source: §4 · p. 5
• The experiments cover Whisper TINY, BASE, SMALL, MEDIUM, and LARGE-V2, spanning 39M to 1.55B parameters.The models share an encoder–decoder transformer architecture.Source: §4 · p. 5
• The scaling factor λ is selected by source-development WER grid search over 0.1 to 1.0, while evaluation uses greedy decoding and WER.Training uses AdamW and up to 40K update steps; the experiments total approximately 500 GPU-hours on one NVIDIA V100.Source: §4 · p. 5

5.1 Can Pseudo2Real improve ASR performance?

• Whisper TINY average WER falls from 89.3 to 57.7 (35% relative improvement over pseudo-label fine-tuning), while SMALL falls from 47.2 to 45.0.Table 1 reports WER (%) on ten accented-English target domains.Source: §5.1 · p. 5, Table 1 · p. 6
• On challenging accents including Ijaw, Idoma, and Yoruba, WER falls by up to 50 points versus pseudo-label fine-tuning.The authors attribute these gains to capturing accent-specific pronunciation patterns and mitigating systematic pseudo-labeling errors.Source: §5.1 · p. 6
• On Twi and Igbo, Pseudo2Real beats the topline with fewer insertion errors; Twi has 262 versus 670, while substitution and deletion counts remain comparable.These are the two accents examined in the tiny→tiny error breakdown.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 How does Pseudo2Real perform across different teacher model sizes?

• For TINY students, BASE and LARGE teachers yield +21.7% and +21.6% average relative improvements; for SMALL students, MEDIUM yields +5.2% and LARGE about +0.1%.The reported improvement rows compare Pseudo2Real with pseudo-labeled fine-tuning; gains vary by pairing and target accent.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 What is the impact of the scaling factor λ?

• Across transfer pairings, WER bottoms near λ=0.2–0.3; λ=0.1–0.2 improves over no correction, while larger scaling can degrade performance, especially LARGE→TINY.TINY→TINY and TINY→SMALL perform best at λ=0.3; the plotted WER averages five fold-1 accents.Source: §5.3 · p. 7, Figure 3 · p. 8

6.1 How is the Pseudo2Real-SC compared with the simple correction vector?

• The ensemble comparison pairs MEDIUM or LARGE teachers with a SMALL student, using eight k-means clusters to test whether subgroup vectors help where single-vector benefits are modest.The selected pairings also show pronounced per-accent variation.Source: §6.1 · p. 8
• Pseudo2Real-SC yields 4.0% average relative improvement with MEDIUM teachers and 6.1% with LARGE teachers over the single correction vector.Hausa gains are +21.8% and +17.5%, respectively; LARGE also improves Swahili by +9.9%.Source: §6.1 · p. 8
• Igbo degrades slightly with both teacher sizes, so subgroup averaging does not improve every accent.The paper notes that excessive averaging can weaken accent-specific corrections.Source: §6.1 · p. 8

6.2 Ablation on the number of clusters

• 41.94 WER at k = 8 is below 45.36 at k = 1, with intermediate WERs of 44.34 at k = 2 and 43.68 at k = 4.These LARGE→SMALL results average five fold-1 accents; Figure 4 shows WER decreasing as cluster count increases.Source: §6.2 · p. 9, Figure 4 · p. 9
• Finer clustering is suggested to capture more localized speaker- or accent-specific correction patterns and improve target-domain generalization.The tested cluster count ranges from 1, the standard Pseudo2Real setting, to 8.Source: §6.2 · pp. 8–9
• More clusters increase computational cost because two ASR models, trained on real and pseudo labels, are required per cluster.The paper identifies k = 4–8 as a practical balance between performance and efficiency.Source: §6.2 · p. 9

• Pseudo2Real replaces the incorrect token with “survives,” restoring the intended meaning while keeping the rest of the sentence intact.The teacher output is “as a vif,” and the pseudo-label-trained student produces “is a vice.”Source: §7 · p. 9
• The student inherits part of the teacher’s lexical confusion, illustrating propagation of a teacher error into pseudo-label training.Source: §7 · p. 9
• The example is presented as evidence that the correction vector adjusts internal representations to reduce systematic substitution errors in pseudo-labeling.Source: §7 · p. 9

• Experiments on AfriSpeech-200 across ten African accents and multiple Whisper sizes show consistent gains, including up to 35% relative WER reduction with Whisper TINY.The method occasionally surpasses topline models trained with true labels.Source: §8 · p. 9
• Pseudo2Real is a parameter-space correction method for systematic pseudo-label errors in ASR domain adaptation that does not require target-domain labels.Source: §8 · p. 9
• Pseudo2Real-SC yields additional improvements in several teacher–student settings.Source: §8 · p. 9
• Future work includes multilingual and spontaneous speech, dynamic correction scaling, interpretability, adaptive cluster-conditional vectors, and testing more than eight clusters.Source: §8 · p. 9

9 Limitation

• The method requires paired source speech and ground-truth transcripts; small, unrepresentative, or mismatched source data may yield underfit or mismatched correction vectors.These conditions can limit transfer to target accents.Source: §9 · p. 10
• Source-to-target bias recurrence is assumed, but highly accent-specific or nonoverlapping channel and recording errors may make correction weak or counterproductive.The correction scale λ also requires tuning on held-out source development data, and excessive scaling can degrade WER.Source: §9 · p. 10
• Experiments use a fixed two-fold split, leaving robustness to other source partitions and accent or language-family mixes uncharacterized.Consistent accent annotations are scarce in public ASR corpora, constraining controlled cross-accent experiments.Source: §9 · p. 10
• Generalization to other languages, far-field conditions, and domains beyond reading or conversational speech is not validated.Source: §9 · p. 10
• Filtering to English accents with complete splits may bias evaluation toward better-represented accents and excludes underrepresented varieties and code-switching scenarios.Source: §9 · p. 10

10 Ethical considerations

• Practitioners are urged to evaluate fairness across subgroups, use representative validation data, respect consent and data-protection rules, and consider possible surveillance misuse.The stated intent is to support low-resource and accessibility-oriented speech technologies.Source: §10 · p. 10

A Model characteristic

• Whisper models share an encoder–decoder transformer architecture but vary in scale; Table 6 reports approximate parameter counts and encoder and decoder layer counts.Source: Table 6 · p. 13, A Model characteristic · p. 13

B Data, Artifacts, and Licensing

• The experiments use public, consented, anonymized speech and open-licensed resources; associated code, correction vectors, checkpoints, and documentation are slated for CC BY-NC-SA 4.0 release after acceptance.The authors report no private-data collection or scraping.Source: B Data, Artifacts, and Licensing · pp. 13–14

C Use of AI assistants

• Large language models assisted with clarity, grammar, and readability, while the authors report conducting the conceptual, experimental, analytical, and interpretive work.Source: C Use of AI assistants · p. 14

D Language families of the ten accents

• The ten accents span three major language families and diverse linguistic structures, so their shared African geography does not make the evaluation linguistically homogeneous.Table 9 lists language-family and subfamily or branch assignments.Source: D Language families of the ten accents · p. 14

E K-Means Implementation

• Speaker clustering uses scikit-learn K-means with k-means++ initialization, Lloyd’s algorithm, a 300-iteration maximum, and tolerance 10^-4.Source: E K-Means Implementation · p. 14

F Baseline Implementation Details

• Error Correction trains T5 on source-domain Whisper hypotheses paired with human transcripts, mapping noisy hypotheses toward references without target ground-truth labels.Source: F Baseline Implementation Details · pp. 14–15
• At inference, T5 corrects target pseudo-labels before the Whisper student is fine-tuned on them, isolating text-level cleaning within the unsupervised adaptation setting.Source: F Baseline Implementation Details · pp. 14–15

G Additional case study

• Table 8 examples show Pseudo2Real fixes an accent-induced “side”→“said” confusion and removes hallucinated “full stop,” restoring “May Day.”Source: Table 8 · p. 15, G Additional case study · p. 16

F Baseline Implementation Details

• EMA maintains separate running weights, updates them after optimizer steps, and uses β = 0.999, the best average validation value among those tried.Evaluation and checkpoint saving temporarily swap in the EMA weights.Source: F Baseline Implementation Details · p. 15