PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo-Labels as Rewards, Not Targets: Group-Relative Policy Optimization for Semi-Supervised ASR

Shashi Kumar, Yanis Labrak, Sergio Burdisso, Esaú Villatoro-Tello, Petr Motlicek, Kadri Hacioglu, Andreas Stolcke

arXiv:2610.32572v1cs.CLeess.AS

TL;DR

Adapting ASR to untranscribed domain audio commonly uses teacher pseudo-labels, but SFT trains on every token, including errors. This paper uses the same labels as reward references for GRPO; it reports better robustness across teachers and settings, with on-policy sampling limiting error reinforcement.

  • Problem

    Domain adaptation often has only untranscribed target audio, while standard SFT treats every teacher pseudo-label token as a target, including erroneous ones.

    Source: §1 · p. 1

  • Method

    The method retains teacher pseudo-labels as WER-based reward references for group-relative policy optimization instead of using them as maximum-likelihood targets.

    Source: §1 · p. 1

  • Results

    Across label qualities GRPO outperforms SFT, and on banking it achieves 14.06% WER with Whisper Tiny labels versus 17.21% WER for SFT with Large-V3 labels.

    Source: §1 · p. 1

  • Takeaways & Limitations

    GRPO limits reinforcement of pseudo-label errors because erroneous words rarely appear in on-policy rollouts, unlike after SFT, which makes them common and reduces reward variance.

    Source: §1 · p. 1, §2.4 · p. 2

    Direct GRPO requires the student to support the target output script; when it does not, a short SFT stage is needed to establish that support.

    Source: §4 · p. 3

Abstract

from arXiv · show

Adapting ASR to a domain with only untranscribed audio typically relies on pseudo-labeling, a strong teacher transcribes the audio and a student is fine-tuned on it. Because supervised fine-tuning (SFT) treats every pseudo-label token as a target, the student inherits the teacher's errors. We show that using the same pseudo-labels as a reward reference for Group Relative Policy Optimization (GRPO) is more robust to label noise. In the banking domain, GRPO yields 29.3% to 56.2% relative WER improvements over SFT on Whisper-generated labels, and the advantage holds across four Whisper sizes, four teacher families, three domains and four languages. While SFT reproduces almost 100% of teacher errors verbatim, GRPO passes through only 27%. Rollout statistics explain why, erroneous pseudo-label words rarely appear in on-policy rollouts, so group-relative advantages stay aligned with the clean transcript, whereas SFT forces them into the policy's support and collapses the reward variance GRPO relies on.

1. INTRODUCTION

Untranscribed domain audio is commonly adapted through teacher-generated pseudo-labels and SFT, which makes teacher errors targets. This work instead uses those labels as reward references for GRPO and reports stronger results across label qualities and evaluation settings.

1. INTRODUCTION

  • SFT treats each pseudo-label token as a target, so teacher errors can be memorized and reproduced by the student.Existing approaches improve, filter, regenerate, or soften labels, but still train the student to reproduce teacher output.Source: §1 · p. 1
  • GRPO keeps the pseudo-labels but uses them as references for WER-based rewards rather than maximum-likelihood targets.The paper studies GRPO with unreliable references in the transcript-free adaptation setting.Source: §1 · p. 1
  • 14.06% WER with GRPO on Whisper Tiny labels beats 17.21% WER with SFT on Large-V3 labels in banking.GRPO outperforms SFT at every label quality and is less sensitive to pseudo-label degradation.Source: §1 · p. 1
  • The advantage extends across teacher families and conversational domains, and to low-resource languages once SFT makes reinforcement learning applicable.Source: §1 · p. 1
  • On-policy sampling explains the robustness: erroneous pseudo-label words rarely enter GRPO rollouts, while SFT puts them into nearly every rollout and removes needed reward variance.Source: §1 · p. 1

2. METHOD

The method adapts a student from untranscribed target-domain audio using teacher pseudo-labels. SFT makes each label token a target, whereas GRPO samples hypotheses and uses group-relative rewards; its noise filtering depends on what the student already generates.

2.1. Setup

  • The setup uses untranscribed target-domain audio, teacher-produced pseudo-labels, and an out-of-domain initialized student; gold transcripts are reserved for evaluation and diagnostic or baseline training.Source: §2.1 · p. 2

2.2. Supervised Fine-Tuning

  • SFT minimizes token-level cross-entropy on pseudo-labels, applying loss to every token whether or not it is correct.Teacher forcing also conditions later predictions on erroneous tokens as if they were correct.Source: §2.2 · p. 2

2.3. Group Relative Policy Optimization

  • GRPO samples a group of hypotheses per utterance, scores them against the pseudo-label, normalizes advantages within the group, and updates the policy with a clipped objective and KL penalty.Source: §2.3 · p. 2

2.4. Why on-policy sampling should filter label noise?

  • At teacher-error positions, group-relative normalization cancels a shared penalty when rollouts agree, so error reinforcement scales with its rollout probability rather than being eliminated.The mechanism requires that probability to be small initially, but not an accurate student; it is self-limiting, not self-correcting.Source: §2.4 · p. 2

3. EXPERIMENTAL SETUP

Experiments use a speech-LLM student and conversational domain data, with additional low-resource language tests. The setup varies teacher family and label quality while holding out gold transcripts for evaluation.

Student model

  • The student is a speech-LLM with a frozen WavLM-Large encoder and Llama-3.2-1B-Instruct, while only its two-layer projector and LoRA adapters are trained.The student is fine-tuned on LibriSpeech 960 hours.Source: §3 · p. 2

Target domains

  • The primary domain is English banking calls with about 54 hours of training audio and 6.55 hours of test audio; healthcare and insurance calls provide additional domains.Source: §3 · p. 2

Multilingual data

  • Low-resource language experiments use Common Voice data for Hindi, Hebrew, Vietnamese, and Serbian, with Whisper Large-V3 pseudo-label WERs ranging from 6.08% to 26.60%.Source: §3 · p. 2

Teachers

  • Teachers include four Whisper sizes as a controlled quality ladder and Qwen3 ASR, Phi-4 Multimodal IT, and Audio Flamingo 3 as additional families.Whisper banking-label WER ranges from 14.61 to 26.42.Source: §3 · p. 2

Training

  • SFT and GRPO use five training epochs and shared optimizer, precision, batch-size, and clipping settings, with distinct learning rates and GRPO-specific sampling and KL settings.Runs use one NVIDIA H100 80 GB GPU; test decoding is greedy with a 256-token limit.Source: §3 · p. 2

4. RESULTS

Across banking, GRPO is more robust than SFT to weaker pseudo-labels, and the advantage extends across teacher families, conversational domains, and low-resource languages after SFT where required.

  • 14.06% WER with GRPO on Tiny labels beats SFT's 17.21% with Large-V3 labels by 3.15 points.Source: §4 · p. 2, Table 1 · p. 3
  • With Tiny labels, SFT→GRPO reaches 22.77 versus 14.06 for direct GRPO; GRPO reduces insertions from 15.17 to 1.87 while deletions rise from 4.91 to 5.69.Source: §4 · p. 3
  • The pseudo-label penalty versus gold-label results is 6.94 versus 1.39 WER points in banking, 6.44 versus 2.04 in healthcare, and 5.57 versus 1.35 in insurance for SFT versus GRPO.Source: §4 · p. 3
  • Across healthcare and insurance, GRPO lowers WER from 18.72 to 14.64 and 19.59 to 15.70, respectively, and it beats SFT with all three additional teacher families.Source: Table 3 · p. 3, Table 2 · p. 3, §4 · p. 3
  • Adding GRPO after SFT reduces CER for all four languages with pseudo-labels, and systems match or outperform gold-label counterparts; Vietnamese CER falls from 58.78 to 49.92.Source: Table 4 · p. 3, §4 · p. 3
  • Direct GRPO is omitted because English-trained rollouts do not produce target scripts; a short SFT stage establishes the script and makes subsequent RL well-posed.Source: §4 · pp. 3–4, Table 4 · p. 3

5. ANALYSIS

The analysis tests whether on-policy sampling explains why GRPO copies fewer pseudo-label errors than SFT and why SFT→GRPO remains behind direct GRPO. Direct GRPO copies fewer errors as corruption rises, while errors rarely enter its rollouts and its reward variance remains higher than after SFT.

5. ANALYSIS

  • On-policy sampling can reinforce a label error only when the policy samples it, whereas SFT trains on every pseudo-label token unconditionally.Gold transcripts identify pseudo-label errors only for the post-hoc analysis, with utterances binned by error count.Source: §5 · p. 4
  • Direct GRPO copies progressively fewer pseudo-label errors as corruption rises, from 68% in the one-error bin to 27% in the 11+ error bin.SFT copies nearly 100% across corruption levels, and SFT→GRPO closely follows it.Source: §5 · p. 4, Figure 1 · p. 4
  • Before direct GRPO, erroneous words appear in only 10 to 30% of rollouts, versus more than 95% after SFT.Direct GRPO's within-group reward standard deviation is 0.06 to 0.09, compared with about 0.01 after SFT.Source: §5 · p. 4

6. CONCLUSION

The conclusion presents GRPO with pseudo-labels as reward references as a way to adapt ASR without target-domain transcripts. It attributes robustness to on-policy sampling and identifies model initialization and output-format compatibility as conditions on the approach.

6. CONCLUSION

  • GRPO with pseudo-labels as reward references reduces WER by up to 18.1 points relative to SFT without target-domain transcripts.The conclusion attributes this robustness to on-policy sampling, which limits reinforcement of pseudo-label errors.Source: §6 · p. 4
  • Direct GRPO is effective when the student supports the target language, whereas SFT is essential when its output format is incompatible with the task.Source: §6 · p. 4
  • Future work will combine GRPO with iterative pseudo-labeling and restrict initial SFT to high-confidence utterances.The stated aim of the SFT restriction is to avoid copying teacher errors.Source: §6 · p. 4

8. COMPLIANCE WITH ETHICAL STANDARDS

The authors report that the research used open-access or institutionally licensed human speech data and did not require ethical approval under the corpora licenses.

8. COMPLIANCE WITH ETHICAL STANDARDS

  • The study used CommonVoice 26 and DefinedAI speech data available in open access or under institutional license.Source: §8 · p. 5
  • The authors state that ethical approval was not required, as confirmed by the corpus licenses.Source: §8 · p. 5
  • The research was conducted retrospectively using human subject speech data.Source: §8 · p. 5

Related papers

Loading 2610.32572v1…