PaperlayerView exact revision on arXiv

Source-linked AI summary

Teacher-Free Self-Distilled Consistency Trajectory Learning for Fast Speech Enhancement

Shuubham Ojha, Carol Espy-Wilson

arXiv:2609.10392v2eess.AS

TL;DR

Diffusion-based speech enhancement can require many reverse steps, and SBCTM relies on a pretrained teacher for trajectory supervision. This paper replaces that teacher with EMA self-distillation and trains with a three-stage curriculum. On VoiceBank+DEMAND, its two-step geometric model reports PESQ 3.01, ESTOI 0.87, and SI-SDR 19.07 dB; schedule sweeps show a perception–fidelity trade-off.

  • Problem

    SBCTM requires a separately pretrained teacher to provide trajectory targets, adding training and memory cost and tying student quality to the teacher.

    Source: §1 · p. 1

  • Method

    The framework replaces the external teacher with an EMA copy of the student and trains through clean-speech prediction, self-distilled shortcuts, and perceptual fine-tuning.

    Source: §1 · p. 1

  • Results

    PESQ 3.01, ESTOI 0.87, and SI-SDR 19.07 dB are reported for the two-step geometric model on VoiceBank+DEMAND using the same backbone as SBCTM.

    Source: §5 · p. 3

  • Takeaways & Limitations

    A geometric schedule at low step counts maximizes perceptual quality, while a higher-step uniform schedule favors signal fidelity.

    Source: §8 · p. 4

Abstract

from arXiv · show

Consistency trajectory models offer a route to fast, high- quality speech enhancement, collapsing the many reverse steps of diffusion-based enhancers into a handful. When instantiated on a Schr\"odinger bridge (SB), which pins the generative process to fixed clean and noisy endpoints, exist- ing consistency-trajectory enhancers (SBCTMs) still require a pretrained teacher to supply trajectory supervision, which raises training cost and ties the final quality to that of the teacher. We propose a teacher-free, self-distilled consistency- trajectory framework that removes the external teacher result- ing in a 5X reduction in per epoch training time. Our model is trained with a three-stage curriculum of clean speech pre- diction, a self-distilled shortcut objective, and perceptual fine-tuning with a multi-resolution short-time Fourier trans- form (MR-STFT) loss. Using the same NCSN++ backbone as SBCTM, our model attains a wide-band PESQ of 3.01, ES- TOI 0.87, and SI-SDR 19.07 dB on VoiceBank+DEMAND compared to 3.57, 0.87 and 12.8 dB for the teacher based model. Further, we find that a geometric schedule at low reverse step count maximizes perceptual quality, while a higher-step uniform schedule favors signal fidelity.

1 Introduction

Generative speech enhancement can achieve strong perceptual quality, but diffusion-based systems often require many reverse denoising steps. This work replaces SBCTM’s pretrained teacher with EMA self-distillation and introduces a three-stage teacher-free framework.

1 Introduction

  • Score-based and diffusion speech enhancers can achieve state-of-the-art perceptual quality, but typically require tens to hundreds of reverse denoising steps.Consistency methods instead learn jumps between points along the generative path to enable few-step generation.Source: §1 · p. 1
  • The Schrödinger bridge pins the process to clean-speech and noisy-observation endpoints, and SBCTMs combine it with consistency trajectories for real-time-factor gains over diffusion SB baselines.The paper identifies SBCTM’s dependence on a separately pretrained teacher as a key limitation.Source: §1 · p. 1
  • The framework replaces the separately pretrained teacher used for trajectory targets with an EMA copy of the student, avoiding a full teacher training run.The paper identifies the teacher’s training-time inference and memory cost, and the cap it places on student quality, as motivations for this change.Source: §1 · p. 1
  • The proposed contributions include a three-stage curriculum and a step-count sweep finding a perception–fidelity trade-off: low-step geometric schedules favor perceptual quality, while higher-step uniform schedules favor signal fidelity.The curriculum comprises x0 prediction, a self-distilled shortcut objective, and perceptual fine-tuning.Source: §1 · p. 1

2 Related Work

Prior speech-enhancement work uses iterative score-based or diffusion denoising, while consistency trajectories offer a way to jump along a generative path. This paper removes SBCTM’s pretrained-teacher dependence through EMA self-distillation and uses a Schrödinger bridge with clean and noisy endpoints.

2 Related Work

  • Score-based and diffusion models enhance speech through iterative denoising, while consistency trajectory learning maps points on an ODE trajectory toward a shared endpoint.CTM can be trained by distillation from a pretrained score model or by enforcing self-consistency along the trajectory.Source: §2 · p. 1
  • SBCTM applies consistency-trajectory learning to a Schrödinger bridge and uses a separately pretrained teacher, whereas this work uses EMA self-distillation and an MR-STFT objective instead of a differentiable-PESQ loss.The figure overview depicts Phase I boundary denoising and Phase II teacher-free targets for learning intermediate jumps.Source: §2 · p. 2, Figure 1 · p. 2
  • The Schrödinger bridge process is pinned to clean speech x0 at t=0 and noisy observation y at t=1; enhancement transports y back toward t=0.The paper adopts a variance-exploding SB marginal and states k=2.6 and c=0.05.Source: §2 · p. 2

3 Proposed Method

The method parameterizes jumps along a Schrödinger-bridge trajectory using a clean-speech-scale network. It trains through boundary denoising, EMA self-distilled shortcuts, and MR-STFT fine-tuning, then uses a warped inference grid whose schedule affects quality.

3.1 Trajectory parameterization

  • The trajectory parameterization conditions gθ on noisy observation y, source time t, and target time s≤t, with output on the scale of clean speech.At s=t, the raw network output is the clean-speech estimate; during inference, the model follows a decreasing time grid from y.Source: §3.1 · p. 2

3.2 Phase I: clean-speech prediction

  • Phase I warms up the boundary denoiser by regressing its clean-speech prediction against x0, retaining this denoising objective as an anchor later.Because Phase I does not learn shortcuts, it uses denoise-renoise sampling: predict clean speech, then reproject the estimate onto the bridge at a lower time.Source: §3.2 · p. 2

3.3 Phase II: self-distilled shortcut learning

  • Phase II uses an EMA student copy to form teacher-free targets by denoising once and reprojecting onto the bridge at a lower time, while training shortcut jumps.The objective combines shortcut learning with a concurrent denoising anchor; inference chains the learned jumps, with one network evaluation per step.Source: §3.3 · pp. 2–3

3.4 Phase III: perceptual fine-tuning

  • Phase III adds waveform-domain MR-STFT loss over three resolutions, with spectral-convergence and log-magnitude ℓ1 terms.The perceptual objective does not optimize the evaluation metric directly, unlike SBCTM’s differentiable-PESQ auxiliary loss.Source: §3.4 · p. 3

3.5 Inference steps and schedule

  • At low inference step counts, schedule and step count affect quality; the two-step geometric grid {1.0, 0.344, 0.03} is the reported best-PESQ operating point.The intermediate grid nodes are warped toward the clean end, and the schedule terminates at t=0.03 rather than 0.Source: §3.5 · p. 3

4 Experimental Setup

Experiments use VoiceBank+DEMAND test utterances and the same NCSN++ trajectory-network backbone as SBCTM, evaluating enhancement quality with several speech metrics.

4 Experimental Setup

  • Evaluation covers all 824 VoiceBank+DEMAND test utterances at 16 kHz.Inputs are complex spectrograms; Table 1 reports performance on the same 824-utterance test set.Source: §4 · p. 3, Table 1 · p. 3
  • The 66.6 M-parameter NCSN++ trajectory network matches SBCTM’s backbone, with an effective batch size of 16.Training uses micro-batches of 2 with gradient accumulation over 8 steps.Source: §4 · p. 3
  • Reported metrics are wide-band PESQ, ESTOI, SI-SDR, and DNSMOS P.808 and P.835; Table 1 treats higher scores as better except for inference steps.DNSMOS P.808 is specified for Table 1.Source: §4 · p. 3, Table 1 · p. 3

5 Results

The teacher-free model is evaluated against SBCTM and across inference schedules. Its two-step geometric configuration leads in PESQ, while the preferred schedule depends on whether perception or signal fidelity is prioritized.

5 Results

  • PESQ 3.01, ESTOI 0.87, and SI-SDR 19.07 dB are reported for the two-step geometric model on the same backbone as SBCTM.The authors identify this as their primary configuration; Table 2 contrasts the models using the identical backbone.Source: §5 · p. 3, Table 2 · p. 4
  • PESQ peaks at 3.01 for two-step geometric, while SI-SDR peaks at 20.07 dB for four-step uniform; three-step geometric maximizes DNSMOS.Four-step uniform has PESQ 2.95; two-step geometric exceeds the single-step model’s PESQ 2.93 and three- and four-step geometric scores of 2.96 and 2.94.Source: §5 · p. 3, Table 3 · p. 4
  • The authors suggest SBCTM’s PESQ advantage may reflect its differentiable-PESQ training loss, unlike this model’s more general MR-STFT objective.The proposed model uses the MR-STFT perceptual objective.Source: §5 · p. 4

6 Ablation Study

The curriculum ablation shows that the shortcut stage supplies most of the perceptual gain, while the clean-speech prediction stage favors signal fidelity. Skipping Phase II still yields a close PESQ result with higher fidelity.

6 Ablation Study

  • PESQ rises from 2.58 in Phase I to 2.99 in Phase II and 3.01 in Phase III at the two-step geometric schedule.The curriculum ablation is reported in Table 4.Source: §6 · p. 4, Table 4 · p. 4
  • Phase I reaches the highest SI-SDR, 19.52 dB, while Phase II reaches PESQ 2.99 with SI-SDR 19.28 dB.The authors relate Phase I’s direct x0 regression to stronger signal fidelity and lower perceptual quality.Source: §6 · p. 4
  • Skipping Phase II still reaches PESQ 2.95, within 0.06 of the full model, while retaining SI-SDR 19.76 and OVRL 3.21.The shortcut loss is also part of the Phase-III objective.Source: §6 · p. 4

7 Training Efficiency

Teacher-free training uses substantially less per-epoch time than SBCTM in the reported configuration, though a strictly batch-matched wall-clock comparison is not possible on the tested GPU.

7 Training Efficiency

  • Phase I, II, and III require 19.3, 45.2, and 64.6 minutes per epoch, respectively, versus 327.1 minutes for SBCTM.Both configurations use an effective batch size of 16.Source: §7 · p. 4
  • Peak allocated memory is 6.1, 11.2, and 15.8 GB across the proposed model’s phases, versus 9.57 GB for SBCTM’s reported configuration.SBCTM uses micro-batch 1 with 16 gradient-accumulation steps.Source: §7 · p. 4
  • SBCTM runs out of memory when its micro-batch is increased to 2 on the same 24-GB RTX 3090, preventing a strictly batch-matched timing comparison.The reported SBCTM configuration uses micro-batch 1.Source: §7 · p. 4

8 Conclusion

The framework replaces SBCTM’s pretrained teacher with an EMA student on a variance-exploding Schrödinger bridge, and its two-step geometric model achieves the reported VoiceBank+DEMAND scores. A schedule sweep finds a perception–fidelity trade-off that varies with step count.

8 Conclusion

  • PESQ 3.01, ESTOI 0.87, and SI-SDR 19.07 dB on VoiceBank+DEMAND are achieved by the two-step geometric model.The model uses the same backbone as SBCTM.Source: §8 · p. 4
  • An EMA copy of the student generates trajectory targets on a variance-exploding Schrödinger bridge, removing SBCTM’s pretrained teacher.The framework is trained with a three-stage curriculum.Source: §8 · p. 4
  • At low step counts, a geometric grid maximizes PESQ, while a higher-step uniform grid favors signal fidelity.The geometric advantage narrows as the grid is refined.Source: §8 · p. 4
Loading 2609.10392v2…