PaperlayerView exact revision on arXiv

Source-linked AI summary

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

Gaspard Botté, Séverin Baroudi, Samir Sadok, Francesco Paissan, Thomas Hueber, Xavier Alameda-Pineda, Ricard Marxer, Mirco Ravanelli

arXiv:2609.37798v1eess.AScs.AIcs.LGcs.SD

TL;DR

Speech SSL relies on engineered targets, leaving open whether they are necessary for competitive representations. GLaS-JEPA predicts current-encoder representations at masked positions and uses SIGReg to regularize them; its 57M model improves on listed non-distilled sub-90M baselines in ASR and slot filling.

  • Problem

    Speech SSL approaches use engineered prediction targets, prompting the question of whether they are necessary for competitive speech representations.

    Source: §1 · p. 1

  • Method

    GLaS-JEPA predicts current-encoder continuous representations at masked positions and uses SIGReg to regularize representation space against collapse.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    GLaS-JEPA achieves 6.89% WER and 25.87% slot-filling CER, improving over the best listed non-distilled sub-90M baselines by 43.1% and 22.0%, respectively.

    Source: §5.1 · p. 4

  • Takeaways & Limitations

    At the demonstrated scale, competitive content, speaker, and semantic transfer can emerge from current-encoder prediction without dedicated target generation.

    Source: §1 · p. 2, §6 · p. 4

    Baseline comparisons use different architectures and training budgets, so they cannot isolate the SSL objective’s effect.

    Source: §5.1 · p. 4

Abstract

from arXiv · show

Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.

1. INTRODUCTION

Speech SSL systems use varied engineered targets, raising whether such targets are necessary for competitive speech representations. GLaS-JEPA instead predicts current-encoder representations at masked positions and uses SIGReg to regularize against collapse.

1. INTRODUCTION

  • Speech SSL targets range from quantized or clustered units to EMA-derived continuous representations and soft Gaussian-mixture assignments.These mechanisms shape the prediction signal, and even simplified S-JEPA constructs dedicated targets.Source: §1 · p. 1
  • GLaS-JEPA predicts the current encoder’s continuous representations at masked positions, bypassing discrete assignments and an EMA target encoder.SIGReg regularizes representation space to address the collapse risk of jointly evolving targets.Source: §1 · p. 1
  • The paper evaluates GLaS-JEPA on SUPERB and reports competitive transfer across content, speaker, and semantic tasks with a simpler training pipeline.Its stated contributions include using SIGReg to prevent collapse without target-generation mechanisms.Source: §1 · p. 2

2. RELATED WORK

Prior speech SSL methods construct prediction targets through quantization, clustering, EMA teachers, or other mappings. GLaS-JEPA tests whether current-encoder speech prediction can instead rely on direct representation regularization.

2. RELATED WORK

  • WAV2VEC 2.0 and VQ-WAV2VEC use quantized targets, while HUBERT and WAVLM use clustered pseudo-labels.BEST-RQ retains a dedicated target mapping despite using a fixed random-projection quantizer.Source: §2 · p. 2
  • DATA2VEC 2.0 regresses processed EMA-teacher representations, whereas GLaS-JEPA uses current-encoder representations with distributional regularization.S-JEPA predicts soft GMM posteriors using MFCC features initially and EMA-encoder features [6].Source: §2 · p. 2
  • The paper investigates direct representation regularization for speech latent prediction and excludes numerical comparisons with general-audio JEPA systems because their settings differ [23] [25].The related-work scope also includes non-distilled speech encoders with contrastive, reconstruction, and multi-task objectives [22].Source: §2 · p. 2

3. GLaS-JEPA

GLaS-JEPA uses a shared encoder to predict continuous representations from masked speech, while SIGReg regularizes full-view representations. The paper compares ways of grouping speech tokens for regularization and reports stronger SUPERB scores for Shuffled Marginal than Time-conditional SIGReg in a controlled model.

3.1. Architecture and forward pass

  • A shared encoder processes full and masked feature views, with a token-wise projector mapping each view into a dedicated loss space.The masked view replaces contiguous input spans with a learnable mask token; the projector is discarded after pretraining.Source: §3.1 · p. 2

3.2. Loss function and regularization

  • Masked-view predictions are trained by MSE against full-view targets at masked positions, while SIGReg regularizes unmasked targets to prevent collapse.SIGReg encourages isotropic Gaussian representations without strictly enforcing exact Gaussianity on finite-length speech latents.Source: §3.2 · p. 3

3.3. Applying Distributional Regularization

  • Time-conditional SIGReg regularizes representations at each time index, whereas Marginal SIGReg groups tokens across times and utterances.Shuffled Marginal uses T groups of B tokens in the controlled ablation; Full Marginal applies one penalty to all BT tokens in the main model.Source: §3.3 · pp. 3–4
  • Shuffled Marginal improves ASR WER from 13.99% to 12.24%, ER accuracy from 57.92% to 58.60%, and SD DER from 7.65% to 6.65%.The comparison holds other settings fixed and motivates Marginal SIGReg for the main model.Source: Figure 3 · p. 3, §3.3 · p. 4

4. EXPERIMENTAL SETUP

The experiments use a Conformer encoder pretrained on LibriSpeech and evaluate frozen-encoder transfer on SUPERB. The setup also specifies masking, optimization, and complementary probe evaluations.

4. EXPERIMENTAL SETUP

  • The model uses an eight-layer, width-576 Conformer and 80-bin log-Mel inputs with 20 ms latent spacing, for approximately 57M downstream encoder parameters.Pretraining uses LibriSpeech 960 h [28].Source: §4 · p. 4
  • Training masks 50% of positions in contiguous 200 ms spans and uses 220,000 AdamW updates with Full Marginal SIGReg over all batch tokens.The learning rate peaks at 10^-4, with λ = 0.01; crops span 2–15.6 s.Source: §4 · p. 4
  • For four SUPERB tasks, the encoder is frozen and task heads use a learned weighted sum of hidden layers; ASR uses no external language model.Linear probes assess phone classification and speaker identification, while RankMe measures effective rank.Source: §4 · p. 4

5. RESULTS AND DISCUSSION

GLaS-JEPA achieves strong transfer against listed non-distilled sub-90M baselines, although performance varies by task and the comparisons do not isolate the SSL objective. Probe results and training dynamics show content and speaker information behave differently, while stable current-encoder training at larger scale remains unresolved.

5.1. Downstream transfer

  • 6.89% WER and 25.87% SF concept error rate (CER) improve over the best listed non-distilled sub-90M baselines by 43.1% and 22.0% relatively.Different architectures and training budgets mean these comparisons test target-free competitiveness rather than whether the SSL objective outperforms alternatives under controlled conditions.Source: §5.1 · p. 4
  • SF F1 reaches 87.72%, while ER accuracy (57.91%) trails S-JEPA (64.83%); ASR remains stronger for WavLM (6.21% WER) and data2vec 2.0 (4.81% WER).GLaS-JEPA reaches 6.46% SD DER, slightly below DeCoAR 2.0 (6.59%) and data2vec 2.0 (6.5%), with approximately 39% fewer parameters.Source: §5.1 · p. 4

5.2. Collapse, representations, and limitations

  • Phone and speaker linear-probe accuracy peak at different encoder depths for GLaS-JEPA and WavLM Base.Phone peaks are 83.9/85.4% at layers 6/8 and 11/12; speaker peaks are 96.5/92.7% at layers 3/8 and 4/12, respectively.Source: §5.2 · p. 4, Figure 4 · p. 4
  • Without SIGReg, effective rank falls to 1 within a few thousand updates and phone accuracy is approximately 16% with imbalanced classes.From ∼140k to 220k steps in the same run, ASR WER improved from 7.52% to 6.89%, while ER fell from 59.20% to 57.91%; the passage says this suggests but does not establish an optimization trade-off.Source: §5.2 · p. 4
  • Stable current-encoder training at ∼95M remains ongoing work, and the 57M results do not establish equivalent scaling to Base WavLM, HuBERT, or data2vec 2.0.Source: §5.2 · p. 4

6. CONCLUSION

GLaS-JEPA predicts the encoder’s continuous representations at masked positions and uses representation regularization instead of engineered targets. At the demonstrated scale, it reaches 6.89% ASR WER with competitive transfer across content, speaker, and semantic tasks.

  • GLaS-JEPA replaces engineered target construction with prediction of the encoder’s continuous representations at masked positions.Source: §6 · p. 4
  • At 57M parameters, GLaS-JEPA reaches 6.89% ASR WER with competitive content, speaker, and semantic transfer.Source: §6 · p. 4
  • Competitive speech representations can emerge from current-encoder prediction without dedicated target generation.Source: §6 · p. 4
Loading 2609.37798v1…