PaperlayerView exact revision on arXiv

Source-linked AI summary

S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning

Georgios Ioannides, Adrian Kieback, Judah Goldfeder, Linsey Pang, Aman Chadha, Aaron Elkins, Yann LeCun, Ravid Shwartz-Ziv

arXiv:2606.19398v1cs.SDeess.ASeess.SP

TL;DR

Hard cluster labels collapse acoustic ambiguity at category boundaries, and offline re-clustering interrupts training. S-JEPA trains a JEPA-style encoder-predictor to match soft GMM posteriors at masked positions; it reports the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at 55% of its parameter count.

  • Problem

    Hard cluster IDs at masked positions collapse acoustic ambiguity at category boundaries and require interrupting training to re-cluster the full corpus.

    Source: §1 · p. 1

  • Method

    S-JEPA matches soft GMM posteriors at masked positions via KL divergence, using a fixed MFCC-feature GMM followed by online encoder-feature updates and adaptive input-layer selection.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    S-JEPA is the lowest-WER SSL method below 90M parameters and matches HuBERT-Base on emotion recognition at 55% of its parameter count.

    Source: §6 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    Held-out predictor entropy shows a substantial minority of frames retain two-way inter-cluster ambiguity that hard-target objectives could not represent.

    Source: §7 · p. 8, §6 · p. 8

    Training and evaluation are in English; extension to tonal languages, low-resource regimes, and non-speech audio remains open.

    Source: §6 · p. 8

Abstract

from arXiv · show

Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.

1 Introduction

S-JEPA replaces hard masked cluster labels and stop–restart re-clustering with soft GMM targets in a continuous training pipeline. It reports strong SUPERB results below 90M parameters and held-out entropy evidence of two-way cluster ambiguity.

1 Introduction

  • Hard cluster assignment collapses acoustic ambiguity at category boundaries, while offline re-fitting interrupts training to recluster the full corpus between iterations.Frames at phone boundaries, transitions, and silence/non-silence shifts are forced into one partition.Source: §1 · p. 1
  • At 51.8M parameters, S-JEPA reaches 12.10% WER on LibriSpeech test-clean and 64.83% emotion accuracy, matching HuBERT-Base at 55% of its parameter count.With 4-gram LM rescoring, LibriSpeech test-clean WER is 8.50%; across three SUPERB tasks, S-JEPA dominates evaluated SSL methods below 90M parameters.Source: §1 · pp. 1–2
  • On 153K held-out frames, predictor entropy is bimodal, with a secondary mode near 1 bit representing two-way ties between competing clusters.Over a third of frames exceed 1 bit of entropy, retaining inter-cluster mass that hard targets would discard.Source: §1 · pp. 1–2
  • S-JEPA matches soft GMM posteriors at masked positions with KL divergence, replacing HuBERT’s hard k-means labels and cross-entropy objective.Its two-phase pipeline uses a fixed GMM over MFCC features, then online minibatch GMM updates with adaptive input-layer selection and alternating EMA decay.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

S-JEPA combines JEPA-style masked prediction with soft categorical targets and online GMM updates. Its distinction from related approaches is the continuous update of soft targets without teacher distillation or an auxiliary latent-regression objective.

2 Related Work

  • HuBERT [20] and WavLM [9] use offline cluster IDs for masked prediction, while contrastive methods and EMA-teacher methods use different target or loss formulations.S-JEPA follows the JEPA architecture [1] but uses soft categorical targets rather than hard cluster IDs or continuous regression targets.Source: §2 · p. 2
  • Unlike HuBERT’s offline hard k-means or BEST-RQ’s frozen random quantizer [10], S-JEPA keeps iterative clustering and updates its GMM online from evolving encoder features.Compared with prior work [21], it uses online encoder-feature GMM updates, a single soft-posterior target without latent regression or a λ schedule, and adaptive layer selection.Source: §2 · pp. 2–3
  • DistilHuBERT, FitHuBERT, and LightHuBERT compress a roughly 95M-parameter HuBERT teacher, whereas S-JEPA reaches the smaller parameter regime through pretraining without a teacher.The distillation methods retain HuBERT pretraining as a prerequisite; the student’s parameter count describes inference-time cost, not the cost of producing it.Source: §2 · p. 3
  • The contribution is a single-pass speech SSL recipe combining soft GMM posteriors, online updates, and JEPA-style masked prediction, not a new clustering method, prediction architecture, or backbone.Soft cluster supervision via Gaussian mixtures matched by KL divergence [38] is classical.Source: §2 · p. 3

3 Method

S-JEPA pairs a masked-frame encoder–predictor objective with soft GMM targets and a continuous two-phase training pipeline that updates and selects encoder-feature targets online.

3.1 Architecture

  • After training, S-JEPA retains only the encoder; a predictor and shared cluster head support masked-frame predictions during training.The predictor uses a learned mask token and positional embeddings, and only its masked-position outputs contribute to the loss.Source: §3.1 · p. 3

3.2 Soft Targets and KL Loss

  • At each frame, a GMM posterior over acoustic features supplies a soft target, and KL divergence matches it to the predictor’s output over selected positions.The GMM features come from MFCCs in Phase 1 or the EMA encoder in Phase 2; the loss set is masked-only or masked plus visible, depending on stage.Source: §3.2 · p. 3

3.3 Two-Phase Training

  • Training runs as one continuous trajectory: Phase 1 uses a fixed MFCC GMM, then Phase 2 initializes an encoder-feature GMM and transitions from masked-plus-visible to masked-only loss.Phase 1 uses denoising augmentation; Phase 2 begins with the same loss and augmentation configuration before its transition.Source: §3.3 · pp. 3–4, §3.4 · pp. 4–5
  • Online minibatch GMM updates track evolving EMA-encoder features without a separate corpus-scale relabeling pass.Responsibility-weighted feature means and variances update the GMM parameters, with no gradient flowing through the GMM.Source: §3.3 · p. 4
  • The GMM input layer is selected by the maximum effective rank, a label-free proxy for representational richness; in this run, the active layer switched from 2 to 4.Switching to layer 4 was correlated with continued development-set WER progress, while the authors explicitly do not claim that higher effective rank causes better performance.Source: §3.3 · p. 4
  • Phase 2 periodically switches EMA decay rates to let the GMM track encoder improvements while limiting target-distribution drift.The faster rate tracks the online encoder more closely; the slower rate approximately holds the vocabulary stable while the online encoder learns.Source: §3.3 · p. 4

4 Experiments

S-JEPA was pretrained on approximately 83,000 hours and evaluated with a frozen encoder on SUPERB and LibriSpeech probing tasks. It establishes a sub-90M SUPERB Pareto frontier and shows strong linear-probe results, with phoneme-probe rankings depending on probe type.

4 Experiments

  • Pretraining used approximately 83,000 hours of English speech, with a 51.8M-parameter six-layer encoder evaluated frozen on three SUPERB tasks.Evaluation followed the SUPERB protocol, training small task-specific heads on top of the frozen encoder.Source: §4 · p. 5

4.1 Frozen-Encoder Results on SUPERB

  • Below 90M parameters, S-JEPA dominates 12 evaluated methods on all three SUPERB tasks, establishing the sub-90M Pareto frontier.Its 12.10% WER compares with 13.37% for DistilHuBERT and 13.02% for DeCoAR 2.0; ER accuracy is 64.83% versus 63.02% for DistilHuBERT.Source: §4.1 · p. 5, Table 1 · p. 5
  • On slot filling, S-JEPA’s 83.05 F1 is essentially tied with DeCoAR 2.0’s 83.28 at 58% of its parameter count, while Base-class models lead by 5.5–7.5 F1 points.Its 64.83% emotion-recognition accuracy is essentially tied with HuBERT-Base’s 64.92% at 55% of HuBERT-Base’s parameter count.Source: §4.1 · pp. 5–6

4.2 Probing Frozen Representations

  • Linear probes favor S-JEPA on Speaker ID, Gender, and Chapter ID; phoneme classification favors WavLM linearly (86.1 vs. 82.8) but S-JEPA with an MLP (88.0 vs. 87.5).Chapter ID is the most discriminative comparison because chapters within a book are typically read by the same speaker, making prosodic and recording variation the subtler cue.Source: §4.2 · p. 6, Table 2 · p. 6

5 Analysis

The analysis examines predictor posteriors on a held-out utterance and measures entropy across sampled speech frames. It finds confident predictions on many frames alongside substantial two-way cluster uncertainty on others.

5.1 Per-utterance Interpretability

  • On a held-out utterance, rank-1 posteriors are near 1.0 in phonetically stable regions but fall to 0.3–0.45 near word boundaries as rank-2 competes.At 20 ms resolution with 500 clusters, cluster IDs change at sub-phonemic rates; Whisper word alignments serve only as a visual reference.Source: Figure 4 · p. 7, §5.1 · p. 7

5.2 Aggregate Distribution of Predictor Uncertainty

  • The per-frame entropy distribution is bimodal, with a confident regime below 0.3 bits, a secondary mode near 1 bit, and 36.6% of frames above 1 bit.One bit corresponds to a perfect two-way tie between clusters; the analysis interprets these frames as retaining inter-cluster mass that hard targets would discard.Source: §5.2 · p. 7, Figure 4 · p. 7

6 Discussion

S-JEPA combines soft GMM targets with online, adaptive training and reports strong low-parameter benchmark results. Its held-out entropy analysis finds substantial two-cluster ambiguity, while the discussion identifies limits in evaluation scope and EMA-schedule comparisons.

6 Discussion

  • S-JEPA combines soft GMM posteriors with JEPA-style masked prediction, eliminating offline re-clustering and hand-selecting the transformer layer for clustering.The discussion frames the method as preserving boundary ambiguity that hard assignments collapse.Source: §6 · p. 7
  • S-JEPA is the lowest-WER SSL method below 90M parameters and matches HuBERT-Base on emotion recognition at 55% of its parameter count.The paper reports these as its two strongest empirical findings.Source: §6 · p. 7
  • 36.6% of predictor posteriors are bimodally structured around a two-way-tie regime that hard targets could not represent.The authors say a controlled, compute-matched hard-target ablation is a natural next step.Source: §6 · p. 8
  • Training and evaluation are English-only; tonal languages, low-resource regimes, and non-speech audio remain open extensions, and the switched EMA schedule lacks a controlled comparison.The relationship between GMM and SIGReg in JEPA formulations is also left for further study.Source: §6 · p. 8

7 Conclusion

S-JEPA uses soft GMM posteriors for masked prediction in a continuous two-phase training pass. With a 51.8M-parameter encoder, it reports the lowest WER among SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at 55% of its parameter count.

7 Conclusion

  • S-JEPA matches GMM posteriors to masked-position predictions with KL divergence, using online GMM updates, adaptive layer selection, and periodically switched EMA decay in one continuous pass.The training phases use a fixed MFCC GMM followed by an online encoder-feature GMM.Source: §7 · p. 8
  • With 51.8M parameters, S-JEPA is the lowest-WER SSL method below 90M parameters on LibriSpeech test-clean and matches HuBERT-Base on emotion recognition at 55% of its parameter count.Source: §7 · p. 8
  • Held-out predictor entropy has a bimodal structure, with a substantial minority of frames in a two-way-competition regime whose inter-cluster ambiguity hard targets cannot represent.Source: §7 · p. 8

A Architecture Details

The architecture pairs a six-layer encoder with a training-only predictor and cluster head. Masked frames are hidden from the encoder and receive a learned mask token in the predictor, while the two-phase algorithm uses fixed then online GMM targets.

A Architecture Details

  • The encoder has six Transformer layers; the predictor and cluster head are used during training and discarded afterward.Source: A Architecture Details · p. 11

A.1 CNN Feature Extractor

  • A seven-layer 1D CNN maps 16 kHz audio to 512-channel frames at 20 ms intervals, with frontend gradients scaled by 0.1.Its kernel sizes and strides match HuBERT; the first layer uses fp32 group normalization.Source: §A.1 · p. 11

A.2 Transformer Stack

  • The six-layer Transformer uses 768-dimensional states, 12 attention heads, a 3072-dimensional feed-forward layer, and convolutional positional encoding.LayerDrop is 0.05; attention, residual, and activation dropout are each 0.1.Source: §A.2 · p. 11

A.3 Encoder Mask Handling

  • The encoder zeros masked frames before positional encoding and Transformer processing; the predictor injects the mask token later.Source: §A.3 · p. 11

A.4 Predictor and Cluster Head

  • The predictor adds a learned 768-dimensional mask token and positional embeddings, then a Transformer layer and linear projection produce features for the cluster head.The shared cluster head is a three-layer MLP; it uses K=100 in Phase 1 and K=500 in Phase 2, with KL supervision on masked positions.Source: §A.4 · p. 11

A.5 EMA Encoder

  • An EMA copy of the encoder supplies Phase 2 GMM input features at the active layer, with its decay rate periodically switched between two values.The EMA encoder is unused in Phase 1.Source: §A.5 · p. 11

A.6 Masking

  • Training samples contiguous temporal mask spans, predicts GMM posteriors at masked frames with KL loss, then updates the EMA encoder and switches its decay in Phase 2.The algorithm uses a fixed MFCC GMM in Phase 1 and an online encoder-feature GMM with adaptive layer selection in Phase 2.Source: §A.6 · pp. 11–12

B Training algorithm

The training procedure transitions Phase 2 to masked-only loss and no augmentation. WER continued to decrease afterward, but the two changes were not independently ablated.

C Phase 2 Transition: Masked-Only Loss and Augmentation Off

  • The Phase 2 transition simultaneously removed visible-position loss and disabled denoising augmentation, guided by downstream WER on a held-out development set.The algorithm is summarized in the appendix; the transition passage describes the changes and their motivation.Source: C Phase 2 Transition: Masked-Only Loss… · p. 12
  • Visible-position predictions were considered easier and potentially redundant with masked-position loss; disabling augmentation made the encoder and EMA GMM use clean audio.The authors suggest the clean-target and augmented-input mismatch may make masked prediction harder at that training stage.Source: C Phase 2 Transition: Masked-Only Loss… · p. 12
  • After both changes, downstream WER continued to decrease and the adaptive layer mechanism settled at layer 4.The authors did not run a controlled ablation toggling each change independently.Source: C Phase 2 Transition: Masked-Only Loss… · p. 12

D Training Hyperparameters

The appendix specifies implementation details for the two-phase GMM pipeline, including posterior computation, fitting and online updates, EMA scheduling, and adaptive layer selection.

D Training Hyperparameters

E.1 Posterior Computation

  • The diagonal-covariance GMM posterior is computed in closed form, with variances floored at 10^-6 for numerical stability.Source: §E.1 · p. 13

E.2 Phase 1 GMM Fitting

  • The Phase 1 GMM is fit once on 39-dimensional MFCC features and then held fixed.The fitting pipeline uses reservoir sampling, five mini-batch k-means iterations for initialization, and 20 EM refinement iterations.Source: §E.2 · p. 14

E.3 Online GMM Updates (Phase 2)

  • In Phase 2, GMM means, variances, and weights are updated online from minibatch responsibility-weighted statistics computed on EMA-encoder features.No gradient flows through the GMM.Source: §E.3 · p. 14

E.4 Chunked Soft Assignment

  • For large cluster counts and batches, posterior computation is chunked over clusters to bound peak memory.Source: §E.4 · p. 14

F Periodically Switched EMA Decay

  • The target encoder alternates between EMA decays of 0.999 and 0.9999, switching on average every 20,000 steps.The fast and slow rates have 1,000- and 10,000-step characteristic timescales; the online GMM tracks more recent encoder outputs during fast intervals and approximately stationary outputs during slow intervals.Source: F Periodically Switched EMA Decay · pp. 14–15

G Effective Rank and Adaptive Layer Selection

  • The GMM input layer is selected by comparing EMA-smoothed effective-rank scores across transformer layers, with a changed selection taking effect at the next minibatch.Layer scores use centered samples of 2,000 embeddings and singular values; switching causes no other training discontinuity.Source: G Effective Rank and Adaptive Layer… · p. 15

H Probing Protocol

The appendix defines frozen-feature probing procedures and describes waveform augmentation, including target-SNR noise mixing and energy-weighted utterance mixing.

H Probing Protocol

  • Probes use frozen encoder features from S-JEPA, HuBERT-Base, and WavLM-Base, with linear or MLP heads and task-specific frame- or utterance-level features.Evaluation uses random LibriSpeech dev-clean splits, with ten splits for utterance tasks and three for phoneme classification.Source: H Probing Protocol · p. 15
  • The protocol covers speaker ID, gender, chapter ID, and frame-level classification across 67 phonemes.Chapter identification can require recording-session and prosodic cues beyond speaker identity.Source: H Probing Protocol · p. 15

I Augmentation Details

I.1 Energy-Based SNR Mixing

  • Noise is mixed with the clean signal at a target SNR using an energy-based scheme.Source: §I.1 · p. 16

I.2 Utterance Mixing

  • Utterance mixing adds a scaled segment from another utterance, with the scale based on a sampled energy ratio.Source: §I.2 · p. 16

J SUPERB Evaluation Protocol

The evaluation follows SUPERB: the pre-trained encoder is frozen, and small task-specific heads are trained on top.

J SUPERB Evaluation Protocol

  • Under the SUPERB protocol, the encoder receives raw waveforms and task heads use frame-level outputs or a learned weighted combination of encoder layers, depending on the task.Source: J SUPERB Evaluation Protocol · p. 16
Loading 2606.19398v1…