PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures

Georgios Ioannides, Adrian Kieback, Judah Goldfeder, Linsey Pang, Aman Chadha, Aaron Elkins, Yann LeCun, Ravid Shwartz-Ziv

arXiv:2602.09040v1eess.AScs.AIcs.LGcs.SD

TL;DR

JEPA can collapse on speech without explicit acoustic grounding, while existing clustering approaches may require iterative reclustering. GMM-Anchored JEPA fits a frozen GMM once and uses its soft posteriors with decaying supervision. On approximately 50k hours, it improves downstream tasks and achieves higher cluster entropy than a matched-compute WavLM-style baseline.

  • Problem

    JEPA can suffer representation collapse on speech without external grounding, while HuBERT and WavLM rely on costly iterative reclustering and hard assignments.

    Source: §1 · p. 1

  • Method

    GMM-Anchored JEPA fits a GMM once on log-mel features and uses its frozen soft posteriors as auxiliary targets with a decaying supervision weight.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    GMM anchoring improves ASR, emotion recognition, and slot filling versus matched-compute WavLM-style baselines, with cluster entropy of 98% versus 31%.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Residual GMM supervision provides ongoing regularization that prevents representation collapse after initial grounding.

    Source: §6 · p. 8

    Evaluation uses English speech only and does not compare against published HuBERT or WavLM checkpoints because of architecture and training-data confounds.

    Source: §6 · p. 8

Abstract

from arXiv · show

Joint Embedding Predictive Architectures (JEPA) offer a promising approach to self-supervised speech representation learning, but suffer from representation collapse without explicit grounding. We propose GMM-Anchored JEPA, which fits a Gaussian Mixture Model once on log-mel spectrograms and uses its frozen soft posteriors as auxiliary targets throughout training. A decaying supervision schedule allows GMM regularization to dominate early training before gradually yielding to the JEPA objective. Unlike HuBERT and WavLM, which require iterative re-clustering, our approach clusters input features once with soft rather than hard assignments. On ~50k hours of speech, GMM anchoring improves ASR (28.68% vs. 33.22% WER), emotion recognition (67.76% vs. 65.46%), and slot filling (64.7% vs. 59.1% F1) compared to a WavLM-style baseline with matched compute. Cluster analysis shows GMM-anchored representations achieve up to 98% entropy compared to 31% for WavLM-style, indicating substantially more uniform cluster utilization. Code is made available at https://github.com/gioannides/clustering-anchored-jepa.

1 Introduction

JEPA can collapse on speech without acoustic grounding, while existing speech SSL solutions use costly iterative clustering and hard assignments. The paper proposes frozen, one-time soft GMM anchoring with decaying supervision and reports improvements across downstream tasks and cluster utilization.

1 Introduction

  • JEPA-based speech SSL can suffer severe representation collapse, and unanchored JEPA produces near-random downstream ASR predictions that discard phonetic distinctions.The stated goal is to test whether frozen acoustic anchors stabilize JEPA without iterative re-clustering while maintaining competitive downstream performance.Source: §1 · p. 1
  • HuBERT [16] and WavLM [9] iteratively recluster intermediate representations and retrain, multiplying compute; their hard assignments also discard uncertainty at acoustic boundaries.The proposed alternative fits a GMM once on input features and uses soft posteriors with a decaying weight from λ = 1.0 to λ = 0.01.Source: §1 · p. 1
  • 14% relative WER reduction on LibriSpeech versus a matched-compute WavLM-style baseline accompanies improvements in emotion recognition and slot filling on approximately 50k hours of speech.GMM-anchored representations also reach up to 98% entropy versus 31% for WavLM-style, indicating more uniform cluster use.Source: §1 · p. 1
  • Frozen GMM supervision with residual weight λ = 0.01 is needed throughout training: removing it causes representations to collapse even after initial grounding.The anchor is therefore continuous stabilization, not merely initialization.Source: §1 · p. 2
  • GMM anchoring improves downstream performance across Conformer and Transformer encoders, with cluster entropy up to 98% versus 31% for WavLM-style baselines.The comparison is reported across ASR, emotion recognition, and slot filling.Source: §1 · p. 2

2 Related Work

Speech SSL methods address collapse through contrasting or masked prediction, but clustering-based approaches can require iterative reclustering and discard uncertainty. GMM-Anchored JEPA uses frozen, acoustically informed soft posteriors to ground its representations.

2 Related Work

  • Contrastive methods such as CPC [30] and wav2vec 2.0 [3] distinguish positive from negative samples but require negative-sampling strategies and codebook-diversity losses to avoid collapse.Source: §2 · p. 2
  • HuBERT [16] and WavLM [9] use offline k-means targets and iteratively recluster intermediate layers, multiplying training compute and requiring careful scheduling.JEPA [1] predicts latent representations without reconstruction or negatives but suffers speech representation collapse without external grounding.Source: §2 · p. 2
  • EMA-teacher methods such as BYOL [15], DINO [8], and data2vec [2] use self-distillation, but speech teachers provide temporal smoothing without stable spectral grounding.The paper anchors JEPA with frozen GMM supervision that cannot co-adapt with the encoder.Source: §2 · p. 2
  • Hard k-means assignments discard uncertainty at cluster boundaries, whereas GMM soft posteriors model gradual transitions between phonemes.Like BEST-RQ, the GMM is frozen to avoid iterative retraining, but it supplies acoustically meaningful targets fitted to input features.Source: §2 · pp. 2–3

3 Method

GMM-Anchored JEPA first fits a frozen GMM to log-mel features, then jointly trains a student encoder to predict masked teacher latents and match GMM posteriors. A decaying cluster-loss weight gives way to JEPA training while retaining acoustic anchoring.

3.1 Framework Overview

  • The framework has two phases: fit a GMM once on log-mel spectrograms, then train the student with JEPA prediction and a cluster head matched to frozen GMM posteriors.The GMM is not updated during training.Source: §3.1 · p. 3

3.2 Phase 1: GMM Fitting

  • A diagonal-covariance GMM fitted to log-mel features yields soft assignments, so boundary frames can retain nonzero probabilities across multiple clusters instead of a single hard partition.Log-mel features are used as stable, low-dimensional acoustic representations; diagonal covariance makes fitting tractable while capturing per-frequency variance.Source: §3.2 · p. 3

3.3 Phase 2: Joint Training

  • The cluster-loss weight λ(t) decays linearly from 1.0 to 0.01, shifting from early GMM grounding toward JEPA while residual anchoring prevents drift.KL divergence matches the cluster distributions, and removing the residual anchor causes representation collapse.Source: §3.3 · p. 3

3.4 Denoising Augmentation

  • Denoising augmentation with energy-based mixing gives the online student augmented audio while the delayed-copy target encoder sees clean audio.This setup is intended to encourage denoising in learned representations.Source: §3.4 · p. 3

3.5 Training Algorithm

  • Training uses block masking of 10–25-frame spans until 40–65% of positions are masked, and retains only the student encoder after training.The supplied training-algorithm description specifies these masking bounds and the retained model.Source: §3.5 · p. 3

3.6 Architecture

  • Architectural choices are shared across models; GMM-JEPA differs from the WavLM-style baseline only in using GMM anchoring plus JEPA loss instead of the original WavLM loss.The visualizations compare predicted-cluster-colored frame embeddings and cluster-use distributions; GMM-JEPA variants form distinct regions and maintain flatter utilization than the baselines.Source: §3.6 · p. 4, Figure 2 · p. 4, Figure 3 · p. 4

4 Experiments

GMM-JEPA and GMM-JEPA-T are evaluated against Pure JEPA and a WavLM-style baseline under matched compute. GMM-anchored variants improve downstream results and maintain more stable cluster assignments.

Setup

  • The comparison evaluates Conformer and Transformer encoders trained on approximately 50,000 hours, alongside Pure JEPA and a matched-compute WavLM-style baseline.The WavLM-style baseline uses the authors’ architecture with k-means clustering on log-mel features.Source: §4 · p. 4

Evaluation

  • Downstream evaluation uses frozen encoder representations for ASR on LibriSpeech, emotion recognition on IEMOCAP, and slot filling on SNIPS.The tasks use distinct task-specific models and evaluation setups.Source: §4 · p. 4

Results

  • Figure 4 plots cluster ID per frame above confidence; GMM-JEPA variants show temporally coherent spans, unlike the frequent switching of WavLM-style.GMM-JEPA-T has confidence of 0.7–0.9, while GMM-JEPA has confidence of 0.5–0.8.Source: Figure 4 · p. 5
  • 28.68% WER for GMM-JEPA-T versus 33.22% for WavLM-style on LibriSpeech dev-clean; Pure JEPA reaches 100% WER.The comparison uses matched compute.Source: §4 · p. 4, Table 1 · p. 5
  • 64.7% Slot Type F1 for GMM-JEPA, 5.6% absolute above WavLM-style on SNIPS.The table caption identifies the result as slot filling on SNIPS.Source: Table 2 · p. 5, §4 · p. 5
  • 67.76% average accuracy for GMM-JEPA-T and 67.30% for GMM-JEPA versus 65.46% for WavLM-style on IEMOCAP.Improvements are consistent across all folds.Source: Table 3 · p. 5, §4 · p. 5

5 Analysis

The analysis examines cluster utilization, temporal behavior, embedding quality, and the effect of removing late GMM supervision. GMM-anchored variants show more uniform utilization and stronger downstream performance, while eliminating residual supervision degrades representations and task results.

5 Analysis

  • The analysis extracts frame-level representations from 1,000 utterances and computes cluster entropy and adjacent consistency to assess utilization and temporal smoothness.The evaluation uses 1024 cluster classes.Source: §5 · p. 5
  • GMM-JEPA and GMM-JEPA-T form localized, well-separated embedding regions, while Pure JEPA collapses and WavLM-style clusters overlap diffusely.The UMAP projections visualize these differences in representation space.Source: §5 · p. 6
  • With 1024 clusters, GMM-anchored variants have flatter utilization distributions than Pure JEPA and WavLM-style; GMM-JEPA-T is most uniform.Figure 3 shows the distribution, while Table 4 reports up to 98% entropy versus 31% for WavLM-style.Source: §5 · p. 6, Table 4 · p. 6

5.1 Embedding Quality Analysis

  • GMM-JEPA matches or outperforms WavLM-style in linear probes, including phoneme-classification gains of 5.4% (linear) and 6.7% (MLP).GMM-JEPA-T achieves comparable probe performance, indicating the anchoring benefits transfer across encoder architectures.Source: Table 6 · p. 6, §5.1 · p. 7
  • Pure JEPA has NMI below 0.11 with other models, while successful models have moderate mutual similarity of 0.47–0.57.GMM-JEPA and GMM-JEPA-T have NMI of 0.47, suggesting architecture also shapes representation geometry.Source: §5.1 · p. 7, Table 7 · p. 7
  • GMM-JEPA has the highest Speaker and Gender alignment, while GMM-JEPA-T has the highest phoneme alignment.The authors interpret this as the Conformer emphasizing paralinguistic structure and the Transformer preserving finer phonetic detail.Source: §5.1 · p. 7, Table 8 · p. 7

5.2 Ablation: GMM Anchoring

  • The ablation removes GMM loss late in training by decaying λ to zero instead of 0.01; the resulting clusters become diffuse and flicker more.Figure 5 contrasts this condition with GMM-JEPA at λend = 0.01.Source: §5.2 · pp. 7–8, Figure 5 · p. 7
  • Removing residual GMM supervision drops entropy from 84.7% to 57.7% and utilization from 1011 to 506 clusters, while WER rises from 29.18% to 40.95%.SER accuracy falls from 67.30% to 63.79%, and slot Type F1 falls from 64.7% to 62.3%.Source: §5.2 · pp. 7–8, Table 9 · p. 8

6 Discussion

Frozen GMM supervision grounds JEPA with acoustic cluster posteriors, while a decaying schedule lets the JEPA objective take over. The discussion highlights persistent anchor supervision, an unvalidated soft-assignment hypothesis, and limits on evaluation and tuning.

6 Discussion

  • Early cluster supervision establishes phonetic structure, while residual GMM supervision helps maintain it; removing the anchor causes representations to deteriorate after initial grounding.The supervision weight decays from λ = 1.0 to λ = 0.01.Source: §6 · p. 8
  • Soft GMM assignments may preserve uncertainty at acoustic boundaries and help explain higher cluster entropy, but their advantage over hard assignments remains unvalidated.The reported entropy is 85–98% for GMM-JEPA versus 31% for WavLM-style.Source: §6 · p. 8
  • Evaluation is limited to English speech, and published HuBERT/WavLM checkpoints are not compared because of architecture and training-data confounds.The study focuses on demonstrating effective acoustic grounding rather than state-of-the-art absolute performance.Source: §6 · p. 8
  • The optimal decay schedule and residual weight λend = 0.01 were selected in preliminary experiments but not extensively tuned.Source: §6 · p. 8
  • Future work could test adaptive GMMs, neural density-estimator anchors, multilingual settings, and GMM anchoring beyond speech.Source: §6 · p. 8

7 Conclusion

The conclusion presents frozen GMM supervision as acoustic grounding for JEPA speech representations. Across matched-compute evaluations, it reports gains on three downstream tasks and higher cluster entropy than a WavLM-style baseline.

7 Conclusion

  • GMM anchoring improves ASR, emotion recognition, and slot filling over matched-compute WavLM-style baselines, with 98% versus 31% cluster entropy.Ablations indicate that residual GMM supervision provides ongoing regularization essential for preventing representation collapse.Source: §7 · p. 8

Impact Statement

The impact statement identifies potential applications in accessibility tools and low-resource languages, while noting possible misuse for surveillance or voice profiling.

Impact Statement

  • Improved representations could benefit accessibility tools and low-resource languages.Source: Impact Statement · p. 8
  • Potential misuse includes surveillance or voice profiling; responsible deployment should include consent and privacy protections.Source: Impact Statement · p. 8

A Architecture Details

The architecture combines a convolutional and Conformer encoder with relative-position bias, layer aggregation, density-adaptive attention, and a cluster prediction head.

A.1 Encoder Backbone

  • A 7-kernel input convolution feeds strided convolutional blocks and a four-layer Conformer stack; channel progression reaches 256 with total stride 320.Source: §A.1 · p. 11

A.2 Snake-Beta Activation

  • The Snake-Beta section identifies a learnable parameter a ∈ R1×C×1 for each channel.Source: §A.2 · p. 11

A.3 Gated Relative Position Bias

  • Relative positions use logarithmic bucketing, and a gated mechanism combines learned position embeddings with a gated update.The text specifies 320 buckets and a maximum distance of 800.Source: §A.3 · pp. 11–12

A.4 Layer Aggregation

  • Layer aggregation uses attention over Conformer layers and pooling over the time dimension.Source: §A.4 · p. 12

A.5 Density Adaptive Attention (DAAM)

  • DAAM computes attention from products of Gaussian kernels over normalized activations, then applies it with a residual connection.Inputs are split across heads processed by independent DAAM modules, then concatenated.Source: §A.5 · p. 12

A.6 Cluster Prediction Head

  • A residual multi-layer MLP maps encoder outputs to K cluster logits, using LayerNorm, linear layers, GELU, and dropout in each block.Source: §A.6 · p. 12

A.7 Predictor Architecture

The predictor combines convolutional projections, GELU, and a Conformer block, while an identically structured target encoder is updated by EMA after each gradient step.

A.7 Predictor Architecture

  • The predictor uses Conv1d input and output projections around a GELU activation and a single Conformer block.Both Conv1d projections map C to C.Source: §A.7 · p. 13
  • The Conformer block shares its relative position bias with the encoder.Source: §A.7 · p. 13

A.8 Target Encoder and EMA Update

  • The target encoder has the same architecture as the predictor and is updated by EMA after each gradient step.Source: §A.8 · p. 13, §A.7 · p. 13

B Training Hyperparameters

The appendix specifies training hyperparameters and implementation details for posterior computation, chunked assignments, signal mixing, and visualization. It also identifies an SER ablation table and an augmentor buffer design.

B Training Hyperparameters

C.1 Posterior Computation

  • The posterior computation uses a diagonal-covariance formulation.Source: §C.1 · p. 14

C.2 Chunked Soft Assignment

  • Chunked soft assignment takes features and GMM parameters as input and produces posteriors, using batches and component chunks to avoid out-of-memory errors.The algorithm computes distances and applies softmax to log mixture weights plus log probabilities.Source: §C.2 · p. 14

D.1 Energy-Based SNR Mixing

  • The energy-based SNR mixing procedure specifies an SNR sampled from −5 to 20 dB with probability 0.25.Source: §D.1 · p. 14

D.2 Utterance Mixing

  • Utterance mixing samples start positions in primary and secondary utterances, extracts equal-length regions of length L, computes their energies, and adds a weighted secondary region into the primary, with probability 0.25 and maximum 50% overlap.Source: §D.2 · p. 14

D.3 Augmentor Buffer

  • The augmentor keeps 64 recent utterances on CPU as noise or mixing sources and updates the buffer after each batch.Source: §D.3 · p. 15

E Visualization Details

Loading 2602.09040v1…