PaperlayerView exact revision on arXiv

Source-linked AI summary

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic

arXiv:2608.19863v1eess.AScs.AIcs.SD

TL;DR

Audio SSL has increasingly depended on elaborate pre-training recipes, while causal next-embedding prediction remains unexplored despite audio’s temporal structure. NAPE trains a causal Transformer to predict successive log-mel spectrogram patch embeddings using causal masking and stop-gradient, and reports strong results across tasks and model scales.

  • Problem

    Audio SSL lacks a causal next-embedding method, despite audio’s intrinsic temporal structure making autoregressive next-patch prediction a natural fit.

  • Method

    NAPE uses a causal Transformer and lightweight predictor to predict each next log-mel spectrogram patch embedding from preceding patches with causal masking and stop-gradient.

  • Results

    Across six audio and speech benchmarks, NAPE achieves strong downstream performance, favorable scaling across three encoder sizes, competitive linear-probing results, and structured attention patterns.

  • Takeaways & Limitations

    NAPE shows that a substantially simpler predictive pre-training recipe can support strong audio representations across downstream evaluation settings.

Abstract

from arXiv · show

Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.

1 INTRODUCTION

NAPE addresses the lack of causal next-embedding audio SSL with a minimalist framework that predicts future spectrogram-patch embeddings from temporal context. Across benchmarks and encoder scales, it reports strong downstream, linear-probing, and structured-representation results.

  • Existing audio SSL methods increasingly rely on reconstruction decoders, acoustic tokenizers, teacher-student setups, auxiliary losses, or vector quantisation.
  • Audio lacks a next-embedding autoregressive SSL method despite its intrinsic temporal ordering making next-patch prediction a natural fit.
  • NAPE linearizes log-mel spectrogram patches and uses a causal Transformer with a predictor head to estimate each next embedding from preceding patches.
  • NAPE’s design is intentionally simple, using causal next-patch prediction while avoiding reconstruction decoders, acoustic tokenizers, and auxiliary training components.
  • Systematic ablations identify scanning order, predictor head, prediction target, patch embedding, causality, prediction shift, and stop-gradient as key design axes.
  • Across six audio and speech benchmarks, NAPE achieves strong downstream results at three encoder scales and competitive linear-probing performance.The tested models contain 19, 85, and 303 million parameters.

2 THE NAPE FRAMEWORK

NAPE trains a causal Transformer to predict the next log-mel spectrogram patch embedding from preceding patches. Its design makes scanning order, causal prediction, stop-gradient, and the prediction head central choices for learning nontrivial audio representations.

  • Input Representation: NAPE converts a log-mel spectrogram into a 2D grid of d-dimensional embeddings from non-overlapping square patches.A patch embedding layer projects each P × P patch into the embedding sequence.
  • Scanning Order: Causal NAPE linearizes the 2D patch grid because ordering determines which patches are visible as past and which remain future targets.The framework evaluates raster, time-major, zigzag, and diagonal scanning orders.
  • Scanning Order: The causal formulation gives scanning order a substantive inductive bias because spectrograms have distinct temporal and frequency axes.This dependence does not arise in the same way under bidirectional attention, where consistent orderings yield equivalent representations given positional embeddings.
  • Prediction Task: At each position t, a causal Transformer and predictor head estimate the next embedding ˆz_t+1 using only z_≤t.A causal attention mask prevents access to later patches, making the task analogous to continuous next-token prediction.
  • Loss Function: NAPE trains with negative cosine similarity between the predicted and target embeddings, applying stop-gradient to the target and averaging across valid positions.Cosine similarity is magnitude-invariant, while stop-gradient sends updates only through the prediction side.
  • Key Components: Causality, prediction shifting, stop-gradient, and an asymmetric predictor jointly define NAPE’s mechanism for avoiding trivial solutions and preserving richer representations.The causal mask blocks identity mapping, while the predictor separates encoder representations from prediction targets.

3 EXPERIMENTS

NAPE’s experiments identify a causal, shifted next-patch prediction recipe whose components are jointly necessary and whose performance scales across model sizes, tasks, and probing settings. The resulting representations support strong downstream performance, accurate embedding prediction, and structured attention patterns.

  • Ablations: Removing prediction shift or stop-gradient causes pre-training to diverge, while removing causal masking sharply degrades downstream performance.Without the shift, the model can copy its input; without stop-gradient, embeddings collapse; without causality, the encoder can attend to the target patch.
  • Ablations: Conv2d patchification outperforms Convstem and Speechstem across all benchmarks, including gaps of 4.8 and 6.0 mAP on AS-20K.The tested patchifiers produce the same sequence length, while Conv2d treats time and frequency as symmetric 2D axes.
  • Ablations: Diagonal, raster, and zigzag scanning perform comparably across six benchmarks, whereas time-major consistently underperforms.The stronger orders advance in time as the causal sequence progresses, allowing temporal context to accumulate.
  • Ablations: Negative cosine similarity with stop-gradient is the default objective, while distance-based alternatives can diverge or underperform.The optimal configuration also uses a three-layer SimSiam-style predictor and patch embeddings as prediction targets.
  • Downstream performance: NAPE improves consistently from Small to Base to Large across all six benchmarks, and raster NAPE-L reaches 50.18 mAP on AS-2M.Against Audio-MAE, NAPE gains +2.6 mAP on AS-2M and +4.1 mAP on AS-20K at Small scale; it ties SSLAM on AS-2M at 50.2 mAP.
  • Linear probing and qualitative results: Linear probing strengthens with model size but remains below fine-tuning, while prediction fidelity is near cosine-similarity ceiling away from boundary regions.The best probing layer is mid-network, and final-layer performance drops by roughly 3-5 mAP points; low-fidelity regions occur at initial, low-frequency, and padded patches.

4 CONCLUSION

NAPE applies causal next patch embedding prediction to audio representation learning with a minimalist pre-training recipe. Across six benchmarks, it delivers strong downstream performance, favorable scaling, competitive linear probing, and structured unsupervised representations.

  • NAPE trains a causal Transformer to predict the next log-mel spectrogram patch embedding using a lightweight predictor and negative cosine similarity with stop-gradient.
  • Across six audio and speech benchmarks, NAPE achieves strong downstream performance while using a substantially simpler pre-training recipe than reconstruction- and masking-based approaches.
  • NAPE scales favorably across Small, Base, and Large encoder sizes and delivers competitive linear-probing results.
  • NAPE develops structured attention patterns and acoustically coherent embedding regions without explicit supervision.
  • The results support autoregressive next-embedding prediction as a simple, scalable, and effective self-supervised objective for audio.

B RELATED WORK

Prior audio representation learning largely adapts supervised or self-supervised paradigms from vision, while predictive learning extends autoregressive next-token and next-embedding objectives across modalities. The supplied related-work material situates NAPE within this progression and contrasts autoregressive and joint-embedding approaches.

  • Supervised Audio Pre-training: Supervised audio pre-training adapts labeled image models to spectrogram inputs, using CNN or Transformer backbones.
  • Self-Supervised Audio Pre-training: Most recent self-supervised audio methods use log-mel spectrograms and masked image modeling, including direct reconstruction of spectrogram patches.
  • Next-token/embedding Prediction: Autoregressive predictive learning progressed from language next-token prediction to visual next-embedding, point-cloud next-token, and next-frame prediction.
  • Next-token/embedding Prediction: Joint-embedding methods predict latent representations of masked regions from context using dual encoders and EMA teachers.
  • The supplied hyperparameter material lists optimizer, scheduler, learning-rate, epoch, batch-size, hardware, sampling, EMA, label-smoothing, and augmentation settings.

C FULL HYPERPARAMETER LIST

The fine-tuning and linear-probing settings differ primarily in learning rate and augmentation, while the supplied passages provide the remaining configuration context through the fine-tuning setup.

  • Fine-tuning Setting: Fine-tuning hyperparameters are reported separately for NEPA Base and Large using the Raster variant, with nearly identical settings for the Diagonal variant.
  • Fine-tuning Setting: NEPA Small uses the NEPA Base fine-tuning hyperparameters except for layer-wise learning-rate decay in some settings.
  • Linear Probing Setting: Linear probing increases the learning rate to 1e−2 and disables all augmentation techniques while keeping other settings unchanged.

D ADDITIONAL ABLATION STUDIES

Additional ablations examine fine-tuning choices, positional encodings, normalization, masking, attention, pooling, scanning, and pre-training budget. The results favor trainable patch embeddings, RoPE, bidirectional mean pooling, no random masking, and temporally advancing scan orders, while larger models benefit from more budget.

  • Freezing the Patch Embedding Layer: Freezing the patch embedding layer is essentially indistinguishable from keeping it trainable across the three tested tasks, so NAPE retains it trainable during fine-tuning.
  • Positional Encoding: RoPE outperforms learned absolute positional encodings by +1.8 mAP on AS-2M and +2.4 mAP on AS-20K.
  • Positional Encoding: RoPE also transfers naturally to clips of different lengths at inference time, whereas absolute encodings require interpolation.
  • Attention Type and Pooling: Bidirectional attention with mean pooling becomes NAPE’s default fine-tuning configuration because bidirectional pooling variants are essentially on par and causal last-token pooling is slightly worse.Causal attention with last-token pooling drops 0.2 mAP on both AS-2M and AS-20K.
  • Normalization Layer: LayerNorm and RMSNorm are effectively on par across the six downstream tasks, so NAPE retains LayerNorm as its default.
  • Random Masking: Random masking has no effect or slightly degrades performance across all six benchmarks, with the largest drops of −1.0 mAP on AS-20K and −1.2 points on ESC-50 at 50% masking.

E NAPE VS JEPA VS LEWORLDMODEL FORMULATION

NAPE differs from JEPA-style methods through a single-encoder, patch-embedding target design and a position-wise prediction head. Compared with LeWorldModel, NAPE uses cosine similarity with stop-gradient and predicts across patches within one input for representation learning.

  • E.1 NAPE VS JEPA-STYLE METHODS: Figure 9 compares the JEPA, LeWorldModel, and NAPE architectures to clarify their structural differences.The comparison covers both JEPA-style prediction and the related LeWorldModel formulation.
  • E.1 NAPE VS JEPA-STYLE METHODS: JEPA-style methods use context and target encoders, with the target updated as an EMA rather than by gradient descent.The predictor jointly receives context and target-position information.
  • E.1 NAPE VS JEPA-STYLE METHODS: NAPE uses a single encoder and takes prediction targets directly from the shared patch embedding layer.The shallow, non-recurrent embedding module supplies stable targets without a separate EMA-tracked encoder.
  • E.1 NAPE VS JEPA-STYLE METHODS: NAPE’s lightweight MLP head predicts one position at a time instead of jointly reasoning over context and target positions with a Transformer.This is a structurally simpler prediction design than the JEPA-style arrangement.
  • E.2 NAPE VS LEWORLDMODEL METHODS: LeWorldModel also predicts next-frame embeddings autoregressively while dispensing with EMA teachers and pretrained encoders through joint end-to-end training.Unlike NAPE, LeWorldModel uses the SIGReg auxiliary regularizer to stabilize learned embeddings.
  • E.2 NAPE VS LEWORLDMODEL METHODS: NAPE uses cosine similarity with stop-gradient to avoid the shrink-to-zero collapse observed with magnitude-sensitive L1 and L2 losses.The loss choice is presented as a difference between NAPE and LeWorldModel’s SIGReg-based stabilization.
  • E.2 NAPE VS LEWORLDMODEL METHODS: LeWorldModel predicts across time steps of action-conditioned trajectories for latent-space planning, whereas NAPE predicts across patches within one input for representation learning.The methods therefore target different problems despite their related autoregressive embedding-prediction structure.

F TRAINING LOSS VISUALIZATIONS

NAPE’s default pre-training loss follows a similar trajectory across model scales, while ablations reveal that causal masking and the chosen loss are essential for avoiding trivial or degenerate solutions.

  • F TRAINING LOSS VISUALIZATIONS: All three default NAPE scales rapidly descend from around −0.3 to below −0.9 with qualitatively similar loss trajectories.The runs cover Small, Base, and Large models using raster scanning, a SimSiam predictor, cosine similarity, and all three key components.
  • F TRAINING LOSS VISUALIZATIONS: Without the causal mask, the Base-scale loss saturates near −1.0 within ∼2,000 steps through a trivial identity mapping.Bidirectional attention enables the objective without learning transferable structure, accompanied by a sharp mAP drop in Table 1.
  • F TRAINING LOSS VISUALIZATIONS: With L1 loss, the Base-scale run drops to ∼0.05, briefly plateaus, then steadily rises as embeddings shrink toward zero.The magnitude-sensitive objective becomes ill-conditioned near this degenerate low-norm regime.

G ADDITIONAL QUALITATIVE RESULTS

The appendix provides additional qualitative visualizations for NAPE-L under raster and diagonal patch orders, alongside broader pre-training loss-curve visualizations across scales and configurations.

  • G ADDITIONAL QUALITATIVE RESULTS: Figure 11 presents additional qualitative results for NAPE-L using raster order.The figure is identified as an additional qualitative-results visualization.
  • G ADDITIONAL QUALITATIVE RESULTS: Figure 12 presents additional qualitative results for NAPE-L using diagonal order.The figure complements the raster-order results with a different patch ordering.
  • G ADDITIONAL QUALITATIVE RESULTS: Figure 10 shows pre-training loss curves for multiple NEPA models across scales and configurations.The passage identifies the figure’s scope but does not state a specific qualitative comparison.
Loading 2608.19863v1…