PaperlayerView exact revision on arXiv

Source-linked AI summary

WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms

Goksenin Yuksel, Pierre Guetschel, Michael Tangermann, Marcel van Gerven, Kiki van der Heijden

arXiv:2509.23238v3cs.SDeess.AS

TL;DR

Spectrogram-based audio learning has STFT latency and loses phase information, while waveform approaches have been less successful for general-purpose audio. WavJEPA predicts semantic target representations from waveform context, and its multi-channel WavJEPA-Nat extension targets naturalistic scenes. The paper reports stronger benchmark performance and computational efficiency for WavJEPA, and noise and reverberation robustness for WavJEPA-Nat.

  • Problem

    Spectrogram learning entails STFT latency and phase-information loss, while waveform methods successful for speech have been less successful for general-purpose audio representation learning.

    Source: §1 · pp. 1–2

  • Method

    WavJEPA learns semantic waveform representations by predicting latent target representations from temporally distributed context, with WavJEPA-Nat extending the framework to multiple channels.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    WavJEPA surpasses state-of-the-art time-domain approaches on HEAR and ARCH with a fraction of their computational resources, while WavJEPA-Nat exceeds other time-domain models in robustness to noise and reverberation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support the feasibility and efficiency of robust, low-latency time-domain audio representation learning from raw waveforms.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Learning audio representations from raw waveforms overcomes key limitations of spectrogram-based audio representation learning, such as the long latency of spectrogram computation and the loss of phase information. Yet, while self-supervised speech representation learning from raw waveforms has been remarkably successful, these approaches have not achieved similar feats for general-purpose audio representation learning from waveforms. Here, we propose WavJEPA, a waveform-based version of the Joint-Embedding Predictive Architecture. WavJEPA leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources. Additionally, to overcome the performance drop that time-domain models typically exhibit in noisy and reverberant real-world acoustic environments, we present WavJEPA-Nat. WavJEPA-Nat is a multi-channel extension of the WavJEPA architecture trained on simulated naturalistic scenes. We find that WavJEPA-Nat is highly robust to reverberation and noise. These results highlight the feasibility and computational efficiency of general-purpose audio representation learning from raw waveforms, showcasing the potential for low-latency, robust time-domain audio foundation models for real-world applications.

1 INTRODUCTION

Spectrogram-based audio representations incur STFT latency and lose phase information, while waveform methods that work well for speech have been less successful for general-purpose audio and can degrade in noisy, reverberant settings. WavJEPA uses semantic latent prediction from raw waveforms; WavJEPA-Nat extends it to naturalistic scenes, with reported benchmark gains and robustness.

1 INTRODUCTION

  • Spectrogram approaches incur STFT latency and lose phase information, while waveform methods have been less successful for general-purpose audio and can degrade in noisy, reverberant environments.These limitations matter for real-time deployment, generative audio tasks, and real-world applications.Source: §1 · pp. 1–2
  • WavJEPA predicts latent target representations from a temporally distributed context representation of the same waveform, applying semantic learning beyond speech-unit or token-level approaches.The paper positions this as a framework for end-to-end general-purpose audio representation learning from raw waveforms.Source: §1 · p. 2
  • WavJEPA surpasses state-of-the-art time-domain approaches on HEAR and ARCH while requiring a fraction of their computational resources.The paper also reports WavJEPA-Nat exceeds other time-domain foundation models in robustness to noise and reverberation on Nat-HEAR.Source: §1 · p. 2

2 RELATED WORK

Prior general-purpose audio representation work predominantly uses spectrograms, while waveform-based predictive or contrastive methods have chiefly targeted speech units. JEPA-style latent prediction has been applied in other modalities, and WavJEPA brings semantic learning to raw-waveform general-purpose audio.

Spectrogram-based audio representation learning

  • Spectrogram methods learn from time-frequency representations, including masked autoencoders that reconstruct masked patches and latent-prediction approaches that avoid reconstructing the input space.The cited approaches include methods reporting state-of-the-art benchmark performance.Source: §2 · p. 2

Waveform-based audio representation learning

  • Waveform-based predictive or contrastive methods commonly learn speech units, and have been less successful at general-purpose audio representation learning.Data2Vec uses teacher-student latent prediction and achieved state-of-the-art speech recognition with minimal fine-tuning.Source: §2 · p. 2

Representation learning with joint embedding predictive architectures (JEPAs)

  • JEPA models predict latent representations of input parts from context representations, and have also been applied to spectrograms, EEG, and fMRI.Figure 1 depicts WavJEPA predicting waveform target representations from context, with target-encoder weights updated by EMA of the context encoder.Source: §2 · pp. 2–3, Figure 1 · p. 3

3 METHODOLOGY

WavJEPA learns from raw waveforms by predicting latent target blocks from context blocks, and WavJEPA-Nat extends this setup to two-channel naturalistic scenes. The methodology also specifies its pretraining data, benchmark evaluation, and ablation dimensions.

3.1 THE WAVJEPA FRAMEWORK

  • WavJEPA encodes a waveform, samples non-overlapping context and target blocks, predicts target latents from context, and minimizes average prediction error against target-encoder representations.The target encoder is updated by an exponential moving average of context-encoder parameters rather than trained directly.Source: §3.1 · pp. 3–4

3.2 EXPERIMENTAL SET-UP WAVJEPA

  • WavJEPA is pretrained on AudioSet’s 1.74 million 10-second clips, sampling eight 2-second sections per clip; training runs for 375,000 steps on two NVIDIA H100 GPUs.The setup uses an effective batch size of 256.Source: §3.2 · p. 4

3.3 THE WAVJEPA-NAT FRAMEWORK

  • WavJEPA-Nat uses two waveform encoders, shared context and target indices across channels, and 2D positional embeddings to represent intra- and inter-channel information.Its training scenes are spatialized with reverberation and noise, including added spatialized WHAMR! noise clips.Source: §3.3 · pp. 4–5

3.4 DOWNSTREAM EVALUATION

  • Evaluation covers HEAR and ARCH, plus NatHEAR for naturalistic scenes; frozen pretrained representations are assessed with shallow downstream classifiers and a normalized generalizability score.WavJEPA-Nat uses both binaural channels on NatHEAR, while single-channel models use the first Ambisonics channel.Source: §3.4 · p. 5

3.5 ABLATIONS:

  • Ablations examine target and context sampling parameters, top-K target-layer averaging, and the clean-to-naturalistic scene ratio for WavJEPA-Nat.The pretraining and downstream evaluation settings are similar to those used for the respective models.Source: §3.5 · p. 6

4 RESULTS

WavJEPA leads the reported HEAR and ARCH benchmark comparisons, while WavJEPA-Nat improves results across clean and naturalistic scenes. Ablations show that naturalistic-data mix, target-layer averaging, and target sampling choices affect downstream performance.

4.1 PERFORMANCE ON DOWNSTREAM TASKS

  • WavJEPA surpasses the reported state-of-the-art models on HEAR (s(m) = 66.0) and ARCH (s(m) = 92.3).The tables report results across the two benchmark suites.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 7
  • WavJEPA improves state-of-the-art results by 8.9 % on DCASE 2016 Task 2 and 13.8 % on FSD50K, and its environmental sound classification accuracy is 19.1 % higher than WavLM Large.It also performs strongly on acoustic events and audio scene analysis.Source: §4.1 · pp. 6–7
  • WavJEPA performs best on all ARCH music tasks, but reaches state-of-the-art on only one HEAR music task.The paper contrasts ARCH’s general music tasks with HEAR’s niche pitch and percussion tasks.Source: §4.1 · p. 7
  • With 90 m parameters, WavJEPA surpasses other time-domain models on HEAR and ARCH after seeing only a fraction of their pre-training data, and its performance scales with more data.Figure 2 compares performance by samples seen and uses symbol size to indicate model parameters.Source: §4.1 · p. 7, Figure 2 · p. 8

4.2 EVALUATION ON NATURALISTIC SCENES

  • Even when trained on non-naturalistic data, WavJEPA generalizes best to naturalistic scenes (s = 62.1) and scores nearly as well on NatHEAR as on HEAR (∆s = −3.9).It also surpasses other AudioSet-trained Base and Large models on most NatHEAR speech and music tasks, but not WavLM Large on speech tasks.Source: §4.2 · p. 7, Table 3 · p. 8
  • WavJEPA-Nat improves performance on both HEAR and NatHEAR and outperforms the other models on both benchmarks, scoring 60.0 on HEAR and 61.2 on NatHEAR.It was pre-trained with a lower batch size than the original WavJEPA; the authors suggest it could benefit from further upscaling.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3 ABLATION STUDIES

  • As the clean-data ratio increases, WavJEPA-Nat performance falls on both HEAR and NatHEAR; naturalistic-scene pre-training also boosts performance on clean-sound tasks.The comparison varies the clean-to-naturalistic pre-training data ratio.Source: §4.3 · p. 9, Figure 3 · p. 9
  • Top-K target-layer averaging improves all HEAR tasks, but the best K varies by task; some tasks peak at K = 8 and decline at K = 12.Other tasks show no performance difference between K = 8 and K = 12.Source: §4.3 · p. 9

5 DISCUSSION AND CONCLUSION

WavJEPA uses semantic learning to produce general-purpose audio representations from raw waveforms. The authors report strong performance, efficiency, and robustness, while noting that speech performance could improve.

5 DISCUSSION AND CONCLUSION

  • WavJEPA’s results favor semantic audio representation learning over speech-unit or token-level representation learning used in common time-domain approaches.The authors also report robustness to noise and reverberation and say WavJEPA requires only a fraction of the training data used by other time-domain models.Source: §5 · p. 10
  • WavJEPA’s speech representation learning could still be improved compared with Large speech models.The authors plan to investigate pretraining on a combination of sound databases such as AudioSet and speech databases.Source: §5 · p. 10
  • The authors present WavJEPA as opening avenues toward real-time audio foundation models and potential high-quality audio generation, separation, and denoising applications.These are described as potential applications of time-domain audio foundation models.Source: §5 · p. 10

A DETAILED TRAINING SPECIFICATIONS

Table 5 is identified as presenting pre-training specifications.

A DETAILED TRAINING SPECIFICATIONS

  • Table 5 presents pre-training specifications.The supplied passage does not specify the table’s individual entries.Source: Table 5 · p. 15

B DOWNSTREAM EVALUATION METRIC

The paper defines HEAR, ARCH, and Nat-HEAR generalizability scores for a model using task metrics. The score compares gains over baseline with the best available task-level improvement and constrains values to 0–100%.

B DOWNSTREAM EVALUATION METRIC

  • The paper calculates HEAR, ARCH, and Nat-HEAR generalizability scores for each model from task metrics, following a procedure similar to SUPERB.The passage introduces the metric as s(m) for model m and st for task t.Source: B DOWNSTREAM EVALUATION METRIC · p. 15
  • The metric ranks a model’s improvement over baseline relative to the maximum improvement over baseline achieved by the current state-of-the-art for that task.Scores below baseline are set to zero; when the state-of-the-art is below baseline, all models receive zero for that task.Source: B DOWNSTREAM EVALUATION METRIC · p. 15
  • The metric restricts task values to a range of improvement from 0% to 100%.This follows from setting below-baseline scores to zero, including when the state-of-the-art is below baseline.Source: B DOWNSTREAM EVALUATION METRIC · p. 15

C HEAR, NAT-HEAR AND ARCH TASKS

The appendix describes task coverage for HEAR, Nat-HEAR, and ARCH, including dataset and classification metadata for ARCH.

C HEAR, NAT-HEAR AND ARCH TASKS

D WAVJEPA-NAT FRAMEWORK

WavJEPA-Nat extends WavJEPA to multi-channel audio, learning inter- and intra-channel characteristics from naturalistic scenes rendered with room and binaural acoustics. Training scenes pair AudioSet sounds with WHAMR! noise and use BRIRs to model reverberation and spatial audio.

D WAVJEPA-NAT FRAMEWORK

  • The naturalistic-scene set contains 85,000 BRIRs from 85 MatterPort3D houses, with RT60 values ranging from 0.2 to 0.5.Scenes are made by randomly positioning a listener, sound source, and noise source; noise may be localized or diffuse.Source: D WAVJEPA-NAT FRAMEWORK · p. 16
  • WavJEPA-Nat uses a dual waveform encoder to learn inter- and intra-channel characteristics and predicts 2D latent target representations from a 2D context block.The target encoder weights are updated by exponential moving average of the context encoder weights.Source: Figure 4 · p. 17
  • Training uses 70,000 scenes, reserving 15,000 for downstream evaluation, and pairs AudioSet clips with WHAMR! noise at SNRs from +5 dB to +40 dB.For diffuse noise, the rendered noise clips are summed before mixing with the target sound.Source: D WAVJEPA-NAT FRAMEWORK · p. 17

E DETAILED RESULTS ABLATION STUDIES

The section reports ablations of context and training-target sampling procedures, evaluated on the HEAR benchmark.

E DETAILED RESULTS ABLATION STUDIES

  • Table 8 reports downstream HEAR benchmark performance for ablations of context and training-target sampling procedures.The caption identifies italicized entries as modifications relative to the baseline; it supplies no performance values.Source: Table 8 · p. 18
  • The ablation scope is the context and training-target sampling procedure.The caption does not specify the particular variants.Source: Table 8 · p. 18
  • Italicized entries in Table 8 denote modifications from the baseline.The caption does not state which modification performs best.Source: Table 8 · p. 18

F DISTRIBUTION OF TARGET AND CONTEXT SAMPLING

The section describes how sound-wave embeddings are divided between context blocks and training targets.

F DISTRIBUTION OF TARGET AND CONTEXT SAMPLING

  • Table 9 reports the proportions of sound-wave embedding w sampled as context blocks and as training targets.Values are given as averages with 95% confidence intervals.Source: Table 9 · p. 18
  • Each sound-wave embedding w contains, on average, four training targets.The caption does not provide the proportion values.Source: Table 9 · p. 18
  • The reported proportions include 95% confidence intervals.The caption does not state their numerical values.Source: Table 9 · p. 18
Loading 2509.23238v3…