CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations

Gabriel Meseguer-Brocal; Yuexuan Kong; Romain Hennequin

Source-linked AI summary

CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations

Gabriel Meseguer-Brocal, Yuexuan Kong, Romain Hennequin

arXiv:2608.30974v1Aug 31, 2026cs.SDcs.AIcs.LGeess.ASeess.SP

TL;DR

JEPA provides rich local latent predictions but typically depends on EMA stabilization, while contrastive learning offers stable global representations with weaker local-task coverage. CoJEPA combines both objectives on a shared backbone, and it outperforms or matches the individual methods across global and local MIR tasks, particularly in harmonic understanding.

Problem
JEPA can be difficult to stabilize and may produce uninformative representations, whereas contrastive learning’s global objective is limited for local MIR tasks.
Method
CoJEPA jointly trains one shared transformer backbone with JEPA on masked sequence tokens and contrastive learning on the class token.
Results
CoJEPA outperforms or matches JEPA and contrastive pre-training across global and local MIR tasks, with a particularly strong advantage in harmonic understanding.
Takeaways & Limitations
CoJEPA indicates that complementary training objectives can guide richer representations without adding backbone parameters or task-specific architectural changes.
Takeaways & Limitations
EMA does not fully prevent collapse or guarantee useful features, and the predictor design choices are not ablated, leaving their effects for future work.

Abstract

from arXiv · show

Joint-Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher--student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence tokens and a contrastive objective on the class token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models.

1 Introduction

Self-supervised learning offers reusable music representations without extensive manual annotation, but its objectives must balance global semantic structure with rich local features. CoJEPA combines contrastive learning and JEPA to stabilize training while improving both representation types.

Motivation: Self-supervised learning derives supervisory signals from data rather than human annotations, enabling reusable representations for downstream tasks.These representations are commonly adapted through probing with lightweight classifiers on frozen backbones.
Motivation: Designing training objectives that produce rich and diverse features remains a central challenge despite SSL’s broad downstream success.SSL has surpassed supervised approaches across a broad set of downstream tasks while requiring fewer annotations.
Limitations of existing objectives: JEPA predicts masked regions in latent space through a teacher–student framework, but EMA stabilization does not eliminate training difficulty, hyperparameter sensitivity, or collapse risk.Contrastive learning is easier and more stable to train, yet its global loss performs worse on local tasks.
CoJEPA: CoJEPA applies contrastive loss to the transformer class token alongside JEPA, using the contrastive objective to regularize the latent space and remove EMA.The method is reported to improve global and local representations over standard JEPA and contrastive pre-training and to remain competitive with MusicFM.
CoJEPA: CoJEPA combines complementary strengths: contrastive learning supplies stable global structure, while JEPA supports local representation learning.The combined method is reported to outperform either individual method on downstream tasks.

2 Literature review

Prior MIR representation learning methods derive signals through joint embedding, masking, equivariance, or hybrid objectives. These approaches offer complementary benefits but retain trade-offs involving global-versus-local transfer, reconstruction targets, transferability, and JEPA stabilization.

Joint embedding and contrastive learning: Joint embedding methods map different views of the same input to nearby shared-space representations, and contrastive learning transfers effectively to global MIR semantics.Because losses are applied to global representations, their local-task coverage is limited.
Masking-based learning: Masking-based methods predict masked input portions from context, encouraging rich information capture and achieving strong results in audio and music.MusicFM and MERT are cited as state-of-the-art examples across a wide range of MIR tasks.
JEPA: JEPA shifts masked prediction from raw input reconstruction to learned latent-space prediction, emphasizing semantic prediction but retaining EMA-related optimization difficulty.The approach has shown strong performance in audio and music domains.
Equivariant SSL: Equivariant SSL aligns representations with known input transformations, but its tight coupling to modeled transformations reduces transferability across diverse downstream tasks.Its effectiveness is therefore more specialized than broadly transferable.
Hybrid objectives: Recent hybrid methods combine multiple SSL principles, including joint embedding with equivariance, contrastive learning with masked prediction, and predictive learning with variance-covariance regularization.The cited JEPA–VICReg combination still requires EMA for training stabilization.

3 Methodology

CoJEPA combines JEPA’s masked latent prediction with contrastive learning on transformer class tokens using one shared backbone. Contrastive learning structures global representations and stabilizes training, while JEPA trains sequence tokens to encode locally predictive information without EMA.

Backbone and roles: CoJEPA uses an updated ViT-1D transformer with a prepended class token and sequence-token representations as its shared backbone.The backbone operates in teacher mode on full sequences and student mode on visible context tokens.
JEPA objective: JEPA masks encoded audio tokens, then predicts their latent representations from visible context tokens and masked-token positional information.A lightweight transformer predictor uses cross-attention from masked positions to context positions.
JEPA objective: JEPA computes its loss only at masked positions by combining L2 distance with cosine similarity between predicted and target tokens.L2 addresses magnitude differences and scale ambiguity, while cosine similarity enforces directional alignment.
Joint training: CoJEPA eliminates EMA by setting teacher and student parameters equal and jointly updating the shared model through the contrastive objective.This design uses the contrastive loss to regularize the backbone and prevent representational collapse.
Contrastive objective: Contrastive learning samples two temporally proximate segments from each track and applies NT-Xent to their class-token representations, using other batch samples as negatives.The class token aggregates sequence information for the global contrastive representation.
Joint training: The combined objective is L = LNT-Xent + LJEPA, with contrastive gradients shaping global embedding structure and JEPA gradients enriching sequence tokens with local predictive information.JEPA targets are detached so its loss does not propagate through teacher mode.

4 Model

The model uses a ViT-1D backbone over mel-spectrogram tokens, with a CLS token for global aggregation and contiguous masking for JEPA prediction. A smaller predictor processes masked-token representations, but its design is not ablated.

Backbone: The input is a 4-second, 16 kHz waveform converted into a 128-bin mel-spectrogram and 126 tokens with embedding dimension 192.Each time frame maps to one transformer token.
Backbone: The ViT-1D backbone has embedding dimension 192, 12 blocks, and 3 attention heads per block, with RoPE, RMSNorm, and gated feed-forward layers.These are the specified backbone architecture and upgrades.
Backbone: The CLS token is a learnable 192-dimensional vector augmented with the mean of projected input tokens and receives the joint-embedding loss.It aggregates global sequence information through self-attention.
Masking: Contiguous temporal masking zeroes all frequency bins in one block, with mask lengths sampled uniformly from 50% to 75% of sequence length.Each sample receives an independently sampled mask length.
Predictor: The predictor uses embedding dimension 192 and six blocks, totaling 3.5M parameters; these design choices are left for future ablation.The passage notes that predictor depth may shape intermediate representations.

5 Experiments

The experiments compare JEPA, contrastive learning, and CoJEPA on global and local MIR tasks using frozen-backbone linear probes at multiple transformer depths. They also report comparisons with the larger MusicFM model under stated dataset and training settings.

Pretraining: Three models are trained: JEPA with standard EMA-based training, contrastive learning, and CoJEPA.This establishes the primary experimental comparison.
Pretraining: The pretraining dataset contains about 2M full tracks curated for musical balance and diversity to improve generalization to unseen data.The dataset is in-house.
Pretraining: All models are trained for 999 epochs with 512 steps per epoch and batch size 128; the JEPA teacher momentum is scheduled from 0.996 to 0.9999.The masking ratio is fixed at 75%, and optimization uses AdamW with cosine schedules.
Downstream tasks: Table 2 reports global-task results at layers 4, 8, and 12, using CLS and mean-pooled sequence tokens for contrastive and CoJEPA, but only mean pooling for JEPA.MusicFM is reported as a single last-layer baseline.
Downstream tasks: Global tasks cover tagging, instrument recognition, pitch, key, and BPM, while local tasks cover beat tracking and chord estimation.Global tasks use sequence-level prediction; local tasks use frame-level prediction.
Evaluation protocol: Linear probes freeze backbone weights and train one linear classifier per task, probing layers 4, 8, and 12 to attribute differences to the training objective.Local probes are applied independently to each sequence token without pooling.
Baselines: MusicFM provides an indicative comparison with a 330M-parameter Conformer model, using only its last transformer layer.The authors acknowledge that intermediate MusicFM layers may perform better.

6 Results

CoJEPA combines JEPA’s local sequence-token learning with contrastive learning’s globally structured embedding, producing complementary representations across musical tasks and depths. It generally outperforms or matches both individual methods, with especially strong gains on tonal and harmonic understanding, despite its small scale.

Overall comparison: CoJEPA achieves the best or tied-best result on most tasks, with the largest benefits on tagging, key, and chords.It remains competitive on rhythmic tasks, where contrastive learning marginally leads BPM and JEPA leads beat tracking.
Layer-wise behavior: JEPA representations peak early and generally degrade with depth on semantic and tonal tasks, while rhythmic performance improves with depth.Pitch drops from .959 to .699, key from .245 to .147, and chords from .211 to .070; BPM accuracy rises from .375 to .833.
Layer-wise behavior: Contrastive representations improve sharply toward the final layer but remain weaker on local tasks than JEPA and CoJEPA.Beat F-measure reaches .737 for contrastive learning at the last layer, compared with .806 for JEPA and .793 for CoJEPA.
Layer-wise behavior: CoJEPA performs best at intermediate depth, reaching its strongest results at layer 8 for most tasks.Layer-8 results include tagging MAP .436, key .640, instruments .952, and beat .793.
Layer-wise behavior: CoJEPA develops tonal representations early but does not preserve all of them at greater depth as contrastive learning increasingly dominates.Pitch and chords peak at layer 4 (.983 and .399), while key peaks at layer 8 (.640).
Token representations: CoJEPA’s class and sequence tokens carry complementary information: the class token favors global tasks, while sequence tokens retain local structure.At layer 8, key performance is .640 with the class token versus .518 with mean-pooled sequence tokens, while sequence tokens are better for BPM and tagging.
Scale comparison: Despite its much larger scale, MusicFM leads CoJEPA only on tagging and beat tracking and matches it on pitch.The comparison uses a 7.1M-parameter CoJEPA model versus MusicFM at 330M parameters.

7 Conclusion

CoJEPA unifies JEPA and contrastive learning on a shared backbone, using complementary objectives to improve global and local music representations without adding backbone parameters. It outperforms or matches both individual methods across tasks, with particularly strong harmonic performance, while its richest representations emerge at intermediate depth.

Conclusion: CoJEPA combines JEPA’s local sequence-token objective with contrastive learning’s globally discriminative class-token objective on one shared backbone.The contrastive gradient prevents collapse without an EMA teacher, while JEPA enriches sequence tokens through local prediction.
Conclusion: CoJEPA outperforms or matches JEPA and contrastive learning across global and local tasks, with a particularly strong advantage in harmonic understanding.The model achieves this despite its small 7.1M scale.
Conclusion: CoJEPA’s richest representations emerge at intermediate depth, unlike the depth patterns observed for JEPA or contrastive learning alone.The framework guides richer representations through training-signal design rather than additional backbone parameters.