Source-linked AI summary
Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?
Yutong Feng, Bowen Liao, See Kiong Ng, Yuxuan Liang
TL;DR
Evidence on JEPA's forecasting benefits is mixed, and prior evaluations have often covered only a narrow range of backbones. This paper evaluates one JEPA instantiation across nine backbones and eleven datasets, finding that its effect varies sharply by backbone across both temporal and spatio-temporal forecasting.
Problem
Prior time-series JEPA evaluations report inconsistent benefits and often test only one backbone or a narrow architecture family, leaving whether effectiveness depends on the downstream model unresolved.
Source: §1 · p. 1Method
The paper evaluates one JEPA instantiation across nine backbones and eleven temporal and spatio-temporal forecasting datasets.
Source: §1 · p. 1Results
JEPA pretraining yields consistent gains for some architectures and consistent degradation for others across both task families.
Source: §6 · p. 4Takeaways & Limitations
JEPA pretraining should be adopted with the backbone choice in mind rather than assumed to help by default.
Source: §6 · p. 4The study establishes that variability exists and matters but does not identify its mechanism.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Joint-embedding predictive architectures (JEPA) have emerged as a promising self-supervised pretraining paradigm for time series, learning representations by predicting target embeddings in latent space rather than reconstructing raw signals. Yet evidence on their benefits remains mixed, and most studies test only a single backbone or a narrow set of architectures, leaving unclear whether JEPA pretraining is a reliable improvement or one that depends heavily on the downstream model. We address this gap through a large scale evaluation of one JEPA instantiation across nine backbones and eleven benchmarks spanning temporal and spatio-temporal forecasting, the most extensive cross architecture assessment of JEPA for time series to date. We find that the benefit of this instantiation varies sharply across backbones, producing consistent gains for some architectures and consistent degradation for others, even on the same dataset. This pattern holds across both task families, indicating the variability is a general property of this instantiation rather than a dataset specific artifact worth accounting for when choosing a backbone in practice.
1. INTRODUCTION
JEPA offers a latent-space alternative to time-series self-supervision, but evidence of its forecasting benefits is mixed and prior evaluations span few architectures. This paper evaluates one JEPA instantiation broadly and finds that its effect depends sharply on the backbone.
1. INTRODUCTION
- JEPA predicts target embeddings from context embeddings in latent space, avoiding raw-signal reconstruction and negative pairs.The introduction contrasts this with masked autoencoding and contrastive approaches.Source: §1 · p. 1
- Prior evaluations often covered only one backbone or a narrow architecture family, leaving uncertain whether JEPA is dependable or backbone-dependent.This gap matters because time-series backbones range from convolutional and graph-based encoders to attention-based transformers.Source: §1 · p. 1
- Earlier reports on JEPA for time series were inconsistent, ranging from clear gains [11] to negligible benefit or degradation [13] [14].Recent work has applied the paradigm to temporal forecasting and spatio-temporal learning [10].Source: §1 · p. 1
- Across nine backbones and eleven datasets, JEPA pretraining ranges from consistent gains to consistent degradation, with this pattern stable across temporal and spatio-temporal forecasting.The paper also distills practical guidance on when JEPA pretraining helps and when practitioners should be cautious.Source: §1 · p. 1
2. INSTANTIATING
The paper adapts JEPA to forecasting by pairing historical context windows with future target windows and applying a consistent encode-predict-decode pipeline to compatible backbones. It uses a shared encoder and existing backbone components where possible, while excluding architectures without a meaningful decomposition.
2. INSTANTIATING
- Historical input and future forecast windows form the context and target views, reusing the forecasting input-label pairs without an added masking scheme.Figure 1 depicts the instantiation for backbones.Source: §2 · p. 2, Figure 1 · p. 2
- A single shared encoder represents both views; LeJEPA uses SIGReg rather than stop gradients or teacher-student asymmetry, so no separate target encoder is needed.The encoder is typically the backbone's existing hidden representation before its final output layer.Source: §2 · p. 2
- The predictor maps context representations to target representations, reusing an existing projection when available; the decoder maps predictions to value space only during finetuning.Because the forecast target follows context at a fixed offset, separate relative-position conditioning is unnecessary.Source: §2 · p. 2
- Backbones are included only when encoder and decoder boundaries and representation shapes are unambiguous; the study excludes architectures that fail this decomposability criterion.The authors avoid adding layers solely to create an arbitrary representation-space split.Source: §2 · p. 2
- The two-stage instantiation pretrains with encoding and prediction, then finetunes with decoding and a supervised forecasting loss kept identical to the baseline.This keeps the finetuning stage and forecasting loss consistent across backbones.Source: §2 · p. 2, Figure 1 · p. 2
3. EXPERIMENTAL SETUP
The evaluation compares JEPA-pretrained and supervised-only versions of nine compatible backbones across eleven public temporal and spatio-temporal forecasting benchmarks, with matched downstream training conditions.
3. EXPERIMENTAL SETUP
- The evaluation covers five temporal and six spatio-temporal forecasting benchmarks, using standard train, validation, and test splits.Temporal datasets are ETTh1, ETTh2, ETTm1, ETTm2, and Weather; spatio-temporal datasets are METR-LA, PEMS-BAY, PEMS03, PEMS04, PEMS07, and PEMS08.Source: §3 · p. 3
- Nine attention-based and graph-based backbones satisfy the decomposability criterion and receive JEPA pretraining; purely linear forecasters were excluded during backbone selection.Source: §3 · p. 3
- The two conditions differ only by JEPA pretraining: total epochs are matched, and finetuning loss, stage, and downstream hyperparameters are identical.Both pretraining and finetuning run for at most 50 epochs with validation-loss early stopping; context and horizon lengths are 128.Source: §3 · p. 3
4. EXPERIMENT RESULTS
JEPA's forecasting effect varies by backbone in both task families: some improve across datasets while others degrade, and several show mixed outcomes.
4. EXPERIMENT RESULTS
- Crossformer and FEDformer improve on both metrics across every temporal dataset, while iTransformer degrades on both metrics for four of five datasets.Autoformer and PatchTST have mixed outcomes, including Autoformer's largest negative temporal effect on Weather: 15.54% in MAE and 12.00% in RMSE.Source: §4 · p. 3
- In spatio-temporal forecasting, STGCN improves on both metrics for five of six datasets, while ASTGCN and STAEformer degrade on both metrics for most datasets.STTN improves on four of six datasets and degrades on the other two.Source: §4 · p. 3
- The spread between backbones is substantially wider in spatio-temporal than temporal forecasting.The reported results show both consistent favorable patterns and mixed or unfavorable patterns within each task family.Source: §4 · p. 3
5. ANALYSIS & DISCUSSION
JEPA’s effects track backbone architecture, but the observed architecture-type pattern is not a confirmed explanation. Across both forecasting families, the performance swing can change which model appears preferable in practice.
Practical implications
- Across both task families, relative improvements differ by more than 25 percentage points, and JEPA can reverse which model appears best on a benchmark.STGCN gains consistently while Autoformer loses over 15% on Weather; the authors advise checking the effect on each architecture.Source: §5 · p. 3, Table 1 · p. 4, Table 2 · p. 4
A pattern related to architecture type
- Dense global self-attention backbones tend toward negative or negligible gains, while structurally constrained mixing tends toward positive gains; the split is imperfect and observational.The authors report this pattern in both task families but say confirming it requires experiments.Source: §5 · p. 3
Results
- Table 1 reports temporal forecasting performance, while Table 2 reports spatio-temporal forecasting performance.Table 1 gives mean ± standard deviation over three seeds, marks relative improvement by color, and states that lower values are better.Source: Table 1 · p. 4, Table 2 · p. 4
Discussion
- One proposed explanation is that backbones differ in how well they capture predictive structure shared across context and target windows, but this remains a hypothesis.The paper calls for targeted ablations rather than treating this as a confirmed mechanism.Source: §5 · p. 4
6. CONCLUSION & FUTURE WORKS
Across nine backbones and eleven datasets, JEPA’s effect depends heavily on the backbone and can range from gains to degradation. The paper establishes that this variability matters, while leaving its mechanism for future study.
Conclusion
- Across nine backbones and eleven datasets, JEPA’s effect depends heavily on the backbone, with consistent gains for some architectures and degradation for others.The pattern holds across temporal and spatio-temporal forecasting and can change which model is selected in practice.Source: §6 · p. 4
Future works
- The study establishes that variability exists and matters but does not identify its mechanism.The proposed link to architecture type remains to be tested by varying inductive bias within one backbone family.Source: §6 · p. 4
- Future work could ablate the predictor and SIGReg separately to test whether variability comes from the pretraining objective or backbone encoder-decoder boundaries.The paper recommends treating JEPA as backbone-dependent rather than assuming it helps by default.Source: §6 · p. 4