Source-linked AI summary
Distributed JEPA: A Self-Supervised Framework for Energy Forecasting
Liana Toderean, Tudor Cioara, Vasilis Michalakopoulos, Efstathios Sarantinopoulos, Ionut Anghel, Elissaios Sarmas
TL;DR
Energy forecasting lacks broadly transferable representations across heterogeneous assets because specialized and reconstruction-focused approaches can emphasize asset-specific or low-level patterns. The paper proposes distributed JEPA, which predicts masked temporal representations while integrating temporal and contextual information in a shared embedding space. JEPA produced stable, diverse representations and matched or exceeded a Transformer across building and photovoltaic forecasting, with particular robustness to missing data.
Problem
Specialized energy forecasting models have limited transferability across heterogeneous assets, while reconstruction objectives may favor low-level features over general temporal dynamics.
Method
Distributed JEPA learns latent representations of masked temporal segments while integrating temporal observations and contextual information in a unified embedding space.
Results
JEPA achieved performance comparable to a Transformer on building energy data and outperformed it across most photovoltaic clients while producing stable, diverse representations.
Takeaways & Limitations
Latent-space prediction can provide transferable representations for heterogeneous energy time-series without directly reconstructing observations.
Takeaways & Limitations
Performance degraded on the low-variability SQ567 Flexible PV site, and evaluation covered only building consumption and photovoltaic generation datasets.
Abstract
from arXiv · showhide
Traditional energy forecasting solutions rely on task-specific supervision and energy asset representations, limiting transferability and the ability to capture general temporal dynamics across heterogeneous assets. We address this by proposing a distributed Joint Embedding Predictive Architecture (JEPA) for self-supervised learning from heterogeneous energy time-series. The framework predicts latent representations of masked temporal segments while integrating temporal observations and contextual information within a shared embedding space. To prevent representation collapse, training combines a latent-space predictive objective with covariance and temporal variance regularization. The evaluation was conducted on energy consumption and generation datasets under data-degradation scenarios and compared with a Transformer forecasting baseline. The learned representations remained stable (cosine similarity $\approx 0.98$; effective rank 185-235). JEPA achieved performance comparable to a Transformer on building energy data, higher $R^2$ in 3/5 consumer clusters, and outperformed the baseline on 9/10 unseen PVs ($R^2$=0.73-0.88 vs. <0.45), while showing greater robustness to missing data.
1. Introduc+on
Energy forecasting needs transferable representations because specialized models struggle to share knowledge across heterogeneous assets and operating conditions. The paper proposes distributed JEPA to predict masked temporal representations in a shared space, separating representation learning from downstream forecasting.
- 1. Introduc+on: Specialized energy forecasting models limit knowledge transfer and fail to exploit shared temporal patterns across related problems.Asset, geographic, and operating-condition differences further complicate unified learning, while privacy constraints restrict access to fine-grained measurements.
- 1. Introduc+on: Reconstruction-based objectives can prioritize low-level signal statistics over higher-level temporal dynamics that generalize across settings.This motivates representation-learning objectives focused on transferable structure rather than direct observation reconstruction.
- 1. Introduc+on: JEPA differs from reconstruction and contrastive approaches by learning predictive objectives directly in latent space.This design is intended to encourage invariant temporal structure and transferable embeddings across energy assets and forecasting tasks.
- 1. Introduc+on: Distributed JEPA predicts latent representations of masked temporal segments instead of optimizing a task-specific forecasting objective.The framework combines temporal dynamics and static contextual information in a unified embedding space for downstream prediction.
2. Methods
The distributed JEPA integrates temporal and contextual energy data into shared latent representations, predicts masked targets, and decodes them for downstream forecasting across distributed clients.
- Representation learning: JEPA combines modality-specific temporal and contextual encoders to map energy measurements and static metadata into a common embedding space.TS2Vec encodes time series, while BERT transforms contextual metadata; the architecture can incorporate additional modalities.
- JEPA architecture: Training separates visible-context processing from full-sequence target encoding, using masked positions for prediction and an EMA target encoder without direct gradient updates.The Context Encoder receives visible metadata and context-window tokens, while the Target Encoder processes the unmasked sequence to provide target representations.
- JEPA architecture: The Predictor cross-attends to contextual latent representations and uses learned positional queries to generate predictions at the H masked positions.Because masked positions are absent from the Context Encoder output, positional embeddings represent the target locations during cross-attention.
- Optimization: The composite objective combines cosine latent-prediction loss with covariance and temporal-variance regularization to prevent collapse and preserve dynamic variation.Covariance regularization promotes feature diversity, while a hinge-based variance term keeps prediction variance above threshold ε across the prediction horizon.
- Downstream decoding: After training, the Target Encoder is discarded and frozen Context Encoder and Predictor representations are used as features for forecasting, classification, or anomaly detection.Forecasting predictions are decoded into raw time-series values through a hierarchical structure with cluster-specific and client-local decoders.
- Distributed preprocessing: TS2Vec training is distributed sequentially across clients, sharing updated weights while retaining raw energy data locally for privacy-preserving representation learning.Each client optimizes from shared weights using only local data before returning updated model weights.
2. Results
JEPA learns stable latent representations and transfers across unseen energy consumers and PV sites. It matches the Transformer on Genome, outperforms it on most PV sites, and remains comparatively robust under missing-data corruption.
- 2.1. Self-supervised learning dynamics: Cosine similarity reaches approximately 0.98, while effective rank stabilizes at 185–200 of 256 dimensions on Genome and reaches 235 during PV pretraining.PV training uses random multi-block masking before future-mask stages, and the newly initialized predictor rapidly recovers after its initial cold-start drop.
- 2.2. Transfer learning capabilities: Test-set median R² is comparable to training performance across Genome clusters and PV sites, indicating transfer to consumers and sites excluded from JEPA training.Genome test median R² ranges from 0.660–0.844 versus 0.648–0.854 on training consumers; PV test median R² is 0.818 versus 0.838 for training sites.
- 2.2. Transfer learning capabilities: Predictions generally follow consumption temporal dynamics but have difficulty capturing high fluctuations and variations.Representative plots correlate low MAE and high R² with closer agreement between predicted and actual energy values.
- 2.3. Energy prediction accuracy: JEPA achieves higher R² than the Transformer in 3 of 5 Genome consumer clusters, while overall Genome performance remains similar between models.The Transformer has lower MAE in all five clusters, whereas RMSE advantages are split across clusters.
- 2.3. Energy prediction accuracy: JEPA outperforms the Transformer on 9 of 10 PV sites, achieving R² of 0.73–0.88 where the Transformer is below 0.45 on several sites.The exception is SQ567_Flexible_PV, where JEPA has R² = −2.82 and the Transformer wins on all three metrics.
- 2.4. Performance analysis under data degradation: JEPA remains more accurate than the Transformer across PV degradation scenarios and is more resilient to random missing information, although the Transformer better tolerates additive embedding-space noise on Genome.Block-missing corruption has only a minor effect on both models, including for 24-hour gaps.
3. Discussion
The distributed JEPA learned stable, transferable representations for heterogeneous energy forecasting and generally matched or exceeded a Transformer, especially under missing-data corruption. Its limitations include weaker performance on an underrepresented PV regime and evaluation confined to building consumption and photovoltaic generation.
- Representation quality: Stable predicted-target alignment and substantial latent-space diversity indicate that covariance and temporal variance regularization mitigated representation collapse.The learned embeddings also used a large fraction of available representational capacity, as reflected by effective rank.
- Cross-domain transfer: JEPA achieved comparable building-energy forecasting performance and consistently outperformed the Transformer across most photovoltaic clients, indicating transferability across energy domains.The building comparison showed advantages for different client clusters, while photovoltaic results favored JEPA across most clients.
- Robustness: JEPA outperformed the Transformer under random missing-data corruption, while the Transformer was more resilient to additive Gaussian noise on the building dataset.On photovoltaic data, JEPA maintained lower errors and higher R² across all evaluated degradation settings.
- Robustness: Both models were largely unaffected by contiguous block-missing corruption, suggesting that TS2Vec representations preserve sufficient temporal context for moderate sensor outages.This robustness pattern differs from the random missing-data comparison, where JEPA consistently performed better.
- Limitations: Performance degraded on the SQ567 Flexible PV site, where generation patterns had substantially lower variability than those seen during training.This suggests reduced effectiveness for rare or underrepresented operating regimes.
- Limitations: The evaluation covered only building consumption and photovoltaic generation, leaving generalization across the broader range of energy forecasting applications uncharacterized.Future work should assess larger and more diverse asset collections and cross-domain transfer with limited adaptation data.
4. Conclusions
The study presented distributed JEPA for self-supervised energy time-series representation learning by predicting latent representations while combining temporal and contextual information. Across building and photovoltaic datasets, the representations supported accurate forecasting, strong photovoltaic-site generalization, and improved robustness to missing data, although evaluation remained limited to those two dataset types.
- 4. Conclusions: Distributed JEPA decouples representation learning from downstream forecasting by predicting latent representations instead of reconstructing observations.It integrates temporal and contextual information within a shared embedding space.
- 4. Conclusions: JEPA representations remained stable, avoided collapse, and supported accurate 24-hour-ahead forecasting with performance comparable to or exceeding a task-specific Transformer.The evaluation used building energy consumption and photovoltaic generation datasets.
- 4. Conclusions: The framework demonstrated strong generalization across photovoltaic sites and improved robustness under missing-data corruption.These results support predictive latent-space objectives for learning transferable temporal representations without task-specific supervision.
- 4. Conclusions: The work was limited to building energy consumption and photovoltaic generation datasets, with transferability evaluated through downstream forecasting tasks.The conclusion identifies larger and more diverse evaluations as a direction for future work.