Source-linked AI summary
Self-Supervised Latent Action-Conditioned World Modeling for Time Series
Chia-Han Kuo, Chengshuai Zhao, Huan Liu, Lo Pang-Yun Ting
TL;DR
Existing JEPA-based time-series approaches often omit information underlying transitions between observations, which may inform forecasting and classification. ACTS-WM learns latent actions from time-series observations and uses them to condition a JEPA-based world model; experiments across 22 datasets show improvements over pretrained JEPA-based encoders on many datasets.
Problem
Existing time-series approaches primarily learn from observed temporal patterns without explicitly modeling information underlying transitions, which may inform forecasting and classification.
Source: §1 · p. 2Method
ACTS-WM learns discrete latent actions from observed transitions without action annotations and uses them to condition a JEPA-based world model for forecasting and classification.
Source: §1 · p. 2, §2.3 · p. 3Results
ACTS-WM improves over pretrained JEPA-based encoders on many of 22 datasets spanning univariate and multivariate forecasting and classification.
Source: §6 · p. 9Takeaways & Limitations
Analyses highlight benefits of learned latent actions and action conditioning for the evaluated time-series tasks.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Time-series data play an important role in various tasks, such as weather forecasting, web traffic analysis, and speech recognition. Recently, pioneering studies have increasingly adopted Joint-Embedding Predictive Architecture (JEPA) based encoders to learn time-series representations through latent-space prediction. However, they often overlook the underlying factors governing transitions between time-series observations. In this work, we hypothesize that explicitly modeling the information behind changes in time series can facilitate time-series modeling. Building on this hypothesis, we propose ACTS-WM, a self-supervised latent action-conditioned world model framework for time-series analysis. First, we introduce a latent action modeling approach that infers the underlying information behind time-series transitions as latent actions, without requiring explicit action annotations. We then develop a JEPA-based encoder and an action-conditioned world model to predict target time-series representations conditioned on the learned latent actions and task information. We evaluate ACTS-WM across 4 representative tasks on 22 datasets from diverse domains, covering univariate and multivariate forecasting and classification. Comprehensive experiments demonstrate that ACTS-WM can improve downstream performance over pretrained JEPA-based encoders by learning complementary representations.
1 Introduction
Time-series foundation models and JEPA encoders learn general-purpose latent representations, but existing approaches do not explicitly model the factors underlying temporal changes [7, 19, 26] [10, 12, 23]. ACTS-WM infers latent actions from observations and uses them in a task-conditioned world model for forecasting and classification.
Motivation
- Explicitly modeling temporal-change information may reveal predictive transition dynamics and invariant semantics, supplying richer supervisory signals for forecasting and classification.The paper draws inspiration from latent-action learning in visual control and robotics [36] [45].Source: §1 · p. 2
Challenges
- Passive time-series data lack explicit action signals, while datasets vary in channel counts, motivating latent transition inference that accommodates heterogeneous channel configurations.Unlike robotics settings with control actions and observed resulting transitions, the latent factors driving time-series changes must be inferred from temporal evolution.Source: §1 · p. 2
ACTS-WM framework
- ACTS-WM learns discrete latent actions from observed transitions without action annotations, infers them from time series alone, and conditions a world model on them to model transitions in JEPA representation space.Its task-conditioned modeling is designed to provide complementary representations for downstream forecasting and classification.Source: §1 · p. 2
Contributions
- The paper presents a task-conditioned latent-action approach to capture temporal evolution and within-class variation, and describes ACTS-WM as the first self-supervised latent action-conditioned world-modeling framework tailored to time series.The authors frame latent factors underlying transitions as valuable supervisory signals for transferable representations.Source: §1 · p. 2
Evaluation
- Experiments span univariate and multivariate forecasting and classification across 22 diverse-domain time-series datasets, with results showing complementary representations and analyses indicating meaningful learned transition patterns.The authors also report ablation studies and qualitative analyses of the learned latent actions.Source: §1 · p. 2
2 Related Work
Prior work spans time-series foundation and JEPA-based models, general world models, and latent-action learning. ACTS-WM addresses the gap by inferring latent actions directly from time-series observations and conditioning world modeling on them without action annotations.
2.1 Time Series Analysis
- Time-series foundation models and JEPA-based models learn representations for downstream analysis, forecasting, and prediction tasks.MOMENT [19] and UniTS [15] target adaptable representations, while Toto [7], Moirai [26] [27] [42], and Time-MoE [38] focus on forecasting. TS-JEPA and CHARM [10] adapt JEPA to time-series tasks; HEPA [32], SC-JEPA [23], and LaT-PFN [40] extend latent prediction to rare-event prediction, anomaly detection, or zero-shot forecasting. Some JEPA-based world models use action signals or information from other modalities [33] [37] [8].Source: §2.1 · p. 2
2.2 World Models
- World-model research covers latent dynamics, pixel-space simulation, latent-space planning, controllable video generation, and jointly trained policy–world models.Examples include Dreamer 4 [22], DIAMOND [2], V-JEPA 2-AC [3], H-JEPA [48], COSMOS [1], Genie [6], PaW [28], and CoLA-World [41].Source: §2.2 · p. 2
2.3 Latent Action Models
- Most time-series models omit transition-governing information, while existing latent-action models have mainly been explored in video and time-series world models often use extra action signals.ACTS-WM learns latent actions directly from time-series observations and uses them to condition world modeling without action annotations.Source: §2.3 · p. 3
3 Preliminaries and Problem Definition
ACTS-WM frames time-series pretraining as learning unannotated latent actions from paired observations and task tags, then predicting target representations. It extends inverse- and forward-dynamics latent action modeling to time-series data.
3.1 Problem Formulation
- Pretraining uses paired observed and target time series with a task tag identifying forecasting and its horizon or classification; the series may be univariate or multivariate.For forecasting, the target is the subsequent temporal segment.Source: §3.1 · p. 3
- ACTS-WM learns latent actions that capture transition information without action annotations, then uses them with a task tag to predict target representations for forecasting and classification.At evaluation, it takes an unseen time series from a pretraining or previously unseen domain.Source: §3.1 · p. 3
3.2 Inverse and Forward Dynamics Models
- ACTS-WM extends latent-action modeling to time series, building on inverse models that infer actions from past and future observations and forward models that reconstruct future observations.Prior approaches use vector quantization as an information bottleneck to prevent the inverse model from copying future observations into the action representation.Source: §3.2 · p. 3
4 Methodology
ACTS-WM learns discrete latent actions from time-series transitions, then uses those actions with a JEPA-based encoder and action-conditioned predictor to predict target representations. Its design combines channel- and temporal-aware fusion, task embeddings, and action-conditioned modeling across varied time-series inputs and tasks.
4 Methodology
- ACTS-WM has two stages: latent action modeling learns discrete actions from observed series, and an action-conditioned world model predicts target representations using those actions and a JEPA-based encoder.Source: §4 · p. 3
4.1 Latent Action Modeling in Time Series
- Latent actions represent information behind transitions between observed and target time-series observations, and ACTS-WM learns them without explicit action annotations.Source: §4.1 · p. 4
4.1.1 Channel and Temporal Fusions.
- The fusion module handles varying channel counts and sequence lengths by combining shared channel-wise convolutions, attention pooling for cross-channel dependencies, and dilated TCNs for multi-scale temporal patterns.The resulting features are used for latent action modeling.Source: §4.1.1 · p. 4
4.1.2 Task Embedding.
- Learnable task embeddings give each task tag a unique index and jointly trained embedding, enabling task-specific representations for distinct transitions such as forecasting horizons and classification.Source: §4.1.2 · p. 4
4.1.3 Inverse and Forward Dynamics Models.
- The inverse model combines fused representations of both observations with the task embedding to infer a continuous action shared across channels, then quantizes it; the forward model uses that action and the first observation to reconstruct the second.A shared-weight one-dimensional U-Net accommodates varying channel counts, while joint training minimizes reconstruction and quantization losses to encode transition-relevant information.Source: §4.1.3 · p. 4
4.1.4 Latent Action Selection.
- Vector quantization divides continuous actions into slots and replaces each with its nearest codebook vector; at evaluation, a policy predicts actions from the first observation and task tag because the target observation is unavailable.The codebook is updated by exponential moving average, and the policy is trained with the pretrained inverse model frozen to regress its continuous actions.Source: §4.1.4 · p. 4
4.2.1 JEPA-based Encoder.
- The world model adapts V-JEPA 2 for time series with one-dimensional temporal patches and shared-weight, channel-independent encoding, using interior-block and suffix masking with an EMA target encoder.A multi-channel consistency term is also introduced to account for dependencies across channels.Source: §4.2.1 · pp. 4–5
4.2.2 Action-Conditioned Predictor.
- The action-conditioned predictor uses the JEPA encoder, quantized latent action, and task tag to predict target representations, and ACTS-WM can integrate with other JEPA-based or general time-series models.The framework is designed to support different domains, sequence lengths, channel counts, and downstream tasks.Source: §4.2.2 · p. 5
5 Experiments
ACTS-WM improves JEPA-based time-series representations across forecasting and classification, with predicted representations especially effective for forecasting and combined observed-plus-predicted representations for classification. Ablations and qualitative analyses indicate that learned latent actions encode useful transition information, while performance depends on model components, codebook size, and prediction horizon.
5.1.1 Dataset.
- Experiments use 21 datasets for ACTS-WM pretraining and 22 for downstream evaluation, drawn from varied sources including UEA, UCR, forecasting benchmarks, WebTraffic, and WikiTraffic.The datasets cover univariate and multivariate time series.Source: §5.1.1 · p. 5
5.1.2 Baselines.
- ACTS-WM is compared with TS-JEPA, SC-JEPA, and JEPAACTS; downstream evaluation tests Base, “Pred. Only,” and concatenated “Obs.+Pred.” representations with task-specific heads.TS-JEPA is evaluated only on univariate tasks because it supports only univariate inputs.Source: §5.1.2 · p. 5
5.1.3 Evaluation and Implementation Details.
- Classification is evaluated with accuracy and F1, forecasting with normalized-target MSE and MAE, and representative main results are averaged over three runs.Forecasting uses input length 320 and horizon 48; the latent-action codebook has 16 vectors with two selections per action, and the JEPA encoder dimension is 128.Source: §5.1.3 · p. 5
5.2 Performance Comparison
- With JEPAACTS, “Pred. Only” reduces MSE on all 11 forecasting settings and achieves the lowest MSE across all five multivariate datasets; WikiTraffic improves in both MSE and MAE for all three encoders.“Pred. Only” generally outperforms “Obs.+Pred.” in forecasting.Source: §5.2 · pp. 5, 7, Table 1 · p. 6
- JEPAACTS “Obs.+Pred.” improves accuracy over Base on 9 of 10 univariate datasets and achieves the best accuracy on 3 of 6 multivariate datasets.SC-JEPA “Obs.+Pred.” improves on Base on 5 of 6 multivariate datasets; the results suggest complementary representations and potential generalization to datasets excluded from pretraining.Source: Table 2 · p. 6, §5.2 · p. 7
5.3 Ablation Study
- The full ACTS-WM is best in three of four task categories; removing task embeddings raises multivariate forecasting average MSE from 0.472 to 0.788, while removing temporal fusion raises MSE to 0.550 and lowers accuracy to 0.858.Random actions consistently degrade performance across all four task categories, especially classification, where “Pred. Only” loses accuracy and F1 and can fall below Base.Source: §5.3 · pp. 7–8, Table 3 · p. 7, Figure 3 · p. 7
5.4 Parameter Analysis
- Increasing codebook size from 2 to 64 generally lowers final training loss, but size 128 slightly raises it; longer horizons of 48, 168, and 336 increase mean MSE across encoders and settings.The horizon-related degradation is especially pronounced for SC-JEPA with “Obs.+Pred.”Source: Figure 4 · p. 7, §5.4 · p. 8
5.5 Interpretation of Learned Latent Actions
- UMAP visualizations show that forecasting latent actions differ across change-magnitude quartiles, while classification actions exhibit class-dependent structures with some overlap, qualitatively indicating transition-related information.The visualized continuous latent representations are inferred before quantization.Source: §5.5 · pp. 8–9, Figure 5 · p. 8
6 Conclusion and Future Work
ACTS-WM learns discrete latent actions to condition a JEPA-based world model for time-series representation prediction. Across 22 datasets, it improves over pretrained JEPA-based encoders on many datasets, and future work will improve latent-action prediction.
6 Conclusion and Future Work
- Experiments across 22 datasets show ACTS-WM improves over pretrained JEPA-based encoders on many datasets.The datasets cover univariate and multivariate forecasting and classification.Source: §6 · p. 9
- ACTS-WM learns discrete latent actions that condition a JEPA-based world model to predict target representations.Further analyses highlight the benefits of learned latent actions and action conditioning.Source: §6 · p. 9
- Future work will focus on improving the latent policy’s ability to predict latent actions.Source: §6 · p. 9
A Appendix · A.1 Dataset for Pretraining and Evaluation
Appendix A.1 catalogs the datasets used for ACTS-WM pretraining and downstream evaluation, including their sources and time-series dimensions.
A.1 Dataset for Pretraining and Evaluation
- Table 4 lists datasets used for ACTS-WM pretraining and downstream evaluation.Source: Table 4 · p. 10
- Classification datasets come from UCR [9] and UEA [4], except WebTraffic [11].Source: Table 4 · p. 10
- Forecasting datasets come from Autoformer [43], except WikiTraffic [17, 18].Source: Table 4 · p. 10
- Table 4 uses #T for the number of time steps and #C for the number of channels.Source: Table 4 · p. 10
- For windowed datasets, total time steps are estimated as average sequence length multiplied by sample count.Source: §A.1 · p. 10
A.2 Detailed Data Preprocessing and Settings · A.2.1 Preprocessing.
Preprocessing standardizes forecasting inputs and target definitions while using task-specific splits. Pretraining avoids downstream-test leakage for evaluated datasets, and classification preserves original lengths and predefined splits.
A.2.1 Preprocessing.
- Last-value imputation extends forecasting datasets and variant lengths to the maximum sample length.The last observed value is used to fill the added positions.Source: §A.2.1 · p. 10
- Univariate forecasting targets are oil temperature for ETDatasets and co2 for Weather.Source: §A.2.1 · p. 11
- Pretraining uses only training data from datasets included in downstream evaluation, but both training and test splits for other datasets.Source: §A.2.1 · p. 11
- Forecasting datasets are split into training, validation, and test sets in a 65:5:30 ratio.Source: §A.2.1 · p. 11
- Classification retains original sequence lengths and predefined train/test splits, with 10% of training data reserved for validation.Source: §A.2.1 · p. 11
A.2.2 Settings for JEPA-based Encoders. … A.4.2 Case study of latent action semantics.
ACTS-WM is evaluated with separate JEPA-encoder pretraining setups, six downstream heads, and diverse forecasting and classification tasks. Its improvements depend on the encoder and head, while a case study suggests latent actions group recurring transition patterns.
A.2.2 Settings for JEPA-based Encoders.
- JEPAACTS pretraining uses 21 datasets, while TS-JEPA uses 12 univariate datasets and SC-JEPA uses 22 datasets because both require fixed-length inputs.The 21 JEPAACTS datasets comprise 16 UEA datasets, four ETT datasets, and Weather.Source: §A.2.2 · p. 11
- Downstream evaluation covers 22 datasets across univariate and multivariate classification and forecasting.The sets include nine UCR datasets and WebTraffic for univariate classification, six UEA datasets for multivariate classification, and six forecasting datasets.Source: §A.2.2 · p. 11
A.3.1 Downstream Head Settings.
- Downstream evaluation uses linear, SVM, and MLP classification heads and linear, ridge-regression, and NLH forecasting heads.The ridge head maps flattened tokens to the forecast horizon per channel, with regularization selected on validation performance.Source: §A.3.1 · p. 11
A.3.2 Hyperparameter Settings.
- ACTS-WM uses 128-dimensional fused and JEPA representations, 16-dimensional actions, and a 16-vector codebook with two selections per action.The latent-action settings also include β = 0.05; JEPAACTS-WM trains for 200 epochs with warmup over the first 5% and cosine decay, while TS-JEPA and SC-JEPA follow their papers’ configurations.Source: §A.3.2 · p. 11
A.4.1 Performance over Downstream Heads.
- ACTS-WM improves forecasting with linear and ridge heads on most datasets, while classification gains vary by encoder and setting.With JEPAACTS, the linear head beats the base model on all forecasting datasets in the predictor-only setting; SC-JEPA with NLH beats its base on all forecasting datasets. JEPAACTS improves on most univariate classification datasets, and SC-JEPA improves on all multivariate datasets when both observation and predictor embeddings feed the linear head.Source: §A.4.1 · p. 11, Table 6 · p. 12, Table 5 · p. 12
A.4.2 Case study of latent action semantics.
- For action codes (5, 6), targets resemble context middle sections; for (14, 11), all three targets trend upward despite varied context tails.The authors speculate that these action codes may capture meaningful patterns interpretable through human-understandable concepts.Source: §A.4.2 · p. 11, Figure 6 · p. 11