WorldTS: World Modeling for Multimodal Covariate-aware Time Series Forecasting

1 INTRODUCTION

• Directly forecasting observations can entangle system-state evolution with its manifestation in observation space, rather than explicitly modeling underlying states and dynamics.Similar observations may correspond to different underlying states, while future evolution largely depends on those states and their dynamics.Source: §1 · p. 1
• External factors shape future states, and multimodal covariates can provide complementary predictive cues that are difficult to infer from historical observations.The paper gives weather conditions for electricity-load dynamics and cloud cover for photovoltaic generation as examples.Source: §1 · p. 2
• WorldTS predicts future states in latent space using historical states and multimodal covariate representations, with encoded ground-truth future states providing direct supervision.It freezes the encoders and predictor before training a decoder to map predicted states back to observations.Source: §1 · pp. 2–3
• On MMSP, predicted future-state distributions with covariates are more consistent with encoded future-state distributions than those without covariates.The paper reports experiments on 21 real-world datasets covering numerical-, text-, and image-covariate forecasting settings.Source: §1 · pp. 2–3
• WorldTS produces a forecast trajectory that more closely matches the ground-truth future sequence than its observation-space variant on MMSP.Obs-space is trained end-to-end using only observation-space loss, without direct supervision on future states.Source: §1 · pp. 2–3
• WorldTS frames multimodal covariates as conditioning future target-state prediction, rather than using them only for observation-space forecasting.The paper presents extensive comparisons and ablation studies to characterize the framework's effectiveness.Source: §1 · p. 3

2.1 LATENT-SPACE PREDICTIVE MODELING

• TimeAlign aligns historical representations with encoded future targets, and LatentTSF predicts and decodes future states; these methods use the target series itself, unlike WorldTS's multimodal covariate-conditioned prediction.WorldTS directly supervises predicted states with states encoded from ground-truth future observations.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 COVARIATE-AWARE FORECASTING

• Numerical- and heterogeneous-modality covariate methods mainly use external information for observation-space forecasting, whereas WorldTS encodes multimodal covariates to condition latent future-state prediction.The cited examples include methods for target–exogenous dependencies, known future covariates, text, and satellite imagery.Source: §2.2 · pp. 3–4

3.1 PROBLEM FORMULATION

• The task is to forecast the next H time steps of a C-variable historical series using multimodal covariates, whose temporal availability may vary by modality.The supplied formulation identifies the future target as x_T+1:T+H.Source: §3.1 · p. 4
• WorldTS maps historical observations and each modality's covariates to representations, then predicts future states conditioned on those representations and decodes them into observations.The decoder maps predicted future states back to the observation space.Source: §3.1 · p. 4
• During training, the shared state encoder maps ground-truth future observations to states that directly supervise the predicted future states.The supervision target is the encoded future-observation state.Source: §3.1 · p. 4

4.1 FRAMEWORK OVERVIEW

• WorldTS first learns future-state dynamics conditioned on multimodal covariates, then freezes its encoders and predictor while training the decoder to recover observations.At inference, it predicts future states from historical observations and available covariates before decoding them.Source: §4.1 · p. 4

4.2 TS ENCODER AND TS DECODER

• The shared TS Encoder maps historical observations and ground-truth future observations into a common state space, while the decoder maps predicted future states back to observations.Local patches capture temporal context; the decoder uses the last Sd−1 historical states with predicted future states and produces one observation prediction per forecast step.Source: §4.2 · pp. 5–6

4.3 MULTIMODAL COVARIATE ENCODING

• Numerical covariates use a separate encoder, while text and image branches extract features and project them through modality-specific MLP adapters.The text branch uses a frozen GPT-2 input embedding layer and the image branch a lightweight CNN.Source: §4.3 · p. 6

4.4 COVARIATE-CONDITIONED STATE PREDICTOR

• The predictor fuses historical states with numerical covariates and uses direct fusion or CFA for modality representations, then refines base future states with future covariates when available.CFA maps modality representations through a low-rank bottleneck into state-space residuals.Source: Figure 3 · p. 6, §4.4 · p. 6

4.5 TWO-STAGE LEARNING OBJECTIVES

• Stage 1 matches predicted future states to encoded ground-truth future states using L1 and cosine losses, with VICReg regularizers to prevent representation collapse.In Stage 2, frozen encoders and predictor feed predicted and encoded future states through the same decoder; prediction and reconstruction losses update only the decoder.Source: §4.5 · p. 7

5.1 EXPERIMENTAL SETTINGS

• The evaluation covers 21 datasets: 12 with numerical covariates, eight Time-MMD text-covariate datasets, and one image-covariate dataset.The comparisons use dedicated covariate-aware and general forecasting baselines, with dataset-specific protocols following DAG, VoT, and UniCA.Source: §5.1 · pp. 7–8

5.2 MAIN FINDINGS

• WorldTS achieves the lowest MSE and MAE on all 12 numerical-covariate datasets, reducing Energy MSE by 18.0% versus the strongest baseline.Table 1 reports averages over short and long horizons; full results are in Table 5.Source: §5.2 · p. 8, Table 1 · p. 8
• WorldTS ranks first in 14 of 16 dataset–metric comparisons across eight Time-MMD datasets, with Health gains of 8.9% in MSE and 9.1% in MAE versus the strongest baselines.Table 2 covers the eight datasets at a 12-step prediction horizon.Source: §5.2 · p. 8, Table 2 · p. 8
• On MMSP, WorldTS achieves the lowest MSE and MAE, reducing them by approximately 9.0% and 3.4%, respectively, versus TFT.Table 3 reports results at a 24-step prediction horizon.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 PARAMETER SENSITIVITY

• Across Colbun, Climate, and MMSP, zeroing any loss weight raises MAE; moderate state dimensions and causal patch lengths generally perform better than extremes.Among tested values, λlatent = 10 gives the lowest MAE on all three datasets; λsim = 15 and λVIC = 0.1 are consistently best or second best, while λVIC = 0.5 markedly increases MMSP error.Source: §5.3 · p. 9, Figure 4 · p. 9

5.4 ABLATION STUDY

• Removing any loss, replacing causal state mappings with pointwise MLPs, training only with observation-space MAE, or removing covariates degrades performance.Covariate removal has a particularly large effect on MMSP; the ablations are reported on representative numerical, textual, and visual datasets.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION

• WorldTS combines direct future-state supervision and two-stage training to learn latent dynamics conditioned on multimodal covariates, then decode predicted states into observations.The conclusion contrasts this approach with methods that mainly use external information for observation-space forecasting.Source: §6 · p. 9

A.1 DATASETS

• The evaluation datasets pair numerical targets with numerical, textual, or image covariates across energy, other domains, and solar-power forecasting.The 12 numerical datasets include energy-market, hydropower, and wind tasks; the eight Time-MMD datasets pair target series with time-relevant reports or records; MMSP pairs solar output with satellite imagery, weather predictions, and plant coordinates.Source: §A.1 · p. 15

A.2 BASELINES

• Baselines span dedicated covariate-aware methods, general-purpose forecasters, text and image multimodal models, and pretrained or covariate-adapted forecasting models.The numerical setting includes methods such as DAG and KITE, the text setting compares time-series-only and multimodal approaches, and the image setting includes specialized, pretrained, multimodal, and UniCA-adapted models.Source: §A.2 · pp. 15–16
• Overlapping causal observation patches produce time-aligned states, while local state patches reconstruct target observations.The pointwise encoder-decoder ablation instead uses pointwise MLP encoders and decoder, with latent-dynamics learning followed by frozen-module observation decoding.Source: Figure 5 · p. 16, §A.2 · p. 17

A.3 IMPLEMENTATION DETAILS

• Numerical and Time-MMD data use chronological 70%/10%/20% train-validation-test splits, while MMSP uses 80%/10%/10%.Experiments use PyTorch on NVIDIA A800 80GB GPUs; both stages use Adam.Source: §A.3 · p. 17
• Lookback windows and prediction horizons follow setting-specific protocols: numerical tasks use short- and long-term horizons, Time-MMD uses 12 steps, and MMSP uses a 2,048-step lookback and 24-step horizon.For Colbun and Rapel, numerical short-term settings use 60 lookback and 10 prediction steps, and long-term settings use 180 and 30; other numerical datasets use 168/24 and 720/360 steps.Source: §A.3 · p. 17
• The appendix’s one-stage observation-space MAE procedure jointly trains the encoders, state predictor, and decoder end to end.Its algorithm computes forecasts from historical observations and covariates, then updates all modules using observation-space L1 loss.Source: §A.3 · p. 18
• The text branch mean-pools valid token embeddings from frozen GPT-2 and maps them to time-aligned text representations.The image branch extracts frame-wise spatial features with a lightweight CNN, then maps them to compact continuous, time-aligned image representations.Source: §A.3 · p. 18

A.4 ABLATION PROTOCOLS

• Pointwise MLPs map observations to states and predicted states back to observations, sharing parameters across time without aggregating neighboring steps.The text and image encoders, state predictor, objectives, and two-stage protocol remain unchanged.Source: §A.4 · p. 18
• The one-stage variant removes future-state encoding and state-space objectives, then trains the retained causal patch encoder and decoder end to end with observation-space MAE.Training lasts N1 + N2 epochs to match the full model's total training epochs.Source: §A.4 · p. 18
• Loss ablations zero one coefficient at a time, while the no-covariate variant sets C = ∅ and retains the full architecture, objectives, and two-stage protocol.Source: §A.4 · p. 18

• WorldTS achieves the best or tied-best average performance on all 12 numerical-covariate datasets.Table 5 gives complete MSE and MAE results across forecasting horizons, along with the averages reported in the main text.Source: B FULL NUMERICAL-COVARIATE RESULTS · p. 19
• In the few individual forecasting settings where WorldTS does not rank first, it typically remains second-best or within a competitive range.Source: B FULL NUMERICAL-COVARIATE RESULTS · p. 19
• The reported full results cover all forecasting horizons on the 12 numerical-covariate datasets and include the main-text averages.Source: B FULL NUMERICAL-COVARIATE RESULTS · p. 19

C LIMITATIONS AND FUTURE WORK

• A rigorous theory of covariate-conditioned predictive-state learning remains open, including links among covariate information, predictive-state quality, and forecasting error.Source: C LIMITATIONS AND FUTURE WORK · p. 19
• WorldTS uses largely general-purpose objectives and regularization; temporal-structure-specific regularization could better capture temporal dependencies and dynamics.Source: C LIMITATIONS AND FUTURE WORK · p. 19
• WorldTS currently targets regularly sampled series and is trained separately for each dataset; pretrained foundation-model integration and irregular or asynchronous settings are proposed extensions.Source: C LIMITATIONS AND FUTURE WORK · p. 19

D.1 TIME-SERIES FORECASTING

• Deep forecasting research has explored representation learning, training objectives, probabilistic forecasting, cross-variable modeling, foundation-model adaptation, irregular sampling, and missing observations.Source: §D.1 · p. 20

D.2 LATENT-SPACE PREDICTIVE MODELING

• TimeAlign aligns historical and future representations, while LatentTSF predicts future states before decoding; WorldTS adds multimodal covariates to directly supervised future-state prediction and separates state-dynamics learning from observation decoding.Source: §D.2 · p. 20

D.3 COVARIATE-AWARE FORECASTING

• Covariate-aware methods use external information through numerical, text, and image inputs; WorldTS conditions latent future-state prediction on covariates and directly supervises it with encoded future observations.Source: §D.3 · p. 20