Source-linked AI summary

Pythia: Toward Foundation World Models for Multimodal Time Series

Xilin Dai, Hongzhou Chen, Yifan Hu, Yiding Liu, Zewei Dong, Jiang-Ming Yang

arXiv:2610.05240v1cs.LGcs.AI

TL;DR

Reusable multimodal predictive representations for time-series forecasting remain underexplored, despite complementary information in text and auxiliary observations. Pythia learns context-conditioned latent dynamics across datasets and adapts a probabilistic decoder to frozen predictive states; on MUSE, it improves errors over the strongest published leaderboard model. The results support separating predictive representation learning from probabilistic readout, while the capacity and training-exposure experiments do not measure scaling with independent data volume.

  • Problem

    Reusable multimodal predictive representations remain underexplored, although textual context and auxiliary observations provide information complementary to numerical history.

    Source: §1 · pp. 1–2
  • Method

    Pythia learns context-conditioned latent predictions across datasets, uses a numerical reference to guide contextual corrections, and separately adapts a probabilistic decoder to frozen predictive states.

    Source: §1 · p. 2, §3.1 · p. 3
  • Results

    Pythia-Tiny reduces normalized MASE by 6.26% and WSQL by 5.00% relative to the strongest model evaluated in the published MUSE leaderboard.

    Source: §5 · p. 11
  • Takeaways & Limitations

    The results support separating predictive representation learning from probabilistic readout, and removing contextual inputs worsens aggregate accuracy despite task-varying benefits.

    Source: §5 · p. 11

    Capacity and training-exposure experiments vary over a fixed corpus and do not measure scaling with independent data volume.

    Source: §4.6 · p. 10

Abstract

from arXiv · show

Time-series foundation models offer a unified approach to forecasting across heterogeneous domains. Textual context and auxiliary observations provide complementary information about temporal dynamics, yet reusable multimodal predictive representations remain underexplored. We introduce Pythia, a foundation world model that learns context-conditioned latent dynamics across datasets through a joint-embedding predictive architecture. A stop-gradient numerical reference guides contextual corrections to predicted future states. A separate probabilistic decoder then adapts to the frozen predictive representation and observed history, decoupling world-model pretraining from observation-space forecasting. On MUSE, Pythia-Tiny's normalized mean absolute scaled error (MASE) and weighted sum quantile loss (WSQL) are 0.6879 and 0.4269, reducing errors by 6.26% and 5.00% relative to the strongest model evaluated in the published MUSE leaderboard. Through a series of controlled experiments, we investigate how to design a time-series world model through shared pretraining and how joint-embedding predictive learning can incorporate multimodal information. The results support separating predictive representation learning from probabilistic readout and show complementary contributions from entity descriptions, events, and covariates.

1 Introduction

Pythia addresses the challenge of learning reusable, context-conditioned predictive states across datasets while preserving the numerical structure needed for forecasting. It combines latent prediction, numerical-reference-guided contextual corrections, and a separately adapted probabilistic readout; Pythia-Tiny improves on the strongest published MUSE baseline.

1 Introduction

  • Numerical history captures only part of the information at a prediction origin; descriptions, events, and covariates offer complementary context.The challenge is incorporating context while retaining numerical structure for useful forecasts.Source: §1 · pp. 1–2
  • Pythia learns context-conditioned predictive states across datasets, refining numerical latent estimates with entity information, timestamp-filtered events, and observed covariates.A stop-gradient numerical reference guides contextual correction errors, with gradients updating only the context pathway.Source: §1 · p. 2
  • Training first learns latent prediction through forecasting and masked reconstruction, then adapts a probabilistic decoder over observed history and frozen predicted states.Both stages learn across datasets without adaptation to individual MUSE tasks.Source: §1 · p. 2
  • Pythia investigates shared latent predictive pretraining as a way to learn temporal dynamics across heterogeneous datasets.The introduction frames its experiments around latent pretraining, numerical initialization, and multimodal integration.Source: §1 · p. 2
  • 6.26% lower normalized MASE and 5.00% lower WSQL than the strongest published MUSE baseline are achieved by Pythia-Tiny.Controlled experiments examine the design of predictive representations, multimodal integration, and two-stage training.Source: §1 · pp. 2–3

2 Related Work

Time-series foundation models learn shared forecasting behavior across heterogeneous datasets, while multimodal work incorporates text and other contextual observations. Pythia distinguishes itself by pretraining context-conditioned latent prediction and separately adapting a decoder to frozen predictive states.

2.1 Time-series foundation models

  • Time-series foundation models learn shared forecasting behavior from heterogeneous series, while Pythia makes context-conditioned latent prediction its pretraining objective and separately adapts a decoder.Earlier approaches include sparse experts, flow-matching predictive distributions, and support for covariate-informed prediction.Source: §2.1 · p. 3

2.2 Toward multimodal time-series forecasting

  • Multimodal forecasting uses textual and auxiliary observations beyond numerical history; Pythia uses semantic context to refine a numerical latent estimate before readout training.The related work describes benchmarks and methods combining text with numerical prediction, including residual-supervision approaches.Source: §2.2 · p. 3

3 Pythia

Pythia predicts future observations as context-conditioned latent states, then fits a probabilistic decoder to the frozen predictive representation. Its four-part recipe separates latent prediction, contextual correction, latent supervision, and probabilistic readout.

3.1 Forecasting through predictive latent states

  • Pythia maps observed history and retained context to predicted future latent states, then fits a probabilistic readout to that frozen mapping.The second stage uses observed history and predicted world states to support the readout.Source: §3.1 · p. 3, §3.4 · pp. 5–6

3.2 Context as a residual correction

  • Context memory supplies a residual correction to numerical predictions, while the numerical estimate remains available as the starting point.Events and covariates are described as contextual inputs elsewhere in the section, but are not specified in the provided passages for this subsection.Source: §3.2 · pp. 4–5

3.3 Latent supervision and numerical-reference guidance

  • Latent supervision aligns world states with frozen numerical targets in direction and log-RMS magnitude; a detached numerical reference guides context-only correction gradients.World-state matching and scale losses update both pathways, whereas the reference-guidance loss updates only the context encoder and correction module.Source: §3.3 · p. 5

3.4 Independent probabilistic readout

  • The probabilistic readout adapts three decoder blocks over observed history, the REG token, and predicted world states while keeping the predictive system frozen.Its pinball-loss objective includes a parameter-anchor penalty, and the final decoder block, normalization, and quantile head remain frozen.Source: §3.4 · pp. 5–6

4 Experiments

Pythia is evaluated on MUSE against published baselines, with both stages pretrained across heterogeneous series and no dataset-specific MUSE adaptation. The experiments compare forecasting performance, training recipes, multimodal inputs, initialization policies, and capacity-budget choices.

4.1.0.1 Benchmark and metrics.; 4.1.0.2 Unified pretraining and evaluation.; 4.1.0.3 Comparisons.

  • The MUSE evaluation uses 14 datasets across eight domains and six frequencies, with selected readout checkpoints and no dataset-specific adaptation.The benchmark retains published baseline protocols; MASE scores median forecasts and WSQL scores nine predictive deciles, normalized against Seasonal Naive by group.Source: §4.1.0.2 · p. 6, §4.1.0.1 · p. 6, §4.1.0.3 · p. 6

4.2.0.1 Overall performance.

  • 0.6879 MASE and 0.4269 WSQL: Pythia-Tiny improves over Chronos2 (0.7338/0.4493) by 6.26% and 5.00%, respectively.The table retains all 31 published baselines and their published training protocols; lower normalized geometric-mean scores are better.Source: §4.2.0.1 · p. 6, Table 1 · p. 7

4.2.0.2 Distribution across tasks.

  • Tiny improves on Seasonal Naive in 27 of 29 groups for MASE and 28 of 29 for WSQL, though some task-specific exceptions remain.No family member dominates every group; Medium improves on Tiny at weekly and monthly frequencies.Source: §4.2.0.2 · p. 7

4.3.0.1 Separate representation learning from readout adaptation.

  • Pythia-Small reduces MASE/WSQL by 2.70%/1.55% over direct forecasting and 10.91%/11.51% over joint readout adaptation.The comparison uses equal 60K total updates; the results support freezing the predictive representation during observation-space readout adaptation.Source: §4.3.0.1 · pp. 7–8, Figure 3 · p. 9

4.3.0.2 Preserve information needed by the observation-space interface.

  • Removing latent magnitude alignment raises MASE/WSQL by 12.38%/14.38%, while removing observed history raises them by 61.09%/66.36%.The readout combines observed numerical memory with predictive information in world states; direction-only latent matching is insufficient for the tested readout.Source: §4.3.0.2 · p. 8

4.4 How multimodal information contributes

  • Full Pythia reduces MASE by 2.61%, 3.00%, and 1.64% versus no-entity, no-event, and no-covariate controls, respectively; group effects are heterogeneous.The paired numerical-only comparison improves 17 of 29 groups in MASE and 18 in WSQL, while several finance and healthcare groups favor numerical-only learning.Source: §4.4 · p. 9, Figure 3 · p. 9

4.5 How should a numerical foundation model initialize Pythia?

  • Preserving the numerical encoder gives the lowest errors under every decoder policy, while adapting the decoder improves F/F’s 1.3271/0.9020 to F/A’s 0.7031/0.4337 MASE/WSQL.The experiments support selective reuse with distinct choices for preserving the latent space and adapting the readout; removing the anchor from the default decoder raises MASE/WSQL by 1.36%/0.88%.Source: §4.5 · pp. 9–10, Figure 4 · p. 10

4.6 Capacity and training budget

  • The best configuration shifts from Tiny to Small to Medium as the per-stage budget rises from 10K to 30K to 50K, but the optimum does not improve monotonically.At 60K total updates, Medium’s 50K+10K allocation improves MASE/WSQL by 3.34%/2.46% over 10K+50K. These experiments vary capacity and exposure over a fixed corpus, not independent data volume.Source: §4.6 · p. 10, Figure 5 · p. 11

5 Conclusion

Pythia-Tiny leads the published MUSE leaderboard comparison, while controlled experiments support separating shared predictive pretraining from probabilistic readout adaptation. The conclusion also reports that contextual inputs worsen aggregate accuracy when removed, though their benefits vary across tasks.

5 Conclusion

  • 6.26% lower normalized MASE and 5.00% lower WSQL than the strongest model evaluated in the published MUSE leaderboard for Pythia-Tiny.The conclusion also reports that removing contextual inputs worsens aggregate accuracy, although benefits vary across tasks.Source: §5 · p. 11
  • Matched-condition experiments support a shared latent interface between contextual pretraining and probabilistic forecasting, with design choices evaluated under matched training conditions.The experiments characterize numerical initialization, contextual correction, and stage-specific adaptation.Source: §5 · p. 11

A.1 Model components and configurations

  • The context encoder uses cached Qwen3-VL-Embedding-8B embeddings, with event content paired with time and each covariate name paired with its numerical summary.The embeddings are unit-normalized, scaled by the square root of their dimension, and projected; entity descriptions also use cached embeddings.Source: §A.1 · p. 17

A.2 Objective evaluation and optimization

The training recipe uses staged optimization, with controls varying whether latent pretraining, contextual inputs, and reference guidance are retained. The pretraining corpus comprises separately counted eligible window pools whose domain and frequency composition is shown in Figure 6.

A.2 Objective evaluation and optimization

  • The stable reference computation uses float32 errors and log1p when b > 0, returns a when b = 0, and avoids overflow with a nonpositive exponential argument.Direction normalization uses ϵn = 10^-12, separately from the 10^-8 log-RMS stabilization.Source: §A.2 · p. 17
  • AdamW training uses a 10^-4 learning rate, 0.01 weight decay, gradient clipping at 1, 2,000 warmup updates, and a cosine schedule.Stage 2 adapts decoder blocks 8–10 at relative learning rates 1, 0.1, and 0.01; the final block, normalization, and native quantile head remain frozen.Source: §A.2 · p. 17

A.3 Training protocols for the learning-recipe controls

  • Direct forecasting omits Stage 1, while joint readout adaptation updates representation and readout; numerical-only controls disable Pythia context and reference guidance.All controls use the same pretraining corpus and no MUSE-specific optimization; the numerical-only two-stage control retains latent pretraining and readout updates.Source: §A.3 · p. 18

B Pretraining corpus

  • Stage 1 contains 18,173,268 eligible windows and Stage 2 contains 18,474,132, counted separately; Figure 6 shows domain and frequency composition.The pools differ in eligibility and are not the optimization sampling distribution; Transport, Industrial, and Finance are the largest domains.Source: B Pretraining corpus · p. 19, Figure 6 · p. 19

C.1 Metrics and aggregation

The evaluation reports normalized MASE and WSQL using MUSE coverage rules and equal-group aggregation against Seasonal Naive. Its coverage spans 12,729 test windows, with context inputs varying in availability across windows.

C.1 Metrics and aggregation

  • MASE uses the forecast median and a seasonal scale from observed context; WSQL averages normalized pinball losses across nine deciles and is not an exact CRPS.Both metrics retain MUSE validity masks and coverage rules.Source: §C.1 · p. 19
  • The overall score is an equal-group geometric mean across dataset–frequency–horizon groups, each normalized by its Seasonal Naive score on matching coverage.It differs from averaging individual windows or dataset scores; Seasonal Naive has normalized score 1.Source: §C.1 · pp. 19–20

C.2 Evaluation coverage and model comparisons

  • Evaluation covers 12,729 test windows across 29 scoring groups, and same-seed run differences are reported descriptively without claims of statistical significance.All baseline rows are taken from the published MUSE table, preserving its scores, training regimes, and missing probabilistic results.Source: §C.2 · p. 20

C.3 Context availability and preprocessing

  • Entity descriptions appear in every window, while events and covariates occur in 73.96% and 82.53%, respectively; 59.92% contain all three inputs.Events are truncated at the last observed history timestamp, and covariate summaries use retained historical observations only.Source: §C.3 · p. 20, Table 4 · p. 20

D.1 Results by scoring group, horizon, and frequency

The results section presents all 29 MUSE scoring groups and horizon- and frequency-level score summaries, alongside selected forecast examples. Quantile diagnostics show higher central-80% coverage for Tiny than Small, but coverage remains below nominal for both.

D.1 Results by scoring group, horizon, and frequency

  • Table 5 reports normalized MASE/WSQL for all 29 scoring groups, while Figure 7 summarizes scores by horizon and frequency.The horizon and frequency slices have different task mixtures and do not isolate horizon or frequency effects on identical data.Source: §D.1 · p. 20, Table 5 · p. 21, Figure 7 · p. 22

D.2 Forecast examples

  • Figure 8 shows Tiny and Medium predictive medians and 0.1–0.9 quantile bands for hourly transport and weekly healthcare examples.The examples use fixed public configurations and windows selected independently of forecast quality.Source: §D.2 · p. 22, Figure 8 · p. 23

D.3 Quantile coverage and ordering

  • Tiny's central-80% coverage exceeds Small's by 2.85 percentage points, yet remains below the nominal 80% level for both models.Coverage is 74.5–74.8% across Tiny horizon groups and 70.6–72.2% for Small; pooled crossings are 3.10% for Tiny and 3.15% for Small.Source: Table 6 · p. 22, §D.3 · p. 23
  • Crossing rates are higher at long horizons and concentrated in Fidel-TS Germany, reaching 10.96–15.01% for Tiny and 11.52–15.12% for Small.Figure 9 and Table 6 report group-level quantile diagnostics using original outputs without reordering or calibration.Source: §D.3 · pp. 22–23, Table 6 · p. 22

E.1 Capacity and sampled training exposure

The capacity-and-budget sweep shows that the lowest-error model changes with update budget, while increasing capacity or training duration alone does not reliably improve results. Batch size, update count, and exposure interact, and equal update or exposure counts do not imply equal computation.

Batch-size and exposure measures

  • With eight devices and no gradient accumulation, sampled-window exposure is D = 8bU, but repeated draws mean it differs from unique eligible windows.The two stages use the same per-device batch size, and total exposure is the sum of stage exposures.Source: §E.1 · p. 23
  • Equal update or exposure counts do not imply equal computation because stages update different modules and larger configurations make each update more costly.More updates or larger batches also change sampled exposure, including repetitions.Source: §E.1 · p. 24, Table 7 · p. 24

Capacity and update budget

  • 0.6879/0.4269 for Tiny at 10K and 0.6981/0.4373 for Small at 30K are the reported lowest-error MASE/WSQL pairs as the best capacity shifts with budget.At 50K, Medium is the best configuration; the available passage does not give its scores.Source: §E.1 · p. 24

Batch-size and exposure measures

  • Batch-size effects vary with model size and training duration, so larger batches are not uniformly better.The figure reports percentage changes from batch 128 within the same size and per-stage update budget, using a common symmetric ±4% scale; absolute scores are in Table 8.Source: Figure 10 · p. 25

Capacity and update budget

  • Increasing capacity or paired update budgets does not reliably improve performance, and the reported sweep does not establish a scaling law or exhausted model capacity.Medium benefits from extending paired budgets from 10K to 30K to 50K in both metrics, unlike Tiny and XLarge.Source: §E.1 · p. 25

E.2 Batch size and stage allocation

Batch size and the allocation of updates between representation pretraining and readout have interacting effects, with no universally best recipe. Checkpoint trajectories also vary within runs and are distinct from comparisons among final checkpoints.

E.2 Batch size and stage allocation

  • 0.7019/0.4357 is Medium’s MASE/WSQL at batch 256 and 50K updates per stage, while the preferred batch varies with model size and budget.At 30K, Medium’s batch 96 is best; at 50K, both metrics improve monotonically as batch increases from 96 to 256.Source: §E.2 · pp. 25–26, Table 8 · p. 25
  • At 60K total updates, Medium’s 50K+10K allocation at batch 256 scores 0.6969/0.4282, versus 0.7210/0.4390 for 10K+50K, reducing MASE/WSQL by 3.34%/2.46%.For Large, the best complete grid point is instead 10K+10K, at 0.6955/0.4279; neither more total updates nor assigning additional updates to Stage 1 is unconditionally better.Source: §E.2 · p. 26, Figure 11 · p. 26
  • At matched sampled exposure, Tiny’s 10K+10K batch-256 recipe improves MASE/WSQL by 1.91%/1.19% over 20K+20K at batch 128, but Large’s counterpart improves MASE slightly and worsens WSQL.The comparisons also change update count and schedule horizon, so they are alternative recipes with matched exposure, not an exposure-only test.Source: §E.2 · pp. 26–27

E.3 Checkpoint trajectories

  • Tiny’s batch-128, 30K+30K trajectory improves readout MASE from 0.7126 at 10K to 0.7079 at 30K, yet remains worse than the independently trained 10K+10K model.The longer recipe changes both representation-training budget and readout learning-rate horizon; it is not a continuation of the short recipe.Source: §E.3 · p. 27, Figure 12 · p. 27
  • Medium’s batch-128, 50K+50K trajectory is nonmonotonic: readout MASE is 0.7022 at 20K and 0.7103 at the final 50K endpoint.These reversals alone do not identify overfitting or establish insufficient data or parameter scale as their cause.Source: §E.3 · p. 27, Figure 12 · p. 27

F.1 Training objectives and input controls

Controlled experiments examine how pretraining objectives, context inputs, readout design, and adaptation choices affect Pythia’s predictive representation and forecasts. The results show that design choices have metric-specific effects, and contextual correction and sensitivity vary across models and tasks.

F.1 Training objectives and input controls

  • Removing log-RMS alignment increases MASE/WSQL by 12.38%/14.38%, while removing task conditioning worsens them by 4.48%/5.39%.Forecast-only pretraining is weaker than the mixed task on both metrics; masked-only is close in MASE and better in WSQL, so the mixture does not uniformly dominate probabilistic accuracy.Source: §F.1 · p. 27, Figure 13 · p. 28

F.2 Readout inputs and decoder design

  • Removing historical numerical memory raises readout errors to 1.1245/0.7275 with REG retained or 1.1627/0.7455 without it, while removing REG alone yields 0.7157/0.4440.The readout uses history as observed numerical memory and world states as a context-conditioned prediction of the future.Source: §F.2 · p. 28, Table 9 · p. 29

F.3 Numerical initialization and adaptation policies

  • The frozen-encoder row has the lowest errors under every decoder policy, but anchored decoder adaptation reduces MASE by 47.02%–59.35% relative to a frozen decoder.The matrix does not isolate why adapted-encoder results are poorer or establish catastrophic forgetting.Source: §F.3 · p. 30, Table 13 · p. 31

F.4 Latent correction and reference weights

  • Contextual correction lowers mean latent error in all 29 scoring groups; the guided model has 7.13% lower numerical error, while the no-reference model has slightly lower final latent error.The guided model nevertheless has better observation-space forecasts; correction improves 99.994% of patches with guidance and 99.984% without it.Source: §F.4 · p. 32, Figure 14 · p. 32, Table 14 · p. 32

G.1 Test-time semantic interventions

  • Tiny and Small incur higher MASE under null or random semantic-embedding replacement, whereas the selected Medium checkpoint changes only slightly.Sensitivity varies by training recipe and task; numerical histories and numerical covariate summaries remain unchanged in these interventions.Source: §G.1 · p. 33, Figure 15 · p. 33, Table 15 · p. 33

G.2 Training-time context controls by benchmark group

Matched training-time controls show that full Pythia improves over numerical-only learning in some MUSE groups but regresses in another, so gains are task-dependent. The related-work passages place this approach among numerical foundation models and multimodal forecasting methods.

G.2 Training-time context controls by benchmark group

  • Full Pythia reduces MASE/WSQL by 28.28%/27.51% on short-horizon CiK transport and 15.96%/16.32% on medium-horizon daily retail, but increases errors by 5.08%/7.52% on short-horizon ACL18 stock.These matched training-time controls measure changes to shared predictive learning, not per-window attribution; aggregate benefits coexist with task-dependent gains and regressions.Source: §G.2 · p. 35, Figure 16 · p. 35, Table 16 · p. 36

H.1 Numerical time-series foundation models

  • Numerical time-series foundation models use varied architectures and objectives, including Chronos tokenization, TimesFM’s decoder-only design, and Sundial’s flow matching.The related-work discussion also describes Chronos-2 as supporting univariate, multivariate, and covariate-informed forecasting.Source: §H.1 · p. 37

H.2 Language models and multimodal forecasting

  • Prior multimodal forecasting work combines text and numerical histories, aligns modalities, or integrates event and structured textual context through different approaches.Pythia’s described approach uses entity descriptions, events, and covariate names to refine numerical predictive states, with detached numerical error guiding contextual learning before readout training.Source: §H.2 · pp. 37–38

H.3 Forecasting datasets and benchmarks

The paper situates MUSE as its primary benchmark for contextual forecasting, then reviews JEPA objectives and temporal latent-world-model approaches that inform Pythia’s design.

H.3 Forecasting datasets and benchmarks

  • MUSE evaluates heterogeneous contextual inputs, domains, frequencies, and horizons across model families; Pythia uses its published baselines and test tasks without dataset-specific adaptation.The paper selects MUSE because contextual observations are central to Pythia’s prediction problem.Source: §H.3 · p. 38

H.4 JEPA objectives and target representations

  • I-JEPA and V-JEPA predict target embeddings rather than pixels, while VL-JEPA predicts continuous text embeddings and uses a separate decoder to produce text.VICReg regularizes embedding variance and covariance, C-JEPA combines it with image-based JEPA, and SALT uses masked latent prediction with a separately pretrained frozen teacher; Pythia preserves a pretrained target space by default and studies initialization adaptation.Source: §H.4 · p. 38
  • Theoretical analyses link latent prediction to predictive features and to temporal dynamical-regime clustering associated with Koopman-invariant representations and a near-identity linear predictor.The cited studies connect latent prediction to information retained in learned representations.Source: §H.4 · p. 38

H.5 Temporal JEPA and latent world models

  • LaT-PFN predicts future embeddings in an EMA-defined target space, while Pythia uses a pretrained numerical foundation model to define its target space and external observations to condition predictive states.Both separate latent prediction from observation-space decoding; Pythia freezes predictive modules before separately adapting a pretrained quantile decoder.Source: §H.5 · pp. 38–39
  • TS-JEPA, CHARM, and SC-JEPA explore masked numerical prediction, channel-text conditioning, and multi-resolution prediction with a soft codebook, respectively.Their downstream settings include classification and regression, transfer, and anomaly anticipation.Source: §H.5 · p. 39
  • Prior world-model approaches span robotic planning, imagined embodied behavior, physiological signals with clinical interventions, and physics-constrained multivariate forecasting; Pythia separates numerical prediction from context-driven correction in a pretrained numerical encoder’s space.The cited examples are V-JEPA 2 [Assran et al., 2025], GenRL [Mazzaglia et al., 2024], NormWear-2 [Luo et al., 2026], and Phys-JEPA [Nie et al., 2026].Source: §H.5 · p. 39
Loading 2610.05240v1…