PaperlayerView exact revision on arXiv

Source-linked AI summary

Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

Haoran Zhang, Haixuan Liu, Xingjian Su, Yong Liu, Zhi Chen, Yuxuan Wang, Jianmin Wang, Mingsheng Long

arXiv:2610.11734v1cs.LGcs.AI

TL;DR

Real-world time series combine complex dynamics with limited control over observed patterns, while synthetic data can be controlled but may miss real-system irregularities. Timer-M1 combines primitive-based data construction with forecasting episodes and gated two-dimensional Transformer blocks. It ranks first on FEV and TIME and second on GIFT-Eval among state-of-the-art foundation models.

  • Problem

    Real-world series have complex, noisy dynamics but limited pattern control, while synthetic datasets offer control but may omit real-system irregularities.

    Source: §1 · p. 1

  • Method

    Timer-M1 composes temporal and relational primitives into multivariate samples, organizes forecasting episodes by variate roles and available observations, and uses gated two-dimensional Transformer blocks.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    Timer-M1 ranks first on FEV and TIME and second on GIFT-Eval among state-of-the-art foundation models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support aligning data construction, forecasting episodes, and model architecture around shared primitives for general forecasting.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce Timer-M1, a pretrained multivariate time series foundation model that learns with primitives for zero-shot forecasting. Across domains, time series share elementary temporal and relational patterns, termed primitives, yet differ in how these primitives manifest and evolve across different contexts. Despite progress in zero-shot and task-general forecasting, existing foundation models may still struggle to generalize to complex real-world scenarios. To this end, we develop a primitive-based data synthesis and pretraining pipeline. The synthesis pipeline generates series with temporal primitives shared across domains and then assembles real and generated series into multivariate samples using relational primitives. Afterwards, samples are organized into episodes by assigning distinct channel roles as target variates, past-only covariates, and known-future covariates, ensuring that the model is optimized on predictable variates using available exogenous information. Technically, Timer-M1 further adapts gated two-dimensional Transformer blocks that dynamically allocate cross-variate attention across layers. Across three large-scale forecasting benchmarks, Timer-M1 ranks first on both FEV and TIME and second on GIFT-Eval among most recent time series foundation models. These results support effective primitive-based pretraining as a route to robust general forecasting technique across domains and task settings.

1 INTRODUCTION

Real series contain complex dynamics but offer limited control over observed patterns, while synthetic series are controllable but may omit real-world irregularities. Timer-M1 combines primitive-based data construction with forecasting episodes and gated two-dimensional Transformer blocks, and ranks first on FEV and TIME and second on GIFT-Eval.

1 INTRODUCTION

  • Real-world series offer complex, noisy dynamics but limited pattern control, whereas synthetic series are controllable but may omit real-system irregularities.This complementarity motivates combining observed dynamics with controllable relations.Source: §1 · p. 1
  • Primitives are recurring temporal and relational patterns shared across domains, including trends, periodicity, stochastic dependence, regime changes, and cross-variate relations.Relational examples include correlation, delayed responses, cointegration, and event effects.Source: §1 · pp. 1–2
  • The pipeline composes temporal and relational primitives from real and generated series into multivariate samples, then constructs forecasting episodes with specified target and observation roles.Repeated roles and masks are intended to encourage general capabilities rather than reliance on one task configuration.Source: §1 · p. 2, Figure 1 · p. 2
  • Timer-M1 combines primitive synthesis and episode-based pretraining with gated two-dimensional Transformer blocks that allocate cross-variate attention across layers.Episodes assign target, past-only covariate, and known-future covariate roles.Source: §1 · p. 2
  • Timer-M1 ranks first on FEV and TIME and second on GIFT-Eval among state-of-the-art foundation models.The model uses a primitive-based data pipeline and episode-based pretraining.Source: §1 · p. 2

2 RELATED WORK

Prior work explores both synthetic and real-data strategies for pretraining, alongside diverse forecasting objectives and cross-variate architectures. These lines of work motivate coordinating data construction with task design and observation availability.

2.1 TIME SERIES PRETRAINING DATA

  • Pretraining data research spans synthetic-task learning, real-series curation, and hybrid approaches that combine controlled generation with empirical diversity.The discussion motivates applying shared relation operators to real and generated series while retaining observed dynamics.Source: §2.1 · pp. 2–3

2.2 TIME SERIES FOUNDATION MODELS

  • Time series foundation models vary in forecasting objectives and representations, with recent architectures adding multivariate attention and other mechanisms for cross-variate interaction.The discussion motivates organizing primitive compositions into episodes with varied variate roles and observation masks.Source: §2.2 · p. 3

3 APPROACH

Timer-M1 aligns primitive-based data construction, forecasting episodes, and a gated two-dimensional Transformer to learn temporal and relational patterns from permitted observations. Real and generated series are assembled into multivariate records, then organized into forecasting tasks.

3.1 DATA

  • The data pipeline combines real and synthesized temporal patterns, derives cross-variate relations and transformed responses, and outputs aligned multivariate records while preserving original multivariate dependencies.Relation operators can create signed responses, shared drivers, and delayed effects; transforms diversify the observed forms of primitives.Source: §3.1 · pp. 3–4

3.2 ARCHITECTURE

  • Each block applies temporal attention, feed-forward processing, then gated variate attention; learned layer-wise gates regulate cross-variate updates while forecast tokens stay active despite hidden values.After valid-history normalization, patches embed normalized values together with observation masks.Source: §3.2 · pp. 4–5

3.3 PRE-TRAINING

  • Episodes specify history and horizon, target and covariate roles, observation availability, and future supervision, masking past-only covariates beyond the forecast origin while showing known-future inputs where permitted.A variate can take different roles across episodes, and generating annotations guide channel selection without revealing operator identities to the model.Source: §3.3 · p. 5
  • Training averages normalized pinball loss over quantiles and target variates, excludes unobserved supervision, and uses horizon-decaying weights to emphasize nearer forecast positions.Large-scale multivariate pretraining is followed by long-context post-training.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate forecasting on public benchmarks and controlled tasks covering temporal extrapolation, relational forecasting, and input reliability. Ablations examine adaptive cross-variate gates and joint real–synthetic data construction.

4.1 PUBLIC BENCHMARKS

  • Timer-M1 ranks first on FEV and TIME and second on GIFT-Eval among compared methods, with strong results across point and probabilistic metrics.FEV results cover 100 matched configurations, TIME 98, and GIFT-Eval 97.Source: §4.1 · p. 6, Figure 5 · p. 7, Figure 6 · p. 7

4.2 PRIMITIVE EVALUATION

  • Across 13 periodic cases, Timer-M1 achieves NMAE 0.02432, with long-period continuation and frequency drift illustrating extrapolation of recurring patterns.The reported result is consistent with reusable temporal primitives rather than simply repeating observed sequences.Source: §4.2 · pp. 7–8
  • Across six joint-target tasks, Timer-M1 achieves NMAE 0.02160, with promotion-driven targets and shared regime changes illustrating cross-variate forecasting.The evaluation exposes dependencies under different variate roles and observation availability.Source: §4.2 · p. 8
  • Timer-M1 reduces target-missingness NMAE by 32.5% relative to Chronos-2, with selected cases illustrating resilience to incomplete histories and misleading inputs.Observation masks and redundant or unrelated variates expose primitives under imperfect inputs.Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • Adaptive gates achieve FEV SQL skill 0.4887, versus 0.4836 with Fixed 0.202 and 0.4814 with Fixed 1.0, improving three of five task types.Learned gate values generally increase with depth, supporting layer-wise allocation of cross-variate interaction.Source: §4.3 · p. 9, Figure 9 · p. 9
  • The Full Recipe improves FEV skill and both GIFT-Eval metrics over Real Only, Synthetic Only, and No-Joint, and achieves the lowest TIME MASE.Its FEV skill exceeds No-Joint by 0.0162, suggesting benefits beyond source diversity alone; joint generation composes real and generated series through shared relation operators.Source: §4.3 · p. 9, Table 1 · p. 9

5 CONCLUSION

Timer-M1 combines primitive-based data construction, episode-based pretraining, and gated two-dimensional Transformer blocks. Evaluations show competitive forecasting performance and strengths in temporal extrapolation and relational forecasting; the authors identify areas for future expansion.

5 CONCLUSION

  • Timer-M1 combines observed dynamics with controlled generation, episode-based pretraining, and gated two-dimensional Transformer blocks to model temporal and cross-variate information.Episodes expose primitives under varied variate roles and observation availability.Source: §5 · p. 9
  • Evaluations on three public benchmarks and controlled primitive tasks demonstrate competitive forecasting performance and strengths in temporal extrapolation and relational forecasting.The authors conclude that data construction, forecasting episodes, and model architecture can be aligned around shared primitives.Source: §5 · p. 9
  • Future work will expand real-data coverage and primitive compositions to better represent irregular dynamics and partially observed dependencies.The authors also identify input-dependent cross-variate interaction and efficient long-context modeling as directions to explore.Source: §5 · p. 9
  • Generative AI tools were used for manuscript support and figure preparation, but not for research methodology, experiments, datasets, analysis, or result interpretation.The authors state that they reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A DATASET STATISTICS

The section describes how temporal and relational primitives, observed data, and generated series form multivariate records and forecasting episodes. It also covers model configuration, layer-wise gates, episode packing, and GroupMix.

A.1 TEMPORAL AND RELATIONAL PRIMITIVES

  • Temporal primitives govern evolution within a series, while relational primitives describe how sources are shared across variates through signed mixing and temporal offsets.The dependencies are specified during construction, rather than inferred as causal mechanisms from real observations.Source: §A.1 · p. 12

A.2 CONSTRUCTION AND OBSERVATION

  • Observed or generated sources are combined through relation operators and transformed into aligned multivariate records; response-only branches omit the original source channels.Source reuse creates shared variation, while transformations can rescale or clip resulting channels.Source: §A.2 · p. 12, A DATASET STATISTICS · p. 12
  • Episodes assign variates target, past-only covariate, or known-future covariate roles, hiding target futures while retaining permitted future covariates and masking observations and supervised targets.Hidden values remain distinguishable from observed zeros, and different role assignments expose the same record under different forecasting conditions.Source: §A.2 · pp. 12–13

A DATASET STATISTICS

  • Stored record, channel-series, and scalar-position counts describe data volume, not independent forecasting tasks or consumed training tokens.Episode sampling can reuse records under different horizons and observation roles.Source: §A.2 · p. 13

B.1 MODEL CONFIGURATION

  • The model uses factorized temporal and variate attention, with a temporal-attention and feed-forward sequence followed by gated variate attention.For C variates, N patches, and width D, factorized attention requires O(CN^2D + NC^2D) operations per layer, rather than O(C^2N^2D) for attention over flattened tokens.Source: §B.1 · p. 13, Table 3 · p. 13

B.2 LAYER-WISE GATES

  • Learned layer gates generally increase with depth, scaling variate-attention residual updates with local deviations rather than a uniform scale.The coefficients are shared across tokens in a layer and remain fixed at inference; their magnitudes alone do not measure information carried by either branch.Source: §B.2 · pp. 13–14, Figure 10 · p. 14

B.3 EPISODES AND PACKING

  • Greedy bin-packing keeps episodes intact and uses separate attention groups, improving batch utilization without changing permitted observations or prediction targets.The heuristic does not require an exact bin-packing solution.Source: §B.3 · p. 14
  • GroupMix changes training-time attention grouping to expose selected episodes to additional variates while retaining sequence values, roles, and supervision targets.It is not enabled for reported results; its potential regularization effect is distinct from the measured gate and data-construction ablations.Source: §B.3 · p. 14

B.4 PUBLIC BENCHMARK PROTOCOLS

The public benchmark evaluation follows the official task definitions and scoring conventions, with forecast inputs determined by each task. It involves no task-specific parameter updates, and training-data exposure is not matched across compared methods.

B.4 PUBLIC BENCHMARK PROTOCOLS

  • Forecast origins, horizons, and permitted observations follow benchmark task definitions, including FEV’s distinction between historical and future-known covariates.Public reference scores are fixed comparison snapshots, not continuously updated rankings.Source: §B.4 · p. 15
  • No task-specific parameter updates are performed during evaluation, and training-data exposure is not matched across compared methods.Compliance with benchmark task definitions and metrics is distinct from training-data eligibility.Source: §B.4 · p. 15
  • FEV reports separately clipped MASE and SQL skill, while TIME and GIFT-Eval report normalized MASE and quantile-based CRPS.Higher skill and lower normalized error indicate better forecasts; missing matched results remain unavailable rather than receiving zero error.Source: §B.4 · p. 15

B.5 PRIMITIVE EVALUATION PROTOCOL

The protocol evaluates temporal extrapolation, relational forecasting, and input reliability under specified conditions, with models receiving the same permitted inputs within each setting. It defines group-level scoring and cautions against interpreting selected groups as a single composite or as evidence of anticipation of unobserved changes.

B.5 PRIMITIVE EVALUATION PROTOCOL

  • Temporal tests use 256 history points and 128 forecast steps to probe periodicity, trends, and shifts, including boundary changes with no advance indication in history.Boundary shifts are stress tests, not evidence that a model can anticipate an unobserved intervention.Source: §B.5 · p. 15, Table 4 · p. 16
  • Relational tests use 256 history points and 128 forecast steps to probe signed, delayed, phase-shifted, scale-varying, cointegrated, and shared-regime relations.Cross-model comparisons assess forecasts with specified inputs, not the gain from joint rather than independent prediction.Source: §B.5 · p. 15
  • Input-reliability tests use 512 history points and 128 forecast steps while varying missingness, corruption, irrelevant inputs, event indicators, or variate order.Missingness results concern forecasting after common interpolation or forward-fill preprocessing, not a separate imputation task.Source: §B.5 · p. 15
  • Group scores average records using a fixed reference scale shared across models, and record counts need not equal independent generated trajectories.The short-window temporal and relational groups use the complete generated trajectory range; reliability tests use the clean historical reference range with a positive floor.Source: §B.5 · p. 16, Table 4 · p. 16
  • MSE and NMAE are not interchangeable, and the selected capability groups do not imply a cross-capability composite score.Unsupported interfaces and unavailable metrics remain unranked rather than receiving an artificial penalty.Source: §B.5 · p. 16

C.1 PUBLIC BENCHMARKS

The public-benchmark comparison reports aggregate results across matched configurations from FEV, TIME, and GIFT-Eval. The benchmarks use different score directions and cover varied forecasting settings.

C.1 PUBLIC BENCHMARKS

  • The public-benchmark numerical aggregates cover 100 FEV, 98 TIME, and 97 GIFT-Eval configurations; scores retain original benchmark direction and normalization, and ranks refer to displayed methods, not official zero-shot leaderboard placement.Source: §C.1 · p. 17
  • FEV reports skill scores, whereas TIME and GIFT-Eval report normalized errors; the aggregate table marks best and second-best displayed scores per column.Dashes indicate unavailable matched results.Source: Table 5 · p. 17
  • FEV evaluates heterogeneous variate roles, TIME emphasizes task-specific forecasting configurations, and GIFT-Eval spans varied temporal resolutions and forecast lengths.The metrics are reported separately rather than combined into an arbitrary point-and-distributional accuracy composite.Source: §C.1 · p. 17

C.2 FEV TASK TYPES

The FEV task-type breakdown separates forecasting configurations by variate roles and aggregates skill across configurations. Primitive-group tables provide selected diagnostics, not an exhaustive inventory or an overall ranking of the full suite.

C.2 FEV TASK TYPES

  • FEV partitions all 100 configurations into five mutually exclusive groups, separating target-only tasks from past-only and known-future covariate settings.The breakdown distinguishes temporal forecasting from inference using related histories and conditioning on observations available over the forecast interval.Source: §C.2 · p. 18, Table 6 · p. 18, Table 7 · p. 18
  • Overall FEV skill is recomputed from configuration-level errors rather than an unweighted mean of the five group scores.Group sizes differ, and the aggregation retains both point and probabilistic scores.Source: §C.2 · p. 18

C.3 PRIMITIVE GROUPS

  • Tables 8–10 show selected temporal, relational, and input-reliability groups, retaining records and compared models within each displayed group.The reported mean squared error is averaged over complete records, with lower values better.Source: §C.3 · p. 19, Table 8 · p. 19, Table 9 · p. 19
  • The displayed diagnostic groups are not exhaustive, and no overall rank over the full suite is inferred from them.Scoring-record counts may include multiple views of the same generated task.Source: §C.3 · p. 19
  • Dashes mark unsupported inputs or unavailable squared-error results, not failed forecasts assigned an artificial penalty.Public-benchmark scores and controlled TimesFM-3 tests use their respective evaluation settings.Source: §C.3 · p. 19

C.4 ABLATION RESULTS

The ablations compare adaptive and fixed gate controls and alternative data-construction routes. Adaptive gates have the highest overall FEV skill, with their clearest advantage in known-future forecasting.

C.4 ABLATION RESULTS

  • Gate controls are independently trained with fixed residual multipliers, while adaptive gates are learned per layer and remain fixed during inference.The comparisons are not inference-only interventions on a shared checkpoint.Source: §C.4 · p. 20
  • Adaptive gates achieve the highest overall FEV skill, with the clearest advantage in known-future forecasting.The table compares skill scores across overall FEV and five task types.Source: Table 11 · p. 20
  • Data-construction variants compare generation routes while keeping shared GIFT-Eval and TIME training collections unchanged.The ablations do not isolate individual operators, and variant names do not describe the provenance of all training observations.Source: §C.4 · p. 20, Table 12 · p. 20

D PRIMITIVE EVALUATION SHOWCASE

The showcase presents selected forecasts illustrating temporal patterns, cross-variate relations, and input conditions. Its examples are illustrative rather than a random sample for estimating overall success rates.

D PRIMITIVE EVALUATION SHOWCASE

  • The appendix presents 24 selected forecasts, eight for each of three capabilities, with a common target and forecast origin within each panel.Dashed boundaries separate history from the forecast interval; shading shows marginal 10–90% prediction quantiles, not confidence intervals for the mean.Source: D PRIMITIVE EVALUATION SHOWCASE · p. 21
  • Figure 11 compares temporal extrapolation examples involving periodicity, modulation, frequency drift, and observed regime changes.Lower strips show Timer-M1’s absolute forecast error, and each panel has its own value scale.Source: Figure 11 · p. 21
  • Figure 12 covers shared, signed, delayed, phase-shifted, cointegrated, and event-driven relations.Lower strips show available related inputs or target histories, not additional forecast targets unless the task specifies them.Source: Figure 12 · p. 22
  • Figure 13 illustrates input-reliability conditions involving missing histories, imperfect event information, irrelevant inputs, and historical outliers.Lower strips show Timer-M1’s absolute forecast error.Source: Figure 13 · p. 23
Loading 2610.11734v1…