Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

Vijay Ekambaram; Arindam Jati; Pankaj Dayama; Sumanta Mukherjee; Nam H. Nguyen; Wesley M. Gifford; Chandra Reddy; Jayant Kalagnanam

Source-linked AI summary

Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam H. Nguyen, Wesley M. Gifford, Chandra Reddy, Jayant Kalagnanam

arXiv:2401.03955v8Jan 8, 2024cs.LGcs.AI

TL;DR

Existing large time-series foundation models provide strong zero/few-shot forecasting but are computationally expensive and often omit cross-channel and exogenous information. TTM addresses this with compact, resolution-aware pre-training and multi-level fine-tuning, reporting state-of-the-art zero/few-shot forecasting with lower computational requirements. Its main scope boundary is that different context lengths require separately trained variants.

Problem
Large time-series pre-trained models demand substantial computation, while many lack explicit support for cross-channel correlations and exogenous variables.
Method
TTM combines a lightweight TSMixer backbone with adaptive patching, diverse resolution sampling, resolution prefix tuning, and multi-level fine-tuning for channel and exogenous modeling.
Results
TTM achieves state-of-the-art zero/few-shot forecasting with models starting at 1M parameters while providing significant computational efficiency on heterogeneous datasets.
Takeaways & Limitations
TTM demonstrates that compact pre-trained time-series models can transfer across diverse datasets while reducing model size and computational requirements.
Takeaways & Limitations
TTM requires different trained models for different context-length settings because its non-transformer backbone is sensitive to context length.

Abstract

from arXiv · show

Large pre-trained models excel in zero/few-shot learning for language and vision tasks but face challenges in multivariate time series (TS) forecasting due to diverse data characteristics. Consequently, recent research efforts have focused on developing pre-trained TS forecasting models. These models, whether built from scratch or adapted from large language models (LLMs), excel in zero/few-shot forecasting tasks. However, they are limited by slow performance, high computational demands, and neglect of cross-channel and exogenous correlations. To address this, we introduce Tiny Time Mixers (TTM), a compact model (starting from 1M parameters) with effective transfer learning capabilities, trained exclusively on public TS datasets. TTM, based on the light-weight TSMixer architecture, incorporates innovations like adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle pre-training on varied dataset resolutions with minimal model capacity. Additionally, it employs multi-level modeling to capture channel correlations and infuse exogenous signals during fine-tuning. TTM outperforms existing popular benchmarks in zero/few-shot forecasting by (4-40%), while reducing computational requirements significantly. Moreover, TTMs are lightweight and can be executed even on CPU-only machines, enhancing usability and fostering wider adoption in resource-constrained environments. The model weights for reproducibility and research use are available at https://huggingface.co/ibm/ttm-research-r2/, while enterprise-use weights under the Apache license can be accessed as follows: the initial TTM-Q variant at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r1, and the latest variants (TTM-B, TTM-E, TTM-A) weights are available at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r2.

1 Introduction

TTM addresses the high computational demands and limited multivariate modeling of large pre-trained time-series models with a compact architecture designed for heterogeneous resolutions. It combines lightweight modeling innovations with transfer learning and reports advantages in model size, runtime, and accuracy.

Motivation: Large pre-trained time-series models achieve strong zero-shot forecasting but often require hundreds of millions to billions of parameters and substantial computational resources.Recent models include Moment, TimesFM, Chronos, Moirai, and Lag-Llama, alongside LLM-based approaches.
TTM proposal: TTM starts at 1M parameters and supports zero/few-shot multivariate forecasting with channel correlations and exogenous signals.It is based on lightweight TSMixer blocks that avoid quadratic self-attention costs.
TTM proposal: Adaptive patching, diverse resolution sampling, and resolution prefix tuning enable resource-constrained pre-training across heterogeneous time-series resolutions.These enhancements target datasets differing in domains, sampling rates, lengths, and channel counts.
TTM proposal: TTM uses multi-level modeling to pre-train independently by channel and later incorporate target-channel correlations and exogenous channels during fine-tuning.This addresses capabilities described as lacking in many existing pre-trained time-series models.
Reported benefits: TTM is reported to outperform popular benchmarks across size, runtime, and accuracy while supporting inference and fine-tuning on CPU-only environments.Figure 1 summarizes model size, per-batch CPU inference time, and relative forecast accuracy.

2 TTM Components

The forecasting setup distinguishes target variables from exogenous variables and uses a lightweight TSMixer-derived backbone to process normalized, patched multivariate histories. TTM’s components are designed to mix information efficiently while accommodating multichannel inputs.

Forecasting setup: Multivariate forecasting predicts future values for multiple channels from historical context, with target variables forecasted and exogenous variables influencing those forecasts.Exogenous variables may have known or estimated values throughout the forecast horizon.
TTM architecture: TTM uses TSMixer-derived MLP blocks and gated attention to mix features within patches, across patches, and across channels with low computational requirements.The backbone is part of a four-component multi-level architecture.
Input processing: Before backbone processing, each channel is instance-normalized and the history is divided into non-overlapping patches.Normalization is reversed before loss computation, while patching preserves local temporal information.

3 TTM Methodology

TTM pre-trains on diverse public time-series data with a direct forecasting objective, then adapts to target domains through zero-, few-, or full-shot workflows. Its backbone and resolution modules support heterogeneous data, while fine-tuning adds channel and exogenous-signal modeling.

Pre-training: TTM pre-trains channel-independently on diverse public datasets using a direct forecasting objective and mean squared error over the forecast horizon.The resulting model captures common temporal dynamics and seasonal patterns from the pre-training data.
Multi-resolution pre-training: TTM targets small models that generalize across datasets with varying resolutions despite the risk of under-fitting from limited capacity.The methodology introduces backbone enhancements specifically for resource-constrained pre-training.
Adaptive patching: Adaptive patching changes patch lengths and patch counts across backbone levels to improve generalization across resolutions and granularities.Patch partitioning and merging reshape representations between levels.
Diverse resolution sampling: Diverse resolution sampling balances high- and low-resolution data by deriving lower-resolution datasets through averaging or decimation while retaining the original data.For example, a 4-second dataset can produce minutely and hourly versions.
Resolution prefix tuning: Resolution prefix tuning adds a learnable resolution-specific embedding to provide an explicit signal for resolution-conditioned modeling.Each input resolution is mapped to an integer and embedded into the model’s hidden dimension.
Fine-tuning: Fine-tuning supports zero-shot evaluation, few-shot updates using 5-10% of target training data, and channel mixing for multivariate targets.The backbone remains frozen while the TTM head is adapted to the target domain.
Exogenous modeling: The exogenous mixer replaces forecasted exogenous channels with known future values and learns lagged cross-channel dependencies before producing target forecasts.It uses overlapping windows and a TSMixer block with channel mixing enabled.

4 Experiments and Results

Across zero-shot, few-shot, head-probing, cross-channel, and ablation evaluations, compact TTM variants achieve strong forecasting performance while using limited model capacity and diverse-resolution design choices.

Benchmark setup: TTM is evaluated against 24 open-sourced state-of-the-art forecasting models spanning pre-trained, LLM-based, self-supervised, and alternative architectures.The evaluation includes models such as Lag-Llama, TimesFM, Moirai, Chronos, Moment, GPT4TS, Time-LLM, PatchTST, TSMixer, and TimeMixer.
Benchmark setup: TTM variants range from 1M to 5M parameters and are pre-trained on approximately 1B samples, with the primary variants requiring 24–30 hours on six A100 GPUs.TTM-Base uses 1M parameters, TTM-Enhanced 4M, and TTM-Advanced 5M.
Zero-shot forecasting: TTMA outperforms Moirai by 4–10% and TimesFM by 19% in zero-shot forecasting despite being 3–62X smaller than Moirai variants and 40X smaller than TimesFM.The comparisons use the reported zero-shot evaluation settings and benchmark variants.
Zero-shot forecasting: TTMB outperforms Chronos by 17–32% and Lag-Llama by 40% in zero-shot forecasting while using substantially fewer parameters.TTMB is reported as 8–709X smaller than Chronos and 2–3X smaller than Lag-Llama.
Few-shot and head-probing performance: In the few-shot 5% setting, TTMB surpasses GPT4TS by 15% and Time-LLM by 10%, while TTM also outperforms Moment by 3–4% in full-shot head probing.Few-shot fine-tuning trains only with 5% of the training data, whereas head probing freezes the backbone and fine-tunes the model head.

5 Conclusions and Future Work

TTM is presented as a lightweight multivariate forecasting model designed to transfer across datasets while modeling cross-channel relationships and exogenous variables. The paper positions it as a faster, more resource-efficient alternative to large pre-trained and transformer-based approaches.

Conclusions and Future Work: TTM is an extremely lightweight pre-trained model for multivariate time-series forecasting.It is designed for efficient pre-training and fine-tuning workflows.
Conclusions and Future Work: TTM captures cross-channel relationships and exogenous variables that popular methods often lack.These capabilities are relevant to multivariate forecasting settings involving interrelated series and external signals.
Conclusions and Future Work: Existing forecasting models generally require in-domain training, limiting transfer to out-of-domain datasets with zero or minimal training.Time-series transfer is difficult because public pre-training data are limited and datasets differ in domain, channels, resolution, and length.
Conclusions and Future Work: Prior transfer studies often evaluate carefully matched source-target dataset pairs, whereas general pre-trained models aim to transfer across new target datasets.Examples include transfers between similar electricity-transformer or biomedical datasets.

C.1 List of Pre-training Datasets:

TTM pre-training uses public Monash and LibCity collections, with chronological splits and moving-window sample construction. The pre-training datasets are kept separate from evaluation datasets.

List of Pre-training Datasets: Temporal cross-validation chronologically divides the time series into training and validation parts before moving-window sample generation.The resulting windows have lengths s_l and f_l.
List of Pre-training Datasets: The pre-training datasets do not overlap with the evaluation datasets.The Australian electricity-demand and weather datasets used for pre-training differ from the evaluation datasets in location, variables, type, resolution, and length.

C.2 List of Evaluation Datasets:

Evaluation covers seven standard multivariate datasets for zero-, few-, and full-shot forecasting plus four datasets containing exogenous or control variables. Experiments also specify dataset splits, model configurations, and computational measurement settings.

List of Evaluation Datasets: The D1 evaluation set contains seven public datasets: ETTH1, ETTH2, ETTM1, ETTM2, Weather, Electricity, and Traffic.These datasets are used for zero-, few-, and full-shot evaluation.
List of Evaluation Datasets: The ETT datasets contain seven channels, Weather contains 21 channels, Electricity contains 321 clients, and Traffic uses 862 sensors.The ETT datasets include hourly and 15-minute sampling intervals.
List of Evaluation Datasets: The D2 evaluation set adds Bike Sharing, Carbon Capture Plant, Application, and Service datasets to assess exogenous-channel information extraction.These datasets include exogenous or control variables relevant to forecasting.
List of Evaluation Datasets: In the Service and Application datasets, business KPIs are targets and IT events are exogenous variables.The forecasting task predicts business KPIs given the IT events.
List of Evaluation Datasets: Fine-tuning enables decoder channel mixing and the exogenous mixer block for datasets requiring cross-channel modeling.Batch size and head dropout are also adjusted by target dataset.
List of Evaluation Datasets: Computational comparisons report GPU and CPU inference time, maximum GPU memory, and parameter count under a fixed evaluation setup.The setup uses s_l = 512, f_l = 96, batch size 32, one A100 80GB GPU, and 16 CPU cores.

D.4 Baseline Implementation Details:

Baseline implementation details are documented separately in Table 10. The referenced material provides the location of those details but does not enumerate them here.

Baseline Implementation Details: Baseline algorithm implementation details are reported in Table 10.
Baseline Implementation Details: Table 10 is the designated source for the baseline implementation specifications.
Baseline Implementation Details: The supplied passage identifies the table without listing individual baseline settings.

E Sample Zero-shot Visualizations

This section presents zero-shot TTM forecasts across datasets and the complete benchmark-table context for TTM variants.

Zero-shot TTM forecasts capture complex trends and seasonal patterns across different datasets.
Table 11 reports zero-shot results for all TTM variants across sliding test windows on the D1 benchmark.
Tables 11 and 12 provide fine-grained D1 benchmark results for TTMQ, TTMB, TTME, and TTMA.

F.2 Full table for zero-shot experiment:

The zero-shot experiments compare TTM with other pre-trained and fully trained forecasting models across datasets, windows, and forecast lengths.

Tables 13 and 15 compare sliding-window and last-window zero-shot results across forecast lengths with Moirai, TimesFM, Chronos, and Lag-Llama.
26-36% improvement is reported for all three TTM variants over LLMTime, alongside a 70,000 to 14,000 reduction in model sizes.
TTM zero-shot performance outperforms the full-shot performance of popular SOTA models trained on the target datasets.

F.5 Full table for 5% few-shot experiment:

The few-shot experiments evaluate TTM across limited-data settings, compare it with transfer-learning baselines, and examine adaptive patching and resolution prefix tuning.

Table 16 reports 5% few-shot results across all forecast lengths and D1 datasets.
TTMQ outperforms SimMTM, Ti-MAE, TST, LaST, TF-C, CoST, and TS2Vec across 10% to 100% training-data settings on ETTH1.
Adaptive patching improves forecast accuracy in both pre-training-data settings, with a larger effect when less pre-training data is available.
Resolution prefix tuning generally improves forecasting, especially with abundant and diverse pre-training data or short context lengths.

G.1 Dataset preparation for TTM embedding analysis:

The embedding analysis examines TTM representations across datasets with different observation frequencies and analyzes decoder attention to exogenous features.

The analysis uses weather, electricity, and traffic datasets with observation frequencies of 10 minutes, 1 hour, and 1 hour, respectively.
Decoder channel mixing uses gated attention to derive feature contributions from mean attention weights across features and samples.

H Limitations and Future Work

TTM is currently limited to forecasting, fixed context-length settings, and point predictions. Future work targets broader downstream tasks, dynamic context adaptation, and probabilistic forecasting.

Scope: TTM currently supports forecasting only, with classification, regression, and anomaly detection identified as future extensions.Other pre-trained models such as Moment and GPT4TS have begun expanding to these downstream tasks.
Context lengths: TTM requires different models for different context-length settings because its non-Transformer architecture is sensitive to context length.The paper introduces three variants optimized for different context-length settings and plans automatic adaptation to dynamic lengths.
Forecasting outputs: TTM currently supports only point forecasting, whereas lag-llama and Moirai also support probabilistic forecasting.Future work proposes adding distribution heads for probabilistic forecasts.