Source-linked AI summary
UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting
Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, Roger Zimmermann
TL;DR
Multivariate forecasting needs models that generalize across domains despite differences in variables, temporal distributions, and convergence rates. UniTime combines flexible inputs, domain instructions, a Language-TS Transformer, and masking to address these challenges, with experiments reporting state-of-the-art forecasting and zero-shot transferability.
Problem
Existing forecasting approaches commonly train dedicated models per domain, while unified learning must handle varying characteristics, domain confusion, and convergence-speed imbalance.
Method
UniTime flexibly handles varying time-series characteristics, uses domain instructions and a Language-TS Transformer for domain identification and modality alignment, and applies masking.
Results
UniTime achieves new state-of-the-art performance on popular forecasting benchmarks and demonstrates transferability to unseen domains.
Takeaways & Limitations
The evaluations support UniTime as a step toward unified or foundation-model-style forecasting across diverse time-series application domains.
Abstract
from arXiv · showhide
Multivariate time series forecasting plays a pivotal role in contemporary web technologies. In contrast to conventional methods that involve creating dedicated models for specific time series application domains, this research advocates for a unified model paradigm that transcends domain boundaries. However, learning an effective cross-domain model presents the following challenges. First, various domains exhibit disparities in data characteristics, e.g., the number of variables, posing hurdles for existing models that impose inflexible constraints on these factors. Second, the model may encounter difficulties in distinguishing data from various domains, leading to suboptimal performance in our assessments. Third, the diverse convergence rates of time series domains can also result in compromised empirical performance. To address these issues, we propose UniTime for effective cross-domain time series learning. Concretely, UniTime can flexibly adapt to data with varying characteristics. It also uses domain instructions and a Language-TS Transformer to offer identification information and align two modalities. In addition, UniTime employs masking to alleviate domain convergence speed imbalance issues. Our extensive experiments demonstrate the effectiveness of UniTime in advancing state-of-the-art forecasting performance and zero-shot transferability.
1 INTRODUCTION
UniTime addresses the challenge of building a unified forecasting model across domains with different data characteristics and temporal distributions. It combines flexible modeling, domain instructions, cross-modal alignment, and masking to support cross-domain forecasting and transfer.
- Multivariate time series forecasting supports web applications including recommendations, economic modeling, microservice-log analysis, and threat early-warning systems.
- Dedicated domain-specific models may overlook the benefits of a unified model that generalizes across time series application domains.
- Cross-domain learning can leverage diverse data to learn shared temporal properties such as recurring patterns and evolving trends.
- Cross-domain modeling must address varying channels and sequence lengths, domain confusion, and imbalanced convergence speeds.
- UniTime accommodates varying input and output characteristics, uses human-crafted domain instructions with a Language-TS Transformer, and applies masking to mitigate convergence imbalance.
- Experiments report new state-of-the-art forecasting performance and transferability to unseen domains.
2 RELATED WORK
Prior work includes deep and Transformer-based forecasting models and recent attempts to use pretrained language models across modalities. UniTime is positioned against approaches that process time series with language models without fully exploiting textual information for cross-domain learning.
- Deep learning and Transformer-based models have shown strong promise for time series forecasting, while Transformer self-attention introduces computational and memory complexity.
- The paper header identifies UniTime as a language-empowered unified model for cross-domain time series forecasting.
- Recent cross-modality research applies pretrained language models to recommendation systems, graph learning, and time series modeling.
- GPT4TS demonstrates language-model-based time series forecasting but primarily uses time series data alone, limiting its exploitation of language processing for cross-domain learning.
3 PRELIMINARIES
The paper formulates cross-domain forecasting for multivariate time series whose channel counts, historical windows, and future prediction ranges may vary by domain. It also contrasts channel mixing with channel independence as a modeling consideration.
- Problem Definition: An observation from domain τ at time t is a multivariate vector with c_τ channels or variables.
- Problem Definition: Cross-domain forecasting allows domain-specific lookback windows L_τ and future prediction ranges T_τ.
- Problem Definition: The model input is a lookback sequence X^τ_{L_τ} in R^{L_τ×c_τ}, while the output forecasts T_τ future vectors in R^{T_τ×c_τ}.
- Channel-Mixing v.s. Channel-Independence: Channel-mixing models embed all channels jointly into a hidden space for multi-channel information fusion.
- Channel-Mixing v.s. Channel-Independence: Varying channel counts across domains make a shared channel-mixing encoder difficult to train.
4 THE UNITIME MODEL
UniTime is an end-to-end cross-domain model that tokenizes time series, combines them with domain instructions in a Language-TS Transformer, and generates forecasts despite varying input and output characteristics.
- Model overview: UniTime comprises a time series tokenizer, a Language-TS Transformer, and a decoder for prediction generation.The tokenizer preprocesses raw signals; the Transformer identifies domains and aligns text with time series.
- Time series tokenizer: Patching aggregates adjacent time points into tokens, capturing local semantic information while reducing computational overhead for long sequences.The tokenizer preprocesses signals through masking, stationarization, and padding before segmentation.
- Time series tokenizer: A shared linear projection embeds tokens into a common hidden space, while token stride adapts to each domain’s historical observation length.Token size is fixed across domains, whereas stride depends on domain-specific history lengths.
- Masking and gated fusion: Masking addresses unequal domain convergence rates by making the model depend on only partial time-series information during learning.Binary masks both hide selected raw signal steps and provide indicators of masked positions after padding and patching.
- Language-TS Transformer: Domain instructions provide explicit identification information, while the Language-TS Transformer aligns time series from different input spaces with language representations.The instructions are human-crafted descriptions of each domain, and GPT2 provides the causal Transformer backbone.
- Decoder: UniTime pads variable-length Transformer outputs to a maximum token length before a lightweight Transformer and linear layer generate forecasts up to a maximum predictive length.For domains requiring fewer predictions, the generated sequence is truncated to the domain’s predictive length.
5 EXPERIMENTS
Experiments evaluate UniTime across diverse forecasting datasets, compare it with cross-domain and dataset-specific baselines, and analyze its design choices, training behavior, and zero-shot transferability.
- Experimental Setup: UniTime is evaluated on eight real-world benchmark datasets spanning diverse time-series application domains and data characteristics.The datasets differ in variable semantics, sampling frequency, and collected-data size.
- Experimental Setup: The comparison includes eight state-of-the-art forecasting methods, all originally trained separately for each dataset and predictive length.The baselines are Informer, Autoformer, FEDformer, NSformer, DLinear, TimesNet, PatchTST, and GPT4TS.
- Main Results: UniTime achieves the best performance in 79 of 80 entries against baselines trained across datasets.For shorter horizons, evaluation uses the first values of UniTime’s 720-value output.
- Ablation Studies: Removing domain instructions significantly reduces performance, while t-SNE representations change from mixed domains to clear clustering when instructions are included.The clusters for the four ETT subsets remain close because they share a domain and underlying temporal characteristics.
- Ablation Studies: Without masking, several datasets overfit after the fourth epoch at different rates; masking produces later, more stable convergence and improves overall model selection.The most severe degradation without masking occurs on Illness.
- Ablation Studies: Removing the light Transformer or reconstruction loss reduces overall performance, and removing all listed designs degrades performance across datasets.Both components contribute to the complete model’s performance.
- Zero-Shot Transferability Analysis: UniTime generally outperforms baselines in zero-shot testing, with all three zero-shot datasets selecting instructions derived from ETTh1.The paper associates this selection with shared or related underlying patterns across the datasets.
6 CONCLUSION
The paper proposes UniTime as a unified forecasting model for diverse time series domains. Extensive evaluations report improved state-of-the-art forecasting performance and zero-shot transferability.
- UniTime targets unified forecasting across diverse time series application domains.
- The paper identifies challenges in constructing a unified cross-domain forecasting model and proposes UniTime to address them.
- Extensive evaluations confirm improved state-of-the-art forecasting performance and zero-shot transferability.
A MORE DISCUSSION ON LANGUAGE MODELS EMPOWERED TIME SERIES FORECASTING
The discussion motivates universal forecasting for workloads spanning many heterogeneous time series, contrasting it with dedicated-model approaches. UniTime is reported to improve knowledge sharing across datasets and zero-shot transfer performance, while GPT4TS is not treated as a unified model without modification.
- Language models and time series forecasting both analyze sequences to identify patterns and predict future elements, with recency emphasized in both tasks.
- Universal forecasting is motivated by cloud workloads whose diverse characteristics and lengths make training or tuning one model per series infeasible.
- UniTime improves 37 of 80 entries to new state-of-the-art results against separately trained dataset baselines and improves zero-shot transfers.
- GPT4TS is not regarded as a unified cross-domain model because its original implementation trains dedicated models for each dataset and predictive length.
B TRAINING CONFIGURATIONS
The training configurations partition datasets chronologically and vary batch sizes according to dataset size. Illness is additionally oversampled to increase exposure to an underrepresented domain.
- Datasets are split chronologically, using 6:2:2 for ETT series and 7:1:2 for the other datasets.
- Batch sizes are larger for datasets with more training samples and smaller for datasets with fewer samples.
- Illness, containing only 600 training samples, is oversampled 12 times to increase exposure to the underrepresented domain.
C MORE RESULTS FOR VARIANTS OF INSTRUCTIONS
The instruction-variant experiments test whether UniTime remains effective across different instruction lengths, rephrasings, and language-model fine-tuning settings. Results show limited performance differences across tested variants and reasonable domain discrimination even with a frozen language model.
- Instruction design: Domain instructions are sentences describing the data in each domain.
- Instruction design: Instruction variants include shortened, expanded, detailed, and randomly selected rephrasings generated from dataset information.
- Fine-tuning settings: The experiment compares frozen-language-model training with training in which all UniTime parameters, including the language model, are tunable.
- Results: Across five instruction sets with varying lengths, fine-tuning generally outperforms no fine-tuning, but performance differences are not notably significant.
- Results: Across instruction rephrasings, performance discrepancies are not notably significant, indicating stability to the tested rephrasing approaches.
D MORE RESULTS FOR UNITIME BACKBONE
UniTime performs better with GPT-2 than T5 as its backbone, potentially because GPT-2’s causal masking preserves temporal order.
- T5 does not surpass GPT-2 in UniTime’s backbone comparison.Table 9 reports average results over all predictive lengths.
- GPT-2’s causal masking preserves input temporal order, potentially benefiting textual and time series data.T5 instead uses bidirectional attention mechanisms.
E HYPERPARAMETER STUDIES
The hyperparameter study finds that a mask ratio of 0.5 and six Language-TS Transformer layers generally provide the best performance, while larger layer counts are limited by GPU memory.
- A mask ratio of 0.5 generally produces the best performance.The model generally performs better with larger mask ratios than smaller ones.
- Six Language-TS Transformer layers appear to be the most favorable choice.The study does not evaluate seven layers because of GPU memory limitations.