UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting

• Multivariate forecasting supports web services including recommendations, economic modeling, microservice-log analysis, and early warning systems.The paper frames forecasting as a tool for anticipating changing patterns and user behaviors in the web ecosystem.Source: §1 · p. 1
• A unified model can leverage data from diverse domains to learn shared temporal patterns, potentially improving prediction, transfer to unseen domains, and deployment.The introduction contrasts this approach with dedicated models trained separately for each domain or dataset.Source: §1 · pp. 1–2
• Cross-domain learning faces variable channel counts and sequence lengths, domain confusion, and differing convergence rates that can lead to overfitting or compromised performance.Existing rigid model designs can limit generalization across domains; simple, regular domains may converge quickly while others require more training.Source: §1 · p. 2
• UniTime accommodates varying time-series characteristics, uses domain instructions and a Language-TS Transformer to align modalities, and applies masking to mitigate convergence imbalance.The model aligns time series from different input spaces with the common latent space of language models; masking limits trivial solutions such as memorizing patterns in domains prone to overfitting.Source: §1 · p. 2
• UniTime reports new state-of-the-art performance on popular forecasting benchmarks and transferability to unseen domains.The introduction presents it as the first attempt, to the authors’ knowledge, to explore a unified model for generalization across time-series application domains.Source: §1 · p. 2

Deep Models for Time Series Forecasting

• Transformer variants reduce attention costs, decomposition and non-stationary methods capture temporal patterns, and versatile frameworks address multiple tasks, yet methods remain separate by domain or dataset.Efficiency approaches include LogTrans, Reformer, Informer, and Pyraformer; temporal-pattern approaches include Autoformer, ETSformer [36], FEDformer, and NSformer. TimesNet targets multiple time-series tasks, while GPT4TS uses pretrained language models.Source: §2 · p. 2

Language Model Powered Cross-Modality Learning

• Language-model methods use instructions for recommendation and molecule tasks, while GPT4TS forecasts with time-series data alone and does not fully exploit language capabilities for cross-domain learning.Instruction-based examples include InstructRec [41] and GIMLET; GPT4TS demonstrates language-model processing of time series but primarily relies on a single modality.Source: §2 · p. 3

Problem Definition

• A time-step observation in domain τ is a multivariate vector with c_τ channels or variables.The observation is represented as x_t^τ ∈ R^c_τ.Source: §3 · p. 3
• Cross-domain forecasting permits lookback length L_τ and prediction range T_τ to vary across domains, with historical inputs and future predictions represented as sequences.The input and output have shapes R^(L_τ×c_τ) and R^(T_τ×c_τ), respectively.Source: §3 · p. 3

Channel-Mixing v.s. Channel-Independence

• Channel independence processes each channel separately, avoiding channel-mixing models’ variable-count constraints and impractical shared embeddings across domains with distinct channel semantics.The paper adopts this configuration, recently introduced in PatchTST, for greater cross-domain flexibility; channel-mixing models are described in.Source: §3 · p. 3

4 THE UNITIME MODEL

• UniTime’s three components are a time series tokenizer, a Language-TS Transformer for domain identification and text–time-series alignment, and a decoder that generates predictions.The model is described from the perspective of a univariate series under a channel-independence setting.Source: §4 · p. 3

4.1 Time Series Tokenizer

• Patching adjacent time points into tokens captures local semantics and reduces computation for long sequences, following ViT and PatchTST.Before patching, the series is masked, stationarized to mitigate distribution shifts, and padded by duplicating its last value.Source: §4.1 · p. 3
• A shared learnable linear projection embeds tokens into the Transformer’s hidden space; token length is fixed across domains, while stride adapts to each domain’s history length.The stride determines the non-overlapping distance between consecutive token starting points.Source: §4.1 · p. 3
• Masking addresses uneven domain convergence by limiting input, discouraging memorization in fast-converging domains and encouraging robust, generalizable representations.A binary mask marks time steps to hide at a zero ratio specified by r_m; it masks raw signals and is also padded and patched to indicate masked positions to the model.Source: §4.1 · p. 3

Language-TS Transformer

• Projected domain-instruction tokens are gated with time-series tokens to help the model identify which information can support predictions.The gate uses learnable parameters and a sigmoid function.Source: §4.1 · p. 4

4.2 Language-TS Transformer

• Human-crafted domain instructions provide identification information to address domain confusion, while the Language-TS Transformer aligns time series from different input spaces into the language model’s latent space.The instructions also incorporate human prior knowledge about each domain’s data.Source: §4.2 · p. 4
• Pretrained GPT2 causal attention processes domain instructions before time-series data, giving the model contextual identifiers while preserving temporal order.The model uses GPT2 because time series are autoregressive; placing instructions first lets the time-series tokens access them under causal masking.Source: §4.2 · p. 4

4.3 Decoder

• Varying input-token and prediction lengths make a direct linear decoder impractical; UniTime pads to a maximum token length, processes the result with a lightweight Transformer, and predicts up to a maximum horizon.The model truncates predictions to the domain’s required horizon when it is shorter than the maximum predictive length.Source: §4.3 · pp. 4–5

4.4 Model Training

• Training combines mean squared error for forecasting and history reconstruction with random single-domain batches and oversampling to mitigate unstable learning, forgetting, and underrepresentation.Each batch contains one domain because channel counts and sequence lengths vary across domains.Source: §4.4 · p. 5

5.1 Experimental Setup

• Across eight real-world benchmark datasets, the experiments cover diverse domains and data characteristics, and compare UniTime with eight forecasting baselines.The datasets differ in variable count, variable semantics, sampling frequency, and data size; the baselines originally train dedicated models for each dataset and predictive length.Source: §5.1 · p. 5

5.2 Main Results

• UniTime achieves the best performance in 79 of 80 entries against cross-dataset baselines and improves 37 of 80 entries to new state-of-the-art against individually trained models.The cross-dataset comparison trains one model across datasets to predict 720 values, while the individual-model comparison trains separate models for each dataset and predictive length.Source: §5.2 · pp. 5–6, Table 2 · p. 6

5.3 Ablation Studies

• Ablations show that domain instructions, LightTrans, and reconstruction improve performance, while removing all designs degrades results across datasets.Without instructions, representations from different domains mix; with instructions they form domain clusters, while related ETT subsets cluster close together.Source: §5.3 · pp. 6–7
• Without masking, domain convergence speeds become imbalanced and some datasets overfit after epoch 4, whereas masking produces more stable later convergence.ETTm2, ETTh2, Exchange, and Illness show severe overfitting beyond the fourth epoch without masking; masking supports later model selection and better overall performance.Source: §5.3 · p. 7, Figure 3 · p. 7

5.4 Zero-Shot Transferability Analysis

• In zero-shot tests, UniTime outperforms baselines in most cases across ETTh2, Electricity, and Weather after training on ETTh1, ETTm1, and ETTm2.The instruction-selection protocol uses forecasting loss on 0.5% of test samples to select an instruction; all three target datasets select ETTh1’s instruction.Source: §5.4 · pp. 7–8, Table 5 · p. 8

5.5 Exploration Studies on Language Models

• Pretrained initialization and full language-model tuning perform best; changing input order causes a relatively small performance gap, partly mitigated by the decoder.Freezing the entire model still gives relatively strong performance, while FPT tunes only positional embeddings and layer normalization, balancing performance and efficiency when resources are limited.Source: §5.5 · p. 8, Table 4 · p. 8

6 CONCLUSION

• UniTime develops a unified forecasting model designed to accommodate diverse time series application domains.The paper frames this as a step toward a foundation model for general time series forecasting.Source: §6 · p. 8
• UniTime is proposed to address challenges in constructing a unified model across time series domains.Source: §6 · p. 8
• Extensive evaluations confirm UniTime’s effectiveness in advancing state-of-the-art forecasting performance and zero-shot transferability.Source: §6 · p. 8

• LM-based time-series forecasting has become a research focus [15] [16]; text and series share sequential pattern prediction and recency, while GPT2 can reason over numeric tokens.The passage says this capability extends to time series, as evidenced by this study and concurrent work including GPT4TS and LLMTime.Source: A MORE DISCUSSION ON LANGUAGE MODELS… · p. 10
• Universal forecasting addresses settings such as cloud workloads, where diverse series and lengths make training or tuning a separate model for each workload infeasible.The proposed paradigm aims to provide a more generalized forecasting solution under these practical constraints.Source: A MORE DISCUSSION ON LANGUAGE MODELS… · p. 10
• UniTime improves 37 out of 80 entries to new state-of-the-art versus separately trained dataset-specific baselines and also improves zero-shot transfer.The discussion attributes these gains to knowledge sharing or transfer across datasets.Source: A MORE DISCUSSION ON LANGUAGE MODELS… · p. 10
• GPT4TS [46] uses dataset- and prediction-length-specific models; a variable-length adaptation performed well individually but dropped cross-dataset on ETTm1 and Illness, while unified LLMTime was reportedly noncompetitive.Source: A MORE DISCUSSION ON LANGUAGE MODELS… · p. 10

B TRAINING CONFIGURATIONS

• Datasets are split chronologically, using 6:2:2 for ETT and 7:1:2 for others; ETTm1, ETTm2, and Weather each exceed 30,000 training samples, while Illness has 600.ETTh1 and ETTh2 have approximately 8,500 training samples each, and Exchange has 5,000.Source: B TRAINING CONFIGURATIONS · p. 10
• Batch sizes are 64 for ETTm1, ETTm2, and Weather; 32 for ETTh1 and ETTh2; 24 for Exchange; and 16 for Illness.Electricity uses 24 rather than its intended batch size of 32 because of GPU memory constraints.Source: B TRAINING CONFIGURATIONS · p. 10
• Illness is oversampled 12 times based on empirical assessments to increase exposure to its 600 training samples and prevent underrepresented domains from being marginalized.Oversampling augments datasets with significantly fewer training samples.Source: B TRAINING CONFIGURATIONS · p. 10

C MORE RESULTS FOR VARIANTS OF INSTRUCTIONS

• Instruction variants include shortened, generally expanded, and dataset-detail-expanded versions, alongside five sets that randomly mix variants across domains.ChatGPT-3.5-turbo generated the rephrasings from prompts; the original instructions served as the baseline.Source: C MORE RESULTS FOR VARIANTS OF… · p. 10
• The length comparison tests both a frozen language model with other UniTime components tuned and a setting where all UniTime parameters are trainable.Both settings receive the instructions and time-series data.Source: C MORE RESULTS FOR VARIANTS OF… · pp. 10–11
• Across five instruction sets of varying lengths, fine-tuning generally outperforms freezing the language model, but performance differences are not notably significant.Table 8 reports test results averaged over predictive lengths.Source: Table 8 · p. 11, C MORE RESULTS FOR VARIANTS OF… · p. 11
• Across five randomly generated rephrasing sets, performance differences are not notably significant, suggesting simple metadata-based instruction generation may not compromise results.The random sets were evaluated with language-model fine-tuning.Source: C MORE RESULTS FOR VARIANTS OF… · p. 11

D MORE RESULTS FOR UNITIME BACKBONE

• T5 does not surpass GPT-2 in the average results over all predictive lengths.The paper suggests GPT-2’s causal masking may preserve temporal order, potentially important for both textual and time series data, whereas T5 uses bidirectional attention.Source: D MORE RESULTS FOR UNITIME BACKBONE · pp. 11–12, Table 9 · p. 12

E HYPERPARAMETER STUDIES

• A mask ratio of 0.5 generally gives the best performance, and larger ratios tend to outperform smaller ones.The assessments are shown in Figures 5 and 6.Source: E HYPERPARAMETER STUDIES · p. 12
• Six Language-TS Transformer layers appear most favorable; the study does not test seven layers because of GPU memory limitations.The study investigates the number of Language-TS Transformer layers, denoted L_lm.Source: E HYPERPARAMETER STUDIES · p. 12