AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

1 Introduction

• Time-series foundation models are limited by scarce large-scale pretraining datasets and uncertainty about scalable backbones, despite shared sequence-modeling properties with language.General-purpose forecasting also calls for variable-length scenarios and auxiliary information from other modalities.Source: §1 · p. 1
• Prevalent LLM4TS methods flatten lookback tokens for one-step prediction, converting decoder-only autoregressive LLMs into non-autoregressive forecasters.The paper argues this may leave LLM capabilities underused because decoder-only autoregressive training is associated with their generalization performance.Source: §1 · p. 2
• AutoTimes embeds time-series segments with next-token prediction, freezes the LLM, and uses autoregressive inference for forecasts of arbitrary lengths.Its time-series token embedding and projection account for up to 0.1% of total parameters.Source: §1 · p. 2
• In-context forecasting uses relevant time series as prompts, while LLM-embedded timestamps add chronological information to align multiple variates.The paper presents these as extensions beyond conventional forecasting.Source: §1 · p. 2
• AutoTimes achieves superior performance to state-of-the-art methods while saving over 80% training and inference time.The paper also reports zero-shot generalizability, in-context forecasting, and scaling behavior empowered by LLMs.Source: §1 · p. 2

2.1 Autoregressive Models

• Autoregression supports variable-length prediction in statistical forecasters; ARIMA combines differencing with autoregressive and moving-average models.Autoregressive language models use position-wise supervision and are not constrained to specific input and output lengths, supporting multi-step generation.Source: §2.1 · p. 3

2.2 Large Language Models for Time Series

• LLM4TS includes text-pair prompting, numerical tokens, representation extraction, unified cross-domain forecasting, and language or soft prompts.AutoTimes instead freezes the LLM and transfers its token transitions for autoregressive next-token prediction with minimal added parameters.Source: §2.2 · p. 3

2.3 Multimodal Language Models

• AutoTimes uses time series themselves as prompts and textual timestamps as position embeddings, avoiding direct concatenation of language and time-series tokens.Table 1 compares methods by autoregression, freezing the LLM, and multimodal use; none before AutoTimes achieved all three.Source: §2.3 · p. 3, Table 1 · p. 3

3 Method

• Given L observed steps across C variates and their timestamps, the forecaster predicts F future steps for varying L and arbitrary F.Source: §3 · pp. 3–4

3.1 Modality Alignment

• Each variate is segmented and embedded in the LLM latent space, while LLM-embedded textual timestamps provide position embeddings that align simultaneous events without increasing context length.Source: §3.1 · pp. 4–5

3.2 Next Token Prediction

• AutoTimes feeds segment embeddings through LLM transition layers, projects and supervises next segments, then iterates autoregressively for arbitrary forecast lengths with one model across lookback/forecast scenarios.Source: §3.2 · p. 5

3.3 In-Context Forecasting

• Earlier historical lookback-forecast windows from the target dataset serve as prompts, concatenated with the lookback series before prediction to extend the context.Source: §3.3 · p. 6

4.1 Time Series Forecasting

• AutoTimes consistently outperforms all counterparts on short-term M4 forecasting.The reported results are averages; full results are in Table 11.Source: §4.1 · p. 6, Table 2 · p. 7
• AutoTimes surpasses other methods on 80% of one-for-all long-term datasets, exceeds TimeLLM by 9.12% on average, and reaches state-of-the-art in 70% of one-for-one settings without per-length training.The one-for-all benchmark rolls predictions across lengths using a single model trained on each dataset.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2 Zero-Shot Forecasting

• In both M4 →M3 and M3 →M4 zero-shot scenarios, AutoTimes outperforms deep forecasters and FPT.The results are reported in averaged SMAPE; AutoTimes also surpasses FPT without tuning intermediate LLM layers.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 In-Context Forecasting

• In-context forecasting improves every M3 subset, reducing average SMAPE by 13.3% versus zero-shot forecasting.Target-domain time-series prompts are concatenated with the lookback window to form prediction context.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Method Analysis

• AutoTimes works with decoder-only LLMs including GPT-2, OPT, and LLaMA, and forecasting accuracy generally improves as LLM parameter count increases.LLaMA-7B delivers the best forecasting performance among the evaluated models; OPT-1.3B offers parameter efficiency as an out-of-the-box forecaster.Source: §4.4 · pp. 8–9, Table 5 · p. 9, Figure 5 · p. 9
• AutoTimes achieves over 5× average training and reasoning speedup against advanced LLM4TS methods, using a 0.79M-parameter MLP, 0.1% of the 7B LLM's parameters.The efficiency comparison reports better results in Table 3 alongside reduced training and reasoning time.Source: §4.4 · p. 9, Figure 6 · p. 9
• Ablations support retaining the decoder-only architecture and autoregressive inference; replacing token-wise projection with a flatten linear head yields consistently inferior results.Table 6 compares AutoTimes with versions replacing the LLM with no LLM, attention, or a transformer block.Source: §4.4 · p. 10, Table 6 · p. 10
• Adding LoRA to intermediate LLM layers consistently improves the forecasting performance of AutoTimes.LoRA fine-tunes token transitions to align future time-series extrapolation.Source: §4.4 · p. 10, Table 7 · p. 10

5 Conclusion

• AutoTimes repurposes off-the-shelf LLMs as autoregressive time-series forecasters, transferring their general-purpose and multi-step generation abilities.The paper contrasts this approach with prevalent non-autoregressive LLM4TS methods that may contradict the decoder-only structure and insufficiently use LLMs.Source: §5 · p. 10
• The authors report state-of-the-art forecasting performance with model efficiency, plus zero-shot and in-context forecasting capabilities.They also report using instructive time series and timestamps.Source: §5 · p. 10
• Future work will further incorporate low-rank adaptation and use emerging language backbones.Source: §5 · p. 10

A Dataset Descriptions

• The seven evaluation datasets are ETTh1, Weather, ECL, Traffic, Solar-Energy, M4, and M3, spanning electricity, meteorology, road occupancy, solar power, and varied business and economic series.ETTh1 has seven factors, Weather 21, ECL 321 clients, Traffic 862 sensors, and Solar-Energy 137 plants.Source: A Dataset Descriptions · p. 14
• Dataset time coverage and sampling vary: ETTh1 spans July 2016–July 2018, Weather is sampled every 10 minutes in 2020, ECL and Traffic are hourly, and Solar-Energy is sampled every 10 minutes in 2006.Traffic data cover January 2015–December 2016.Source: A Dataset Descriptions · p. 14

B Implementation Details

• AutoTimes represents timestamps textually, embeds them with the LLM, and uses shared timestamps to express relationships between variates even under channel independence.Timestamp embeddings can be precomputed efficiently because their count is based on time points rather than variates.Source: B Implementation Details · p. 14
• Training updates SegmentEmbedding and SegmentProjection while keeping LLM parameters frozen; at inference, the last generated token becomes a prediction for autoregressive variable-length forecasting.The generated output is fed into subsequent prediction steps, allowing one model to predict sequences of different lengths.Source: B Implementation Details · pp. 14–15
• Experiments use Adam, MSE loss, channel independence for multivariate series, timestamp position embeddings, and LLaMA-7B by default.The reported setup uses 10 epochs, and the paper reports performance standard deviations from three random seeds.Source: B Implementation Details · p. 15, Table 9 · p. 16

C Hyperparameter Sensitivity

• AutoTimes is reported insensitive to embedding and projection layer configurations; longer context can improve performance, and larger segments are favored for long forecasts.The hyperparameter sensitivity curves each correspond to a specific forecast length.Source: C Hyperparameter Sensitivity · p. 16, Figure 7 · p. 16

D.1 Time Series Forecasting

• The one-for-all evaluation uses rolling forecasts across long-term datasets, while short-term M4 results follow the TimesNet protocol.Table 10 covers ETTh1, ECL, Traffic, Weather, and Solar-Energy; Table 11 reports short-term results.Source: §D.1 · p. 17, Table 10 · p. 17, Table 11 · p. 18
• Additional benchmarks are reported, with Table 15 covering datasets beyond the primary forecasting evaluations.The authors state that they plan to evaluate on more diverse benchmarks.Source: §D.1 · p. 17, Table 15 · p. 19
• The one-for-one comparison contrasts AutoTimes, which handles prediction lengths autoregressively, with models trained separately for each length.Tables 12 and 14 report averaged and full results; Table 13 compares original and reproduced LLM4TS results.Source: §D.1 · p. 17, Table 12 · p. 18, Table 13 · p. 18

D.2 Zero-Shot Forecasting

• Zero-shot evaluation trains on one dataset and predicts directly on another, testing transfer between M4 and M3 subsets with matching sampling frequencies but different distributions.The reported directions are M4 → M3 and M3 → M4, using subset-matched models where possible.Source: §D.2 · p. 20, Table 16 · p. 20

D.3 Method Generality

• AutoTimes works with decoder-only LLMs, and performance generally improves as model size increases across GPT-2, OPT, and LLaMA experiments.Tables 17 and 18 provide configurations and results for these alternative language models.Source: §D.3 · p. 20, Table 17 · p. 20, Table 18 · p. 21

D.4 Variable Lookback Length

• 9.3% average promotion is reported when increasing available lookback observations from 384 to 672 after adapting AutoTimes at context length C = 672.Figure 8 evaluates the adapted forecaster at different lookback lengths; the text contrasts this with deep forecasters that may not improve as lookback grows.Source: §D.4 · p. 20, Figure 8 · p. 21

D.5 Timestamps as Position Embeddings

• Textual timestamps consistently promote performance across multivariate datasets and prediction lengths, with larger gains at longer prediction lengths.The authors attribute the improvement to aligning variates on the timeline and note the value of chronological information for long-term forecasting.Source: §D.5 · p. 21, Figure 9 · p. 21

• In-context forecasting adds earlier time points from the target sequence to a lookback window, while the evaluation trains on M4 and tests on M3 sequences.The implementation concatenates the first 2F points from the same sequence and excludes sequences too short to avoid prompt-lookback overlap.Source: §D.6 · p. 22
• Prompts from unrelated series can deteriorate averaged performance despite enlarging the context, while relevant same-series prompts usually provide better context.The prompt-strategy comparison contrasts out-of-series prompts with same-series prompt choices.Source: §D.6 · p. 22, Table 19 · p. 22
• Inter-period prompts can outperform simply extending the lookback window, underscoring the influence of prompt selection.Table 20 reports prompt-selection strategies based on periodicity.Source: §D.6 · p. 22, Table 20 · p. 23
• Using earlier, periodically related history can add relevant context beyond a continuous lookback, whereas a longer continuous window may include noisy information.The paper illustrates this with weather prediction using the same day in prior years; its broader evaluation compares a periodic prompt with random prompt strategies and a longer lookback.Source: §D.6 · pp. 22–23

D.7 Ablation Study

• Non-autoregressive projection performs consistently worse than autoregressive AutoTimes in the ablation study.Table 21 compares AutoTimes with a FlattenHead variant that replaces segment-wise projection with a flatten-and-linear head.Source: §D.7 · p. 23, Table 21 · p. 23

E Showcases

• AutoTimes provides the most accurate showcased future-series predictions among AutoTimes, TimeLLM, FPT, and PatchTST.The appendix also includes Traffic and M4 Monthly showcases and examples of zero-shot and in-context forecasting.Source: E Showcases · p. 23, Figure 10 · p. 23, Figure 11 · p. 24

F.1 Impact on Real-world Applications

• The authors describe AutoTimes as promising for real-world applications, including helping prevent risks and improve decisions under limited computational budgets.They also state that the paper focuses on scientific research and has no obvious negative social impact.Source: §F.1 · pp. 24–25

F.2 Impact on Future Research

• AutoTimes uses next-token prediction and iterative generation for arbitrary-length forecasts, while in-context forecasting adds earlier history as prompts.The authors present compatibility with LLMs and autoregressive design as potentially instructive for future time-series foundation models.Source: §F.2 · p. 25

G Limitation

• AutoTimes does not support probabilistic forecasting because it maps time-series segments to LLM latent embeddings rather than discrete language tokens.The authors state this limits the method's forecasting scope.Source: G Limitation · p. 25
• Advanced low-rank adaptation and more compatible time-series embedding and projection layers remain under exploration.The authors identify these as potential ways to better align token transitions and support time-series tokenization.Source: G Limitation · p. 25
• Applying AutoTimes to real-world multimodal time series, such as news-stocks or logs-measurements, remains future work.Source: G Limitation · p. 25