Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

1 Introduction

• Domain-specific adaptation can hinder cross-domain generalization, while centralized cross-domain training requires raw records that owners may withhold over privacy and copyright concerns.UniTime (Liu et al. (2024)) trains a unified model on mixed cross-domain data centrally.Source: §1 · p. 2
• The design addresses heterogeneous input dimensions and histories, rigid prompts (Liu et al. (2024); Jin et al. (2024)), and the need to balance shared representations with personalized forecasts.These challenges complicate modality alignment, robustness to unseen domains, and customized prediction.Source: §1 · p. 2
• Time-FFM converts time series into text tokens, dynamically selects domain-specific prompts, and combines a global encoder with personalized prediction heads in federated training.The prompt adaptation is determined dynamically rather than through hand-crafted instructions.Source: §1 · pp. 2–3
• The paper presents its work as the first federated foundation model for time-series forecasting that exploits language-model sequence reasoning without disclosing local data.Federated learning exchanges model parameters or gradients without sharing raw records.Source: §1 · pp. 2–3
• The authors report that TIME-FFM achieves state-of-the-art performance in mainstream forecasting, especially in few-shot and zero-shot settings.The introduction summarizes extensive evaluation results but supplies no metric values here.Source: §1 · p. 3

FMs for Time Series Forecasting

• Time-series foundation models and cross-modal LM adaptations face data scarcity and limited temporal commonality, while centralized cross-domain forecasting raises privacy concerns that federated learning may address.The reviewed approaches can unify model structure without unifying parameters across downstream tasks.Source: §2 · p. 3

Federated Fine-tuning of LMs

• Federated LM fine-tuning methods address communication costs and data heterogeneity, but focus on NLP rather than cross-modal LM adaptation for time-series forecasting.The passage describes implementation frameworks, communication-efficient methods, and personalized federated instruction tuning.Source: §2 · p. 3

3.1 Problem Definition

• For each domain i, forecasting maps a lookback window Xi of length Li to future targets Yi of length Fi using its local dataset Di and personalized model parameters.Source: §3.1 · p. 3

3.2 Model Structure

• Frozen GPT-2 processes concatenated prompt and patch tokens, and a prediction head flattens its representations and projects them to forecast results.Source: §3.2 · pp. 4–5
• TIME-FFM splits multivariate series channel-wise, normalizes each channel, patches it to retain local temporal information, and linearly maps patches to LM input tokens.Source: §3.2 · p. 4
• Rather than handcrafting instructions, prompt adaption compresses pretrained LM word embeddings into prototypes and uses multi-head attention to select the M most related to patch tokens.The selected prototypes serve as prompts for the LM backbone.Source: §3.2 · p. 5
• To balance cross-domain generalization with domain specificity, TIME-FFM shares modality-alignment and prompt-adaptation parameters while keeping prediction heads local; the GPT-2 backbone remains frozen.Only modality-alignment and prompt-adaptation parameters are communicated and averaged across domains.Source: §3.2 · pp. 4–5

3.3 Training Process

• Each federated round distributes global parameters, domains optimize them with local prediction heads and upload updates, and the server averages the updates for the next round.Source: §3.3 · pp. 5–6

4.1 Main Results

• TIME-FFM improves MSE by 39.01% over the second-best federated method across datasets, except ETTh2, and its averaged predictions outperform centralized models, including UniTime.The comparison is based on results averaged over four prediction windows.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Few-Shot Forecasting

• In 5% few-shot forecasting, TIME-FFM reduces averaged MSE by 20% versus UniTime and outperforms centralized methods; it also outperforms other federated methods in 10% settings.Except for UniTime, methods perform worse with 10% than with 5% of training steps; the authors attribute this to UniTime fully tuning its pretrained LM, while other methods retain LM reasoning with fewer samples.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 8

4.3 Zero-Shot Forecasting

• In zero-shot forecasting, TIME-FFM lowers prediction MSE by 13.9% versus the second-best method across ETTh2, Electricity, and Weather, outperforming UniTime.For cross-domain targets Electricity and Weather, the method reuses ETTh1 local parameters; the authors attribute its gains to knowledge transferability across domains.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Model Analysis

• Ablations show that removing prompt adaptation, using fixed instructions, or removing personalized heads compromises forecasting; removing personalized heads hurts performance most.Without personalized heads, a global prediction head is learned across domains, limiting personalization for heterogeneous data; removing shared temporal knowledge also weakens cross-domain generalization of adaptation modules.Source: §4.4 · pp. 8–9, Table 5 · p. 9
• Full LM tuning performs best, followed by tuning positional embeddings and layer normalization, while frozen-backbone performance remains comparable; using 12 rather than 6 layers improves results.The authors interpret these findings as evidence that LMs align effectively with time-series tokens and that scaling behavior carries over to forecasting.Source: §4.4 · p. 9, Table 6 · p. 9
• TIME-FFM reduces training and communication parameters with insignificant training-time increase, and parameter counts remain unchanged across backbone depths.The efficiency analysis is conducted on ETTh1.Source: §4.4 · p. 9, Table 7 · p. 10
• Prompt-adaptation attention focuses on a small set of text prototypes per patch token, with dataset-specific prototypes and shared patterns across ETTh1, Electricity, and ILI.The authors interpret these patterns as evidence that global prompt adaptation generalizes across datasets while preserving domain personalization.Source: §4.4 · pp. 9–10, Figure 3 · p. 9

• TIME-FFM casts time series as text tokens, adaptively selects text prototypes as prompts for LM reasoning, and uses personalized federated learning for domain-customized predictions.The method is presented as a federated foundation model for time series forecasting that leverages pretrained language models.Source: §5 · p. 10
• Extensive experiments across various scenarios indicate that TIME-FFM promises an effective foundation model for time series forecasting.Source: §5 · p. 10
• The authors report increased training time compared with TY1 and suboptimal performance in some cases, and propose more efficient modality alignment and studying patch-to-word embedding correspondence.They aim to explore whether time series can be seamlessly translated into natural language.Source: §5 · p. 10

Implementation

• Implementation uses Adam at an initial learning rate of 10−4, the first six pretrained GPT-2 layers, one local epoch, and 100 global rounds.Lookback windows are 36 for ILI and 96 for other datasets; prediction windows are {24, 36, 48, 60} for ILI and {96, 192, 336, 720} for others. The model checkpoint is selected by lowest validation loss.Source: A Experimental Details · p. 14

Training configurations

• Training uses eight real-world benchmark datasets across five domains, with batch division and oversampling following Liu et al. (2024).Per round, training batches are apportioned by domain using normalized ratios of 0.65 for ILI and 0.05 for each other domain; local training proceeds sequentially until each domain’s batch quota is reached.Source: A Experimental Details · p. 14, Table 8 · p. 14

Oversampling evaluation

• Oversampling balances contributions to global knowledge, and ILI’s augmented local knowledge improves forecasting for ILI and other domains.The comparison averages results over four prediction windows for each dataset group; the supplied table caption does not provide cell values.Source: A Experimental Details · p. 14, Table 9 · p. 14

B Hyperparameter Sensitivity

• M=12 achieves the best performance, while fewer prompt tokens may not cover all useful knowledge.The sensitivity study is on ILI.Source: B Hyperparameter Sensitivity · pp. 14–15, Figure 4 · p. 15
• Both lower and higher V ′ values yield subpar performance because prototype count affects knowledge coverage for prompt adaption.Fewer prototypes mix relevant and irrelevant knowledge in each prototype; with more prototypes, a stable prompt-token count may not cover all relevant knowledge.Source: B Hyperparameter Sensitivity · pp. 14–15
• Increasing H does not always improve performance, as more attention heads may break the semantic integrity of text prototypes and patch embeddings.Source: B Hyperparameter Sensitivity · pp. 14–15

C Full Results

• 32 out of 42 full-results instances show TIME-FFM achieving state-of-the-art forecasting performance across eight benchmarks.The authors attribute this performance to cross-modality adaptation, including modality alignment and prompt adaptation, as well as personalized prediction heads.Source: C Full Results · p. 15
• In both 10% and 5% few-shot settings, TIME-FFM outperforms other federated-learning methods in TY1.The complete comparisons are presented in Tables 11 and 13, respectively.Source: C Full Results · p. 15
• At 10% few-shot, TIME-FFM achieves comparable performance against methods in TY2.The comparison is reported for the 10% few-shot forecasting setting.Source: C Full Results · p. 15
• 20 out of 48 instances across five benchmarks show TIME-FFM attaining state-of-the-art performance in 5% few-shot learning.The authors conclude that the results support TIME-FFM as an effective few-shot forecaster.Source: C Full Results · p. 15

D Error Bars

• TIME-FFM’s TY1 results are means and standard deviations across three runs, and the authors report that they demonstrate superiority consistent with Table 1.The statistics are presented in Table 15; no specific values are provided in the passage.Source: D Error Bars · p. 15

E Border Impacts

• Time-FFM avoids requiring data owners to share private samples, which the authors say mitigates privacy concerns and supports data-protection regulations such as GDPR.The authors identify this as one of the model’s two promising advantages.Source: E Border Impacts · p. 15
• The authors say Time-FFM can tackle “data islands” and enable satisfactory performance despite data scarcity.They identify this as the model’s other promising advantage.Source: E Border Impacts · p. 15
• The authors state that, to their knowledge, the research has no obvious negative social impacts.Source: E Border Impacts · p. 15