Random Initialization as a Baseline for Time Series Foundation Models

1 Introduction

• The central question is how much performance comes from pretrained weights rather than architecture and aggregation in multichannel classification.Many current models process multivariate inputs independently by channel and aggregate their representations before classification.Source: §1 · p. 2
• Randomly initialized representations can be useful, making architecture-matched comparisons relevant to separating weight contributions from architecture and aggregation effects.ROCKET demonstrates strong time-series classification with random convolutional features.Source: §1 · p. 2
• Prior evidence includes comparable pretrained and random Mantis representations on four EEG benchmarks, but broader TSFM and general-purpose MTSC coverage remains unclear.The open scope includes model families, model scales, classifier capacities, and MTSC datasets.Source: §1 · p. 2
• The study compares seven TSFM instances with architecture-matched random counterparts across four downstream classifiers and five random seeds.The models are evaluated on the UEA MTSC archive alongside ROCKET, Hydra, and Quant.Source: §1 · p. 2
• Pretraining yields a mean improvement of 1.43–5.19 percentage points per model instance and 2.58 percentage points overall across 183 paired comparisons; only NuTime remains significant after Holm correction.Source: §1 · p. 2
• The findings motivate evaluating both architecture-matched random initialization and strong time-series-specific methods as controls for large-scale pretraining.The pretrained–random difference remains modest on average across linear and nonlinear classifiers, while established non-pretrained methods remain competitive.Source: §1 · p. 2

2 Related Work

• MOMENT uses masked reconstruction, Mantis and MantisV2 use contrastive pretraining, NuTime uses BYOL-based self-distillation, and UniTS combines tasks in a unified backbone.Other foundation models primarily target forecasting, whereas this study focuses on MTSC.Source: §2 · p. 2
• Earlier MTSC benchmarking showed many foundation models process channels independently but did not compare architecture-matched random initialization.Related comparisons covered EEG classification and few-shot forecasting; broader frozen MTSC evaluation across TSFM families and scales remained the focus here.Source: §2 · p. 2
• ROCKET, Hydra, and Quant provide complementary non-pretrained reference points, while fixed architecture and representation pipelines isolate pretrained-weight contributions.ROCKET uses random convolutional kernels, Hydra competing convolutional kernels, and Quant interval-based quantile features.Source: §2 · p. 3

3 Experiments

• The study is organized around pretrained-weight contribution, dependence on downstream classifier, and comparison with time-series-specific classifiers.Source: §3 · p. 3

3.1 Foundation models

• Seven publicly released instances comprise MOMENT Small, Base, and Large, Mantis, MantisV2, UniTS, and NuTime, evaluated with released pretrained checkpoints.Source: §3.1 · p. 3

3.2 Architecture-matched random initialization

• Five random variants per model use the same configuration and official initialization, with no pretrained parameters retained and no pretraining or fine-tuning.The pretrained and random conditions share preprocessing, inference, representation extraction, aggregation, and evaluation; encoder weights are the experimental variable.Source: §3.2 · p. 3

3.3 Downstream representation evaluation

• Evaluation uses standard UEA splits and includes only datasets with all four classifier results under both conditions, yielding 26 paired datasets for most instances and 27 for UniTS.The cross-method comparison uses the 24 datasets shared by all methods; classifiers are ridge, logistic regression, RBF SVM, and random forest.Source: §3.3 · p. 3

3.4 Time-series-specific reference methods

• ROCKET, Hydra, and Quant use default aeon configurations without dataset-specific tuning, and direct comparison uses a common subset with valid results for all methods.Source: §3.4 · p. 4

3.5 Statistical analysis

• Per-instance paired dataset scores are tested with two-sided Wilcoxon signed-rank tests and Holm correction; pooled testing is secondary because datasets recur across models.The method comparison is descriptive, using mean accuracy, average rank, and dataset wins.Source: §3.5 · p. 4

4.1 How much does pretraining contribute?

• NuTime has a 4.20-point median improvement and wins on 21 of 26 datasets; MantisV2 and UniTS have smaller median than mean gains.MantisV2’s median and mean differences are 0.80 and 2.58 points; UniTS’s are 0.40 and 2.24 points, suggesting their means reflect a subset of larger positive differences.Source: §4.1 · p. 4
• Pretraining improves mean accuracy by 1.43–5.19 points per instance and 2.58 points across 183 paired comparisons; only NuTime remains significant after Holm correction.Table 1 reports the architecture-matched comparison; three per-instance tests have raw p < 0.05, while the pooled test is secondary because datasets recur across instances.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Does the result depend on classifier capacity?

• The average pretrained advantage is 2.33 points for linear and 2.83 for nonlinear classifiers, without a consistently larger nonlinear benefit.Patterns vary by model: MOMENT-Large’s difference falls from 2.59 points under linear classifiers to 0.93 under nonlinear classifiers, while Mantis and MantisV2 show the opposite pattern.Source: Table 2 · p. 5, §4.2 · p. 5

4.3 How competitive are time-series-specific methods?

• On the common 24-dataset subset, ROCKET has the highest mean accuracy (0.754) and best average rank (3.48), while Hydra records the most dataset wins (10).Quant’s mean accuracy is 0.752; these time-series-specific methods avoid large-scale pretraining.Source: Table 3 · p. 6, §4.3 · p. 6
• Mantis is the strongest foundation model on that subset, with mean accuracy 0.731 and average rank 3.94, and six dataset wins.Its mean accuracy is below ROCKET, Hydra, and Quant, but its average rank is better than Quant’s; the remaining foundation models rank below all three references on average.Source: §4.3 · p. 6
• The cross-method comparison is descriptive, not architecture-matched, and compares frozen MTSC representations rather than fine-tuning or other downstream tasks.Foundation-model results average four classifiers, while ROCKET, Hydra, and Quant use end-to-end outputs; the study also does not measure pretraining time, energy, or emissions.Source: §4.3 · p. 6, §4.4 · p. 7

4.4 Discussion and limitations

• Architecture, preprocessing, and aggregation can contribute to performance independently of learned weights, so pretrained results should be measured against architecture-matched random baselines.The evaluated models largely process channels independently before model-specific aggregation; this study does not establish equivalence between pretrained and random representations.Source: §4.4 · pp. 6–7

5 Conclusion

• 2.58 points overall across 183 paired dataset-level comparisons, with per-instance gains of 1.43–5.19 points; only NuTime retains a significant per-instance advantage after Holm correction.The pooled comparison is also significant (p < 0.001), but Holm-corrected per-instance tests are the primary evidence.Source: §5 · p. 7
• ROCKET, Hydra, and Quant remain competitive with the evaluated pretrained foundation models without large-scale pretraining.The findings support comparing pretrained weights against both architecture-matched random initialization and strong time-series-specific methods.Source: §5 · p. 7
• Full fine-tuning may show benefits not captured by the study’s frozen-representation evaluation.Source: §5 · p. 7

A Dataset coverage

• A model-dataset result enters the aggregate analysis only when all four downstream classifier results are available under both pretrained and randomly initialized conditions.Table 4 summarizes datasets where at least one foundation-model instance lacks a complete four-classifier result.Source: A Dataset coverage · p. 10, Table 4 · p. 10
• The common cross-method subset contains 24 UEA datasets, excluding DuckDuckGeese, FaceDetection, InsectWingbeat, PenDigits, PEMS-SF, and StandWalkJump.Source: A Dataset coverage · p. 10

B Implementation details

• Released pretrained checkpoints and the corresponding official implementations are used for each foundation-model instance.Source: B Implementation details · p. 10
• Three-fold stratified cross-validation on the official training split selects hyperparameters, followed by refitting on the full training split and one test-split evaluation.Accuracy is the selection metric, and standardization is included for ridge classification, logistic regression, and RBF SVM.Source: B Implementation details · p. 10
• The same classifier grids and cross-validation procedure apply across datasets, instances, pretraining conditions, and random encoder seeds.Each random-initialization seed is tuned independently, and no selection uses the UEA test split.Source: Table 5 · p. 10, B Implementation details · p. 10

C Per-dataset foundation-model results

• Aggregate table means use entries with complete results for all four downstream classifiers, and pretrained–random comparisons use paired datasets available under both conditions.The paired dataset-level values also determine mean and median differences and win/tie/loss counts.Source: C Per-dataset foundation-model results · p. 11
• Per-dataset random-initialization accuracy is averaged across four classifiers and five seeds, with incomplete classifier results shown as dashes.The appendix lists all 30 UEA datasets, while unavailable model-dataset results are represented by dashes.Source: Table 6 · p. 11, C Per-dataset foundation-model results · p. 11
• Per-dataset pretrained-model accuracy is averaged across the four downstream classifiers, with incomplete results shown as dashes.Source: Table 7 · p. 12
• Random-initialization top-one accuracy tables report ridge classification, logistic regression, RBF SVM, and random forest results averaged across five seeds.Source: Table 8 · p. 12, Table 9 · p. 13, Table 10 · p. 13

• Table 16 reports classification accuracy for ROCKET, Quant, and Hydra across all 30 UEA datasets.Source: D Time-series-specific baseline results · p. 16
• ROCKET, Quant, and Hydra provide external time-series-specific reference points for the foundation-model comparisons.Source: D Time-series-specific baseline results · p. 16
• These methods are not architecture-matched controls for the foundation models.Source: D Time-series-specific baseline results · p. 16

E Controlled CPU diagnostic

• The diagnostic uses single-thread CPU inference on synthetic standard-normal float32 input shaped [1, 3, 512], with each shared architecture measured once.Source: E Controlled CPU diagnostic · p. 16, Table 17 · p. 17
• Neural-model latency covers the frozen encoder and aggregation, whereas aeon latency covers fitted feature transforms; aeon fit time is separate and not comparable to TSFM pretraining.Source: E Controlled CPU diagnostic · p. 16, Table 17 · p. 17
• NuTime and UniTS are omitted because their released multivariate inference paths do not match the common input protocol; measurements use 20 warm-ups and 100 timed calls unless stated otherwise.Source: E Controlled CPU diagnostic · p. 16
• Registered operation counts cover matrix multiplications and convolutions only, excluding normalization, softmax, activations, interpolation, indexing, allocation, and non-Torch operations.Source: E Controlled CPU diagnostic · p. 16, Table 18 · p. 18
• MOMENT median latency rises from 82.48 ms for Small to 354.91 ms for Base and 1354.18 ms for Large, while registered operations rise from 7.400 to 119.591 GFLOPs.Source: E Controlled CPU diagnostic · p. 18
• MantisV2 and Mantis record encoder-forward latencies of 15.88 ms and 18.61 ms, respectively.Source: E Controlled CPU diagnostic · p. 18
• Fitted aeon transforms record median latencies of 14.26 ms for Hydra, 28.99 ms for Quant, and 46.83 ms for ROCKET.Source: E Controlled CPU diagnostic · p. 18
• The component measurements are not a direct end-to-end ranking and should not be extrapolated to native sequences, other hardware or batching, or memory efficiency.Source: E Controlled CPU diagnostic · p. 18