PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-learning framework with applications to zero-shot time-series forecasting

Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, Yoshua Bengio

arXiv:2002.02887v3cs.LGstat.ML

TL;DR

Deep forecasting can require large problem-specific datasets, creating a cold-start challenge for deployment. This paper develops a broad meta-learning framework and analyzes residual connections as an adaptation mechanism; experiments report zero-shot performance across diverse datasets, supporting transfer without target retraining.

  • Problem

    Deep time-series models face a cold-start problem because useful forecasting outputs require training on a large problem-specific dataset.

    Source: §1 · p. 1

  • Method

    The paper develops a broad meta-learning framework and analyzes N-BEATS residual connections as generating task-specific parameters from the input, with a linearized interpretation as sequential final-layer updates.

    Source: §1.3 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    Zero-shot N-BEATS outperforms M4’s best selected statistical model, is at par with its second-place boosted trees, beats M3’s Theta winner on M3 and TOURISM, and is close to or better than other neural models on ELECTRICITY and TRAFFIC.

    Source: §4 · p. 7

  • Takeaways & Limitations

    A neural model trained on a diversified source dataset can be deployed on a different target dataset without explicit retraining, with performance at least as good as advanced statistical models tailored to the target.

    Source: §1 · p. 1

    The adaptation setup treats the input history as task data, relying on complete time series whose input dependencies resemble those of the forecasting targets.

    Source: §3 · p. 4

Abstract

from arXiv · show

Can meta-learning discover generic ways of processing time series (TS) from a diverse dataset so as to greatly improve generalization on new TS coming from different datasets? This work provides positive evidence to this using a broad meta-learning framework which we show subsumes many existing meta-learning algorithms. Our theoretical analysis suggests that residual connections act as a meta-learning adaptation mechanism, generating a subset of task-specific parameters based on a given TS input, thus gradually expanding the expressive power of the architecture on-the-fly. The same mechanism is shown via linearization analysis to have the interpretation of a sequential update of the final linear layer. Our empirical results on a wide range of data emphasize the importance of the identified meta-learning mechanisms for successful zero-shot univariate forecasting, suggesting that it is viable to train a neural network on a source TS dataset and deploy it on a different target TS dataset without retraining, resulting in performance that is at least as good as that of state-of-practice univariate forecasting models.

1 Introduction

The paper addresses the cold-start cost of deep time-series forecasting by asking whether models can transfer forecasting knowledge across datasets. It proposes a broad meta-learning framework and reports that zero-shot univariate forecasting is feasible.

1 Introduction

  • The paper targets the cold-start cost of deep forecasting, asking whether forecasting knowledge can transfer across datasets and support deployment without target-data retraining.The paper defines zero-shot use as a setting where target history is too limited to train a deep model feasibly.Source: §1 · p. 1

1.1 Background

  • The paper formalizes univariate forecasting over a fixed horizon and describes N-BEATS as shared-parameter blocks producing backcasts and partial forecasts.Forecast accuracy can be evaluated with sMAPE.Source: §1.1 · p. 2

1.2 Related Work

  • Unlike prior target fine-tuning work, this study focuses on zero-shot forecasting and compares neural transfer with classical statistical forecasting approaches.The paper contrasts learned forecasting strategies with classical models that estimate parameters using predefined rules.Source: §1.2 · p. 2

1.3 Summary of Contributions

  • The proposed framework recasts existing meta-learning algorithms and interprets N-BEATS residual connections as task adaptation without inference-time gradient steps.The paper also introduces a zero-shot univariate forecasting task and releases loaders and evaluation code with FRED, a dataset of 290k time series.Source: §1.3 · p. 2
  • The authors report that deep-learning zero-shot univariate forecasting is feasible and that meta-learning mechanisms matter for zero-shot generalization.The claim is stated across a range of datasets.Source: §1.3 · p. 2

2 Meta-learning Framework

The framework separates cross-task learning from task-specific adaptation, then uses shared equations to express existing meta-learning algorithms. It also frames time-series forecasting as learning a data-driven parameter-estimation strategy across tasks.

2 Meta-learning Framework

  • The framework separates shared meta-parameters from task-adaptive predictor parameters, using initialization and update functions to produce task-specific parameter sequences.The meta-learner is trained across tasks to generalize to new tasks.Source: §2 · p. 3

2.1 Meta-learning and time-series forecasting

  • For forecasting, meta-learning replaces predefined parameter-estimation rules with a learned strategy optimized across multiple time series.The paper assumes training tasks represent forecasting tasks encountered at inference.Source: §2.1 · p. 3

2.2 Expressing Existing Meta-Learning Algorithms in the Proposed Framework

  • The shared equations derive four optimization- and metric-based meta-learning algorithms, with three additional algorithms analyzed in the appendix.Examples include MAML-style gradient updates and Prototypical Networks, whose task-specific prototypes act as dynamically generated final-layer weights.Source: §2.2 · pp. 3–4

3 N-BEATS as a Meta-learning Algorithm

The analysis casts N-BEATS blocks as input-specific meta-learning updates that expand the architecture’s expressive power. Under linear approximation, these updates can resemble sequential updates to the predictor’s final linear layer.

3 N-BEATS as a Meta-learning Algorithm

  • Each N-BEATS block acts as an inner meta-learning loop, generating time-series-specific input shifts while shared predictor parameters are learned across tasks.The task-specific training input is the observed time series itself, which the paper motivates by its internal dependencies resembling those of forecasting targets.Source: §3 · p. 4
  • Even with shared block parameters, adding blocks can expand expressive power through the growing sequence of input-specific shift parameters.The paper expects identical added blocks to improve generalization because they increase the parameter space spanned by the shifts.Source: §3 · p. 4

3.1 Linear Approximation Analysis

  • Under certain conditions, linear approximation makes block-by-block input manipulation equivalent to iteratively updating the predictor’s final linear-layer weights.The paper relates this behavior to final-layer updates identified in LEO, MAML, and PNs.Source: §3.1 · pp. 4–5

3.2 The Role of Q

  • The first-order condition for Q makes the second block’s correction tend to compensate, on average, for the default forecast error.The analysis does not provide a closed-form solution for Q.Source: §3.2 · p. 5

3.3 Factors Enabling Meta-learning

  • Residual connections and nonlinear f enable the meta-learning inner loop; with linear f, the analysis instead yields a non-data-adaptive equivalent predictor under stated conditions.For linear f and infinitely many blocks, the architecture can be interpreted as a linear predictor of a signal in colored noise.Source: §3.3 · pp. 5–6

4 Empirical Results

The evaluation tests N-BEATS trained on source datasets against diverse, distinct target time series, with target histories used for forecasting. In zero-shot tests, N-BEATS often matched or outperformed established models, and shared-weight blocks improved performance on M3 and TOURISM up to about 30 blocks.

Base datasets

  • The evaluation covers M4, M3, FRED, TOURISM, ELECTRICITY, and TRAFFIC, spanning varied univariate series and sampling frequencies.The reported comparisons use dataset-specific metrics: sMAPE for M4, FRED, and M3; MAPE for TOURISM; and ND for ELECTRICITY and TRAFFIC.Source: §4 · p. 6

Zero-shot transfer learning

  • The zero-shot task trains on one base dataset and forecasts series in a distinct target dataset, using each target series’ history as input and its test segment for error measurement.The target dataset is accessed one series at a time, and any parameter or hyperparameter adjustment is limited to that series’ history.Source: §4 · p. 6

Key results

  • N-BEATS trained on FRED beat the best M4 statistical model and matched the competition’s second entry; on M3 and TOURISM it beat M3 winner Theta.Across zero-shot comparisons, it also outperformed most dataset-tailored statistical models and DeepAR trained on M4, and performed close to or better than other neural models on ELECTRICITY and TRAFFIC.Source: §4 · pp. 6–7

Meta-learning Effects

  • With shared weights, increasing blocks improved zero-shot performance up to about 30 blocks on M3 and TOURISM, with small deterioration beyond 30.Figure 1 plots mean performance and one standard deviation interval against block count for M4-trained N-BEATS on the two targets.Source: §4 · p. 7, Figure 1 · p. 7
  • Shared-weight multiple blocks improved M3 from 12.60 to 12.44 (1.2%) and TOURISM from 20.40 to 18.82 (7.8%) versus one block.With unique weights, multiple blocks changed M3 from 12.60 to 12.40 (1.6%) and TOURISM from 20.40 to 18.91 (7.4%); adding unique weights sometimes helped marginally but often hurt.Source: §4 · p. 7

5 Discussion and Conclusion

The discussion interprets residual connections as an internal meta-learning mechanism for zero-shot transfer and considers the practical value of the resulting parameter sharing. It also identifies further study of residual architectures as future work.

Zero-shot transfer learning

  • Residual connections produce input shifts that generate compact, task-specific parameter updates for identical blocks, and a first-order approximation interprets them as iterative updates to the final linear layer.The discussion says N-BEATS can compress relevant dataset knowledge in one block rather than 10 or 30 blocks with individual weights, potentially yielding 10–30 times neural-network weight compression.Source: §5 · p. 7
  • The framework’s analysis covers residual architectures generally, but their deeper study is left for future work.The discussion suggests the analysis might explain some of their success.Source: §5 · p. 7

Memory efficiency and knowledge compression

  • A network with unique block weights includes the identical-weight network as a special case, allowing training to combine meta-learning effects with unique-weight effects..Source: §5 · p. 7

A TS Forecasting Accuracy Metrics

The appendix defines common scale-free forecasting metrics and explains how they normalize forecast errors. It distinguishes metrics that account for seasonality or support competition ranking from the machine-learning-popular ND metric.

A TS Forecasting Accuracy Metrics

  • sMAPE scales error by the average of the forecast and ground truth, while MASE scales it by the naïve predictor’s average error and thus accounts for seasonality.For monthly series, the naïve predictor uses the observation 12 periods earlier.Source: A TS Forecasting Accuracy Metrics · p. 10
  • OWA is an M4 ranking metric that normalizes sMAPE and MASE so a seasonally adjusted naïve forecast scores 1.0..Source: A TS Forecasting Accuracy Metrics · p. 10
  • ND is less standard in traditional forecasting but popular in machine-learning time-series papers; the appendix identifies observed values as ground truth and forecast values as point predictions.The summation in the last equation runs over series indices and samples.Source: A TS Forecasting Accuracy Metrics · p. 10

B.1 Architecture Details

The architecture description focuses on a generic N-BEATS model, setting aside output partitioning. Its blocks are chained in a doubly residual structure, and their forecasts combine into the final forecast.

B.1 Architecture Details

  • The generic architecture treats output partitioning as irrelevant, without loss of generality.Figure 2 depicts the adapted N-BEATS architecture.Source: §B.1 · p. 10, Figure 2 · p. 10
  • Blocks are chained together using a doubly residual architecture..Source: §B.1 · p. 10
  • The final forecast is the sum of the individual forecasts produced by the blocks..Source: §B.1 · p. 10

C The analysis of additional existing meta-learning algorithms

The framework is used to express matching networks, TADAM, and LEO as meta-learning algorithms with different task-conditioned parameter updates. Transfer setup also maps source and target frequencies to align forecast horizons and seasonal patterns.

C The analysis of additional existing meta-learning algorithms

  • Matching networks adapt embeddings and predictor components using the task training set, with attention LSTMs and a relation module in the fully conditional variant.The vanilla form uses embedding-label pairs, while the fully conditional variant updates embeddings and adds a relation module implemented with an attention LSTM.Source: C The analysis of additional existing… · p. 10
  • TADAM conditions its embedding function on task data through FiLM scale-and-shift vectors, then updates class prototypes using the conditioned embedding.Its initialization uses zero FiLM vectors and an average of class prototypes as the task embedding.Source: C The analysis of additional existing… · pp. 10–11
  • LEO meta-initializes a latent space from task data, optimizes it by gradient descent, and decodes it iteratively into task-adaptive final-layer weights.The predictor uses a fixed pretrained embedding function.Source: C The analysis of additional existing… · p. 11

D Training setup details

  • Source-to-target transfers generally preserve frequency splits, with exceptions chosen to align forecast horizons or hourly seasonality.M4 Quarterly maps to M3 Others because their horizons are both 8; M4 Hourly is used for ELECTRICITY and TRAFFIC, while upsampled FRED Monthly data are used for those hourly targets.Source: D Training setup details · p. 11

D.1 N-BEATS training setup

N-BEATS uses per-window scaling for zero-shot transfer, while DeepAR receives dataset-specific parameter tuning and scaling changes. The M4 series span multiple domains and frequencies.

D.1 N-BEATS training setup

  • In zero-shot transfer, scaling inputs and outputs by the input-window maximum prevents catastrophic failure when source and target scales differ substantially.The authors report no statistically significant accuracy effect in same-dataset train/test use.Source: §D.1 · p. 11

D.2 DeepAR training setup

  • DeepAR uses optimized dataset-specific hyperparameters, MaxAbs scaling, and median ensembles of 30 independent runs.The implementation is GluonTS 1.6; dropout is 0.1 for Electricity and 0.0 otherwise.Source: §D.2 · p. 11, Table 2 · p. 11

E.1 M4 Dataset Details

  • M4 contains heterogeneous business, financial, and economic series across forecast horizons, with broad variation in series lengths and smoothness.All series have positive observed values, so none are classified as intermittent or lumpy.Source: §E.1 · p. 11

E.2 FRED Dataset Details

The appendices document the datasets, transfer configurations, and benchmark comparisons used in the study. They also note uncertain Traffic date gaps and a check finding only negligible sequence overlap between M4 and FRED.

E.2 FRED Dataset Details

  • FRED is a paper-introduced dataset of about 290k US and international economic time series from 89 sources.It is a subset of Federal Reserve economic data; the assembled set excludes series with missing values and omits sparse frequencies and hourly data.Source: §E.2 · pp. 11–12

E.3 M3 Dataset Details

  • M3 is smaller than M4 but remains diverse across business, financial, and economic series, with wide variation in series lengths.Its series have positive observations throughout and are not intermittent or lumpy.Source: §E.3 · p. 12

E.4 TOURISM Dataset Details

  • TOURISM includes multiple sampling frequencies, broad variation in series length, and a higher fraction of erratic series than M4 and M3..Source: §E.4 · p. 12

E.5 ELECTRICITY and TRAFFIC Dataset Details

  • ELECTRICITY and TRAFFIC are hourly datasets with strong hourly and daily seasonality, aggregated respectively by sum and mean.They represent electricity use from 370 customers and occupancy rates across 963 San Francisco Bay Area freeway lanes.Source: §E.5 · p. 12

2. Jan. 21, 2008 (Martin Luther King Jr. Day)

  • Traffic date reconstruction remains uncertain: only the first six labeled gaps matched public holidays, and the authors report discrepancies in the total gap count.The passage reports 10 labeled gaps across 455 days, implying 444 rather than the 440 days in the dataset under its stated assumption.Source: §2 · p. 13, §E.5 · p. 13

E.6 Overlaps Between Datasets

  • A sequence-overlap check found shared M4–FRED sequences only, and the authors judged their counts insignificant relative to dataset sizes.The reported counts were 3 yearly, 34 quarterly, and 195 monthly sequences.Source: §E.6 · p. 14

F Empirical Results Details

  • The empirical comparisons include same-dataset N-BEATS baselines and zero-shot variants trained on M4 or FRED, with sharing switched on or off.The reported configurations use ensembles of 90 models, and seasonal-pattern mappings connect source and target datasets.Source: F Empirical Results Details · p. 14, Table 6 · p. 14

F.1 Detailed M4 Results

  • M4 results are compared with five representative competition entries, while FRED comparisons include Naïve, ARIMA, Theta, SES, and ETS using sMAPE.The M4 entries span pure machine learning, statistical, combined, and deep-learning/time-series hybrid approaches.Source: §F.1 · p. 14, §F.2 · p. 14

F.3 Detailed M3 Results

The detailed results sections specify benchmark sets and dataset-specific metrics for M3, TOURISM, ELECTRICITY, and TRAFFIC. The meta-learning analysis plots performance as block count varies, contrasting shared and distinct block weights.

F.3 Detailed M3 Results

  • M3 zero-shot transfers from FRED and M4 are reported in Table 10 using the original M3 sMAPE, alongside Theta, DOTA, EXP, ForecastPro, and DeepAR comparisons.The M3 sMAPE differs from equation (1) by omitting absolute values in the denominator.Source: §F.3 · p. 14, Table 10 · p. 16

F.4 Detailed TOURISM Results

  • TOURISM comparisons include three statistical benchmarks, two Kaggle entries, and DeepAR trained on TOURISM or transferred from M4.Table 11 reports MAPE.Source: §F.4 · p. 15, Table 11 · p. 17

F.5 Detailed ELECTRICITY Results

  • ELECTRICITY results use ND across three data splits and compare with MatFact, DeepAR, Deep State, and Deep Factors.The reported results are in Table 12.Source: §F.5 · p. 15, Table 12 · p. 17

F.6 Detailed TRAFFIC Results

  • TRAFFIC results use ND across three data splits and compare with MatFact, DeepAR, Deep State, and Deep Factors.The reported results are in Table 13.Source: §F.6 · p. 15, Table 13 · p. 18

G The Details of the Study of Meta-learning Effects

Related papers

Loading 2002.02887v3…