PaperlayerView exact revision on arXiv

Source-linked AI summary

SteerCast: Retrieval-Based Latent Steering for Decoder-Only Time Series Forecasting

Van Dai Do, Huu Hiep Nguyen, Minh Hoang Nguyen, Hung Le

arXiv:2610.11229v1cs.LG

TL;DR

Fine-tuned decoder-only time-series forecasters can retain errors, while common refinement approaches may require added training or retrieved input tokens. SteerCast retrieves latent corrections from similar training histories and applies them during generation; across diverse benchmarks and backbones, it consistently improves accuracy without further training.

  • Problem

    Fine-tuned decoder-only forecasters can retain errors in rare patterns, abrupt shifts, and long horizons, while additional training or prepending retrieved histories carries cost or context-window constraints.

    Source: §1 · p. 2

  • Method

    SteerCast stores history representations with steering vectors contrasting ground-truth and predicted continuations, then retrieves and aggregates vectors for hidden-state injection during autoregressive generation.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Across ten multivariate benchmarks and three decoder-only backbones, SteerCast consistently improves forecasting accuracy over the corresponding fine-tuned models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    SteerCast refines already fine-tuned forecasts at inference without additional training, particularly for long horizons where it mitigates error accumulation.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Time series forecasting aims to predict future values from historical observations and auxiliary features. We propose \textbf{SteerCast}, a retrieval-based latent steering method that improves decoder-only forecaster at inference time, without updating its parameters. SteerCast constructs a database from the training set by storing a representation of each history window together with a \emph{steering vector} computed in the forecaster's latent space, defined as the difference between representations induced by the ground-truth continuation and by the model's own prediction. At test time, SteerCast retrieves nearest neighbors for a query history, aggregates their steering vectors, and injects the resulting signal into the forecaster's hidden states at every step of autoregressive generation, guiding predictions toward trajectories consistent with similar training cases. Experiments across diverse multivariate benchmarks and multiple horizons show that SteerCast consistently improves forecasting accuracy over the fine-tuned backbone and retrieval-based baselines, while requiring no additional training beyond the original fine-tuning and using only the training set as a retrieval corpus.

1 Introduction

SteerCast addresses persistent errors in fine-tuned decoder-only forecasters with retrieval-guided latent steering at inference time, without parameter updates. Across three backbones and ten benchmarks, it consistently improves forecasting accuracy.

1 Introduction

  • Transformers are a dominant forecasting backbone, with prior approaches targeting efficiency, trend and seasonality, periodicity, patch-based scaling, and multivariate interactions.The cited examples include sparse attention [34], decomposition [30] [35], periodic representations [31], patch-based tokenization [20], and variate-centric tokens [15].Source: §1 · p. 1
  • Fine-tuning can leave errors in rare patterns, abrupt shifts, and long-horizon forecasts, while further training or scaling raises deployment costs and may reduce robustness under distribution shift.Prepending retrieved histories is also constrained by context windows and quadratic self-attention cost.Source: §1 · pp. 1–2
  • SteerCast retrieves similar training histories and injects aggregated, model-aware latent corrections into hidden states during generation, without updating parameters.Its steering vectors contrast representations induced by ground-truth and model-predicted continuations; similarity-based modulation and normalization stabilize injection.Source: §1 · p. 2
  • Across ten multivariate benchmarks and three decoder-only backbones, SteerCast consistently improves forecasting accuracy over the corresponding fine-tuned models.The evaluation covers multiple horizons and reports MSE and MAE.Source: §1 · p. 2

2 Related Work

Prior forecasting work adapts transformer architectures, retrieves historical examples, or intervenes on latent representations. SteerCast applies retrieval information as a latent correction to already fine-tuned decoder-only forecasters without extra input tokens or parameter updates.

Transformer-based Time Series Forecasting

  • Encoder- and decoder-based forecasting approaches have pursued efficient long-sequence modeling, patching, multivariate interactions, and autoregressive generation.Examples include Informer [34], PatchTST [20], iTransformer [15], TimeXer [27], AutoTimes [16], Timer-XL [17], and Time-MoE [23].Source: §2 · p. 2

Retrieval-based Forecasting

  • Retrieval methods guide forecasts by reusing similar histories, often through retrieved futures or a trained forecasting pipeline.RATD [14] uses retrieved segments in a diffusion framework, while RAFT [7] uses subsequent values as a predictive signal; retrieved sequences also face context-window constraints.Source: §2 · pp. 2–3
  • Unlike input-space retrieval augmentation, SteerCast uses retrieved cases as latent corrections, avoiding extra input tokens and parameter updates.Figure 1 contrasts model fine-tuning, retrieval-based fine-tuning, and SteerCast’s latent-space editing without gradient updates.Source: §2 · p. 3, Figure 1 · p. 3

Latent Steering and Representation Interventions

  • Latent-steering work intervenes on hidden representations, and SteerCast derives its directions from ground-truth versus predicted continuations.Prior examples include frozen-language-model steering [24], time-series activation edits, and in-context vectors [22].Source: §2 · p. 3

3 Method

SteerCast builds a training-set memory of history keys and latent steering vectors, retrieves and averages vectors for similar test histories, and injects the correction during autoregressive forecasting. It uses a temporal-mean hidden-state key, a continuation-based vector, and gated, normalized updates to control the intervention.

3.1 Overview

  • For each training history, SteerCast stores a retrieval key and latent correction, then retrieves and aggregates similar cases to guide hidden states during autoregressive generation.The multivariate task is handled as shared-backbone, channel-independent univariate sequences; the correction is injected at every generation step.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Database Construction

  • The retrieval key averages the forecaster’s final-layer hidden states across time to represent the history’s global context.Keys are derived from the model’s processing of a window rather than its surface statistics.Source: §3.2 · p. 4
  • Each steering vector contrasts latent representations for the same history followed by the ground-truth future versus the model’s predicted future.This displacement isolates continuation-related disagreement and summarizes the rollout because the final-token state spans the full continuation under causal masking.Source: §3.2 · p. 4

3.3 Inference Time Steering

  • At test time, SteerCast retrieves the k nearest history keys by Euclidean distance and mean-pools their stored residuals for the query.The resulting estimate is injected into the autoregressive rollout.Source: §3.3 · pp. 4–5
  • Cosine gating weakens updates when hidden states align with the steering direction, while L2 normalization and global strength bound perturbation magnitude.The gate retains a positive floor; normalization replaces query-dependent residual scale with tunable λ, and ε guards against near-zero norms.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate SteerCast on ten datasets using three decoder-only backbones, chronological train/validation/test splits, and fixed- or horizon-specific retrieval databases. SteerCast performs strongly under both database protocols, with the fixed-history database retaining gains across forecasting horizons.

4.1 Experimental Settings

  • The evaluation uses ten datasets, three decoder-only backbones, chronological test splits, and MSE and MAE across multiple forecast horizons.The database uses training data only; validation separates it from the chronologically last 20% test split.Source: §4.1 · pp. 5–6

4.2.1 Various Look-back Window Forecasting

  • For horizons 96, 192, 336, and 720, the various-history protocol uses input lengths 512, 1024, 2048, and 3072, respectively, with a separate database for each input–prediction pair.The stricter temporal-isolation analysis drops the last P + F = 608 train candidates at the 512→96 setting.Source: §4.2.1 · p. 6, Table 3 · p. 7
  • SteerCast achieves the lowest or tied-lowest MSE on 25 of 30 dataset–backbone entries and reduces MSE by 7.4% on average compared with RAF.Table 1 covers ten datasets, averaged over prediction lengths, under the various look-back window setting.Source: Table 1 · p. 6, §4.2.1 · p. 7

4.2.2 Fixed Look-back Window Forecasting

  • The fixed-history protocol uses input length 512 for every horizon and reuses the database built for the 512-96 setting.This tests whether one retrieval correction transfers across forecasting lengths.Source: §4.2.2 · p. 7
  • With one 512-96 database reused across horizons, SteerCast achieves the lowest or tied-lowest MSE on 24 of 30 entries, reducing MSE by 5.1% over FT, 5.3% over RAF, and 2.7% over RAFT.This setting reduces the need to build horizon-specific databases.Source: Table 2 · p. 6, §4.2.2 · p. 7

5 Ablation Studies and Model Analysis

The analyses test temporal isolation and distribution shift, database design, steering choices, efficiency, and additional retrieval configurations. SteerCast remains effective across these settings, while results favor sufficiently large or horizon-specific memories and carefully calibrated, distributed steering.

5.1 Temporal Isolation Ablation

  • Under stricter temporal isolation, dropping the last P + F = 608 train candidates changes MSE by only about 0.3% on ETTh1 and 1.0% on ETTh2, in opposite directions.The default database uses the chronological training portion, with validation as a buffer before test data.Source: §5.1 · p. 7

5.2 SteerCast Works Well Under Temporal Distribution Shift

  • With training memory restricted to the first 50%, SteerCast achieves the best overall performance on ETTh1 and ETTh2, with lower average error than fine-tuning and retrieval baselines.The setting uses chronologically ordered ETT data, separating the earlier training memory further from the held-out test period.Source: §5.2 · p. 7

5.3 Database Design

  • Per-horizon databases become increasingly beneficial as forecast horizons grow, while on ETTh2 larger databases lower MSE from 0.524 at 10% to 0.488 at 100%.The larger database surpasses RAFT at 50% and above; the protocol comparison averages MSE over nine datasets.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 Anatomy of the Steering Rule

  • Applying the steering correction at all transformer blocks performs best; inserting it at only one block raises average MSE by 2.0%–3.0%.The comparison is on ETTh1 with retrieval fixed at history length 512.Source: §5.4 · p. 8, Table 4 · p. 8
  • Without adaptive gating, error worsens as the horizon grows, with particularly large errors at 720 steps.The ablation replaces the cosine-based gate with a constant gate.Source: §5.4 · p. 8, Table 4 · p. 8
  • A steering strength of λ = 0.01 yields the lowest MSE and MAE across four ETT datasets; λ = 0.05 and 0.1 substantially degrade performance.The analysis attributes the degradation to over-correction and recommends tuning near 0.01.Source: §5.4 · p. 9, Figure 4 · p. 9
  • The last-layer retrieval representation yields the lowest average error and remains consistently strong across horizons, while first- and middle-layer keys are slightly worse.The gap is more evident at longer horizons.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 Efficiency

  • SteerCast adds minimal inference overhead over fine-tuning, comparable to RAF and substantially less than RAFT, while the two database protocols require nearly identical cache storage.The paper also reports further storage, latency, and time-complexity analyses in appendices.Source: §5.5 · p. 9, Table 5 · p. 9, Table 6 · p. 9

6 Conclusion

SteerCast applies a model-aware latent correction at inference time, and benchmarks across ten datasets and decoder-only backbones show consistent accuracy improvements without additional training. Qualitative examples connect relevant retrieved histories to trajectory corrections, while the retrieval-metric analysis favors Euclidean distance, especially at longer horizons.

6 Conclusion

  • SteerCast improves forecasting accuracy across ten datasets and state-of-the-art decoder-only backbones without additional training, particularly for long horizons where it mitigates error accumulation.It derives a model-aware correction from historical representation errors and injects it into hidden states during autoregressive generation.Source: §6 · p. 9

A.1 Qualitative Analysis

  • A close look-back match supplies a compatible future pattern; in the illustrated case, the steered prediction tracks the ground truth more closely as the base forecast drifts.The analysis describes the intervention as correcting the trajectory rather than merely smoothing output.Source: §A.1 · p. 13, Figure 5 · p. 13

A.2 Different Retrieval Metric

  • Euclidean retrieval is comparable to or better than cosine across horizons, with the gap widening at 336 and 720; scale information may help select better-calibrated correction neighbors.The proposed explanation also notes that anisotropic representations can make cosine scores less discriminative.Source: §A.2 · p. 13

A.3 Retrieval-key Ablation

On ETTh1, mean-pooling is the better retrieval-key representation than the last-token state at every tested horizon. The authors attribute its advantage to aggregating evidence across subsequences and reducing sensitivity to the boundary state.

A.3 Retrieval-key Ablation

  • Mean-pooling acts as a temporal regularizer, aggregating evidence across input subsequences and reducing sensitivity to the precise boundary state.The last token's full-lookback receptive field does not translate into better retrieval in this comparison.Source: §A.3 · p. 13
  • Euclidean retrieval yields lower average MSE than cosine retrieval across the three datasets, with the gap widening at longer horizons.Figure 6 compares forecasting horizons 96, 192, 336, and 720 using Time-MoE; lower MSE is better.Source: Figure 6 · p. 14
  • The last-token key incurs an average MSE penalty of +1.83% versus mean-pooling on ETTh1 across the four tested horizons.The comparison holds other hyperparameters fixed; Table 7 reports positive penalties at h=96, 192, 336, and 720.Source: §A.3 · p. 13, Table 7 · p. 14
  • The authors retain mean-pooling as the default because the last-token variant is uniformly worse on this benchmark.This conclusion matches the positive MSE penalties reported at every horizon in Table 7.Source: §A.3 · p. 14, Table 7 · p. 14

A.4 Gate Hyperparameters Ablation

These ablations find that SteerCast is robust to moderate gate and neighbor-count changes, while retrieval quality and correction design affect performance. The tested latent steering method outperforms output-level ensembling, and uniform neighbor averaging remains the preferred aggregation.

A.4 Gate Hyperparameters Ablation

  • Across the gate sweep, MSE ranges only from 0.4178 to 0.4215, a relative range under 1%, indicating robustness to moderate perturbations.The Exchange sweep varies one gate parameter at a time; larger baseline floors and margins degrade performance, while a sharper gate slightly improves it.Source: §A.4 · p. 14, Table 8 · p. 15

A.5 Fine-grained Sensitivity to the Number of Neighbors

  • The preferred neighbor count varies by dataset: ETTh1 favors small k, ETTm1 is insensitive, ETTh2 plateaus after moderate k, and Monash datasets benefit from larger k.The observed pattern is consistent with a trade-off between local correction and noise-reducing averaging.Source: §A.5 · p. 15, Table 9 · p. 15

A.6 Retrieval Quality and Failure-mode Analysis

  • Larger retrieval databases lower average neighbor distance and MSE, linking retrieval quality to forecasting error.The table caption describes this relationship as database size increases.Source: Table 10 · p. 16

A.7 Comparison with Output-level Ensembling

  • SteerCast reduces average MSE from 0.613 to 0.542 (−11.6%) versus output-level ensembling across nine datasets.The baseline averages the fine-tuned forecast with the mean of the top-4 retrieved future windows, using η = 0.1.Source: Table 11 · p. 16, §A.7 · p. 16

A.8 Distance-weighted Neighbor Aggregation

  • Softmax- and inverse-distance-weighted aggregation yield slightly worse MSE and MAE than the uniform mean on every ETT subset.The authors suggest that weighting near-identical top-k distances can amplify retrieval noise; they retain uniform averaging.Source: §A.8 · p. 16

A.9 Storage at Scale on Multivariate Datasets

The appendix documents SteerCast's storage and construction costs, decoder-only scope, experiment settings, and supplementary evaluations. It reports gains from steering pretrained checkpoints in most tested settings and against a matched output-residual baseline on two of three datasets.

A.9 Storage at Scale on Multivariate Datasets

  • The Traffic cache is 39.12 GB, about 2.2× Weather's despite approximately 13.7× as many entries; subsampling retains effectiveness with 50–75% of the full bank.Storage is described under the channel-independent formulation at history length 512.Source: §A.9 · pp. 16–17, Table 13 · p. 17

A.10 Database Construction Time

  • Cache construction runs offline once and is amortized across subsequent inference queries; its build time is dominated by running the forecaster on training windows to extract latent representations.Table 14 reports average construction time per dataset in seconds across horizons with history 512 using Time-MoE.Source: §A.10 · p. 17, Table 14 · p. 18

A.11 Why SteerCast Is Scoped to Decoder-only Forecasters

  • SteerCast is scoped to decoder-only autoregressive forecasters because encoder-based and direct-prediction models lack a per-step latent trajectory to steer.Applying the approach to those models would require a different correction signal and test-time injection rule; the paper frames this as a deliberate scope choice, not a fundamental limitation.Source: §A.11 · p. 17

A.12 Dataset Details

  • The dataset details cover multivariate ETT, Weather, and Illness series as well as diverse Monash univariate series.The appendix notes distinct sampling rates and temporal patterns across these benchmark groups.Source: §A.12 · pp. 17–18

A.13 Hyperparameters; A.14 Shared Parameters; A.14.1 Fixed Look-back Window Forecasting; A.14.2 Various Look-back Window Forecasting

  • The appendix separates fixed-lookback and varied-lookback hyperparameter tables, and states that parameters are shared across forecasting horizons within each dataset.It also provides shared parameters and universal parameter values in separate tables.Source: §A.13 · p. 18, §A.14 · p. 19, §A.14.1 · p. 19

A.15 Matched Output-Space Residual Correction

  • SteerCast beats the best tested output-residual correction on ETTh1 and ETTh2 and ties it on ETTm2, but this does not establish that all output-space methods are inferior.Increasing β beyond 0.1 progressively worsens all three results; the comparison does not isolate intervention location from normalization and adaptive gating.Source: §A.15 · p. 19, Table 19 · p. 20

A.16 Steering Pretrained Backbones Without Fine-Tuning

  • Steering reduces MSE in five of six pretrained-checkpoint settings, with the largest relative reductions on ETTh2: approximately 4.3% for Time-MoE and 3.1% for TimesFM.Weather instead regresses by approximately 1.3%; these tests use labeled training examples as retrieval memory without downstream parameter updates.Source: §A.16 · p. 20, Table 20 · p. 20

A.17 Temporal Coverage and Retrieval Stress Tests

The stress tests examine retrieval coverage and quality, latent-state changes, steering-direction construction, and retrieval runtime, while identifying limits on the empirical conclusions. The appendix also separates offline and inference-time complexity and defines its notation.

A.17 Temporal Coverage and Retrieval Stress Tests

  • SteerCast has the lowest absolute MSE at both full and first-half training coverage levels on both datasets, supporting strong absolute performance under reduced coverage.For ETTh1 memory databases built from the first, middle, or last training-split third, the absolute MSE ranges are 0.002, 0.004, 0.007, and 0.007 at horizons 96, 192, 336, and 720, respectively; these modest variations do not establish invariance to all temporal regimes.Source: §A.17 · p. 21
  • Transferring ETTh1 memory to ETTh2 raises horizon-average MSE from 0.488 to 0.499 but remains below ETTh2 FT at 0.537; excluding close neighbors generally reduces steering gains.At 90% neighbor exclusion, MSE is approximately 0.9% above FT on ETTh1 and 2.3% above FT on ETTh2; the response is not monotonic on ETTh2, and poor retrieval does not guarantee recovery to FT.Source: §A.17 · p. 21, Table 23 · p. 22, Table 24 · p. 22

A.18 Latent-State Diagnostics During Autoregressive Rollout

  • The local bound is approximately 0.0123 under the reported parameters, but it bounds an individual intervention, not complete steered-versus-unsteered trajectories.Subsequent layers and feedback through generated predictions can amplify earlier changes, so this bound alone does not establish long-horizon stability.Source: §A.18 · p. 22
  • At ETTh1 horizon 720, relative hidden-state displacement grows from 0.033 at the first forecast token to 0.277 at token 720, alongside a relatively small change in ellipsoidal coverage.Coverage declines by 0.6 percentage points unsteered and 2.2 points steered; the endpoint gap is 1.5 points. This coarse ellipsoidal diagnostic does not establish forecasting relevance or absence of cumulative drift.Source: §A.18 · pp. 22–23, Table 25 · p. 22, Table 26 · p. 23

A.19 Supervision and Query Matching in the Steering Direction

  • Query-matched SteerCast reduces MSE beyond the global supervised mean by 0.004 on ETTh2 and 0.044 on US Births; the prediction-only control slightly worsens MSE versus FT.Shuffling worsens performance relative to SteerCast on both datasets, though it retains an improvement over FT on US Births; these matched-subset comparisons do not uniquely identify a causal latent mechanism.Source: §A.19 · p. 23, Table 27 · p. 23

A.20 Euclidean Retrieval Versus Dynamic Time Warping

  • DTW takes approximately 25.4× as long on ETTh1 and 14.5× as long on US Births, while MSE differs by 0.001 in opposite directions.With a pool of 10,000, DTW evaluations did not complete within 4.5 hours per dataset; because representation and distance both differ, this does not isolate the metric alone or establish statistical equivalence.Source: §A.20 · p. 24, Table 28 · p. 24

A.21 Negative Results and Limits of the Empirical Evidence

  • SteerCast does not improve every dataset–backbone combination, and its improvement over a backbone does not imply superiority to every retrieval baseline.Examples include Time-MoE on Weather at MSE 0.238 versus FT 0.237 and RAFT 0.236; the conclusions are limited to the reported datasets, checkpoints, and retrieval perturbations.Source: §A.21 · p. 24

B Computational Complexity of SteerCast

B.1.1 Offline: Database construction

SteerCast builds a training-set database of history keys and latent corrections, then retrieves and injects an aggregate correction during autoregressive forecasting. The appendix describes the construction, storage, inference steps, and reuse across horizons.

B.1.1 Offline: Database construction

  • Each training entry stores a mean-pooled final-layer history key and a steering vector defined by the difference between latent representations induced by ground-truth and predicted continuations.Source: §B.1.1 · p. 25
  • Computing each key requires a history-window forward pass and pooling; computing each steering vector requires generating a prediction and representing both ground-truth and predicted continuations.The offline construction cost is considered over N stored windows, with latent-difference formation included after representations are obtained.Source: §B.1.1 · p. 25, F (P, F) + 2 CostF(P+F) + Ld · p. 25
  • The database stores one key in R^d and one steering vector in R^(L×d) per entry.Source: F (P, F) + 2 CostF(P+F) + Ld · p. 25, F (P, F) + 2 CostF(P+F) · p. 25

B.2 Inference: Retrieval and latent steering

  • At inference, SteerCast computes the query key, retrieves top-k Euclidean neighbors, mean-pools their vectors, then applies a cosine-gated normalized update at each generation step and layer.The described extra operations include exact distance computation and per-step, per-layer latent injection.Source: §B.2 · pp. 25–26

B.3 Multiple horizon settings

  • A database reused across horizons pays its offline construction cost once when retrieval keys and stored steering vectors are shared.Building separate databases for multiple look-back and forecast-horizon settings scales offline cost linearly with the number of settings.Source: §B.3 · p. 26

C.1 Architecture Scope

SteerCast is formulated for decoder-only autoregressive forecasters, and its neighbor count is selected on validation data. The other listed hyperparameters are largely fixed across experiments.

C.1 Architecture Scope

  • SteerCast targets decoder-only autoregressive forecasters; encoder-based and direct-prediction models would require a different correction and injection formulation, not a drop-in replacement.The paper frames this as a scope choice and notes that its named current state-of-the-art backbones are decoder-only.Source: §C.1 · p. 26

C.2 Hyperparameter Selection

  • The neighbor count k is validation-selected per dataset, adding modest overhead relative to fully training-free baselines, while the other listed hyperparameters are largely fixed.Source: §C.2 · p. 26

D Full Results

The appendix presents full evaluation tables for varying and fixed look-back settings, alongside checklist statements about the paper’s scope and limitations. It also reports point-estimate results without statistical significance measures and notes societal risks relevant to forecasting use.

D Full Results

NeurIPS Paper Checklist

  • The checklist identifies the work’s scope as decoder-only time-series forecasting and says the abstract and introduction reflect its contributions.It notes that the contributions appear in a separate introduction paragraph.Source: NeurIPS Paper Checklist · p. 29

2. Limitations

  • The reported tables contain point-estimate MSE and MAE without error bars or significance tests; multi-seed re-fine-tuning is described as computationally prohibitive.Evaluation is deterministic for a fixed checkpoint, split, and database.Source: §2 · p. 31
  • The authors warn that miscalibration under distribution shift or on rare regimes can mislead high-stakes decisions, and that latent representations of training windows warrant careful release.They advise that forecasts should not substitute for human oversight.Source: §2 · p. 33

Related papers

Loading 2610.11229v1…