PaperlayerView exact revision on arXiv

Source-linked AI summary

SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting

Guo Cheng, Zhengzhuo Xu, Chenchen Jing, Jingyi Hou

arXiv:2610.11170v1cs.LG

TL;DR

Long-horizon forecasting commonly uses flatten readouts that share a projection across future positions, while explicit historical-to-future alignment and robustness to corrupted inputs and noisy labels remain concerns. SACQ replaces that readout with staged, memory-conditioned refinement and batch-adaptive robust training. It achieves top-tier benchmark performance and reports gains across backbone swaps and corruption settings, with added inference cost.

Abstract

from arXiv · show

Long-term time series forecasting (LTSF) models predominantly employ patch-based encoders terminated by a flatten readout head that maps the entire encoded historical memory to all future steps through a single shared projection. This implicit coupling of future positions obscures position-specific historical-to-future alignment and amplifies sensitivity to corrupted inputs and extreme supervision noise. We present SACQ, a plug-in structured prediction head that replaces flatten readout while keeping the encoder unchanged. SACQ adopts a two-stage decoding pipeline: it first establishes a coarse patch-grid forecast scaffold, then refines each future position through cross-attention over historical memory and merges the attention-derived correction with the coarse scaffold via a learned per-patch gate. To stabilize optimization under long horizons and noisy labels, we further propose a batch-adaptive scaled log-cosh loss that automatically calibrates robustness to the current residual scale, suppressing outlier gradients while preserving MSE-like sensitivity for typical errors. SACQ attains top-tier test MSE/MAE across PatchTST, DLinear, and patch-Mamba backbones with only modest incremental overhead in parameters and latency. Under inference-time input corruption and training-set label-noise stress tests, SACQ substantially outperforms flatten readouts, with ablation studies validating each architectural component.

I. INTRODUCTION

The introduction identifies flatten readout as a bottleneck in long-horizon forecasting and presents SACQ as a structured, encoder-preserving alternative. It situates this proposal amid diverse backbone families and prior work on prediction heads.

I. INTRODUCTION

  • Flatten readout couples future positions through shared parameters, leaving historical-to-future alignment implicit and increasing sensitivity to corrupted inputs and extreme label residuals.The introduction describes these as representation and optimization coupling, with test-error increases under input perturbations and label contamination.Source: §I · p. 1
  • SACQ replaces flatten readout without changing the patch encoder, using staged coarse prediction, memory-conditioned cross-attention refinement, gated fusion, and batch-adaptive scaled log-cosh training.The proposed loss calibrates its robust transition region to batch residual scale, dampening extreme-outlier gradients while retaining MSE-like sensitivity to typical errors.Source: §I · pp. 1–2

A. Backbone Paradigms for LTSF

  • LTSF research spans Transformer, linear, state-space, and efficiency-oriented backbones, motivating decoder-side redesign rather than a focus on one encoder family.The introduction lists diverse model families and argues that limitations are unlikely to be tied to a single encoder family.Source: A. Backbone Paradigms for LTSF · p. 2

B. Prediction Heads as Structured Decoders

  • Prior prediction heads include lightweight linear, MLP, and flatten projections, while decoder-style forecasters motivate making historical-to-future correspondence more explicit.Prediction-side operations such as normalization and rescaling do not by themselves define explicit alignment.Source: B. Prediction Heads as Structured… · p. 2

C. Robustness Evaluation for LTSF

This section frames long-term forecasting and its readout pipeline, noting that robustness tests focused on localized input defects and label noise remain scarce. It describes SACQ's structured, coarse-to-fine decoding intuition.

C. Robustness Evaluation for LTSF

  • Readout-focused stress tests for localized input corruption and label noise remain scarce, despite clean-split benchmarks, covariate-drift methods, and evidence that flatten readouts degrade under inference-time perturbations.The related-work discussion distinguishes these stressors from covariate drift and notes TSRBench's severity-controlled spike and level-shift corruptions.Source: C. Robustness Evaluation for LTSF · pp. 2–3

A. Problem Definition

  • The task predicts the next Tf time steps jointly across C measurement channels from N past observations sampled on a uniform grid.Inputs and forecasts are represented as X ∈ R^(B×N×C) and Ŷ ∈ R^(B×Tf×C), respectively.Source: A. Problem Definition · p. 3

B. Framework Overview

  • The pipeline normalizes and patches each channel, encodes historical patches, decodes future patch targets, reconstructs per-step forecasts, and applies inverse RevIN when original-scale forecasts are needed.The backbone is kept fixed while SACQ replaces the flatten readout with coarse prediction, memory attention, and gated residual fusion.Source: B. Framework Overview · p. 3

1) Methodology intuition:

  • SACQ separates coarse patch-grid forecasting from memory-conditioned refinement, making future-position alignment explicit and merging coarse and refined signals through learned gating.The design principles also describe concatenating coarse priors with cross-attention outputs before linear projection and further refinement.Source: 1) Methodology intuition · p. 3, 2) Design principles · p. 3
  • The coarse future grid is obtained by mapping flattened historical memory through an affine transformation and reshaping the output into future patches.The number of future patches is Nf = ⌈Tf/p⌉ under contiguous, non-overlapping patching.Source: 3) Coarse Structural Initialization · p. 3

3) Coarse Structural Initialization:

SACQ initializes a future patch grid from encoded historical memory, then couples cross-attention updates with coarse patch content before further decoder refinement.

3) Coarse Structural Initialization:

  • The coarse future grid is formed by an affine map from flattened historical memory and reshaped into Nf patches, each initially an unconstrained patch-level coordinate vector.The parameterization has the same scalar count as a flatten head over the full future patch lattice.Source: 3) Coarse Structural Initialization · p. 3
  • Each coarse patch is lifted to width D with a shared affine map, while future-query positional codes continue the encoder patch-position ordering.The future positional-code segment is superposed with the coarse content embedding to initialize decoder queries.Source: 3) Coarse Structural Initialization · p. 3

4) Cross-self Coupling:

  • At each cross-self layer, attention-refined future queries are concatenated with coarse content and linearly projected into a coupled representation for further self-attention and feed-forward refinement.The projection is shared across future-patch positions and is described as synchronizing retrieval evidence with coarse structure.Source: 4) Cross-self Coupling · pp. 3–4

5) Gated Residual Readout:

SACQ fuses each coarse forecast patch with a gated residual correction. Its SAR variant changes future-query self-attention to impose a causal near-to-far dependency pattern while retaining one-shot forecasting.

5) Gated Residual Readout:

  • Each future patch's prediction adds a gate-scaled residual aligned to the coarse scaffold, with the gate computed from both the scaffold patch and its residual.The gate is a sigmoid of a linear map, with parameters shared across future patches.Source: 5) Gated Residual Readout · p. 4
  • SAR replaces bidirectional future-patch self-attention with a causal near-to-far bias, while retaining cross-attention from every future query to historical memory.The stated motivation is to avoid later future queries contributing to earlier updates and to preserve one-shot inference.Source: 6) SAR: causal self-attention over… · p. 4
  • The causal mask lets patch i aggregate only from strictly earlier future patches, creating monotonic information flow across future-patch indices.The mask changes the internal query dependency graph; it does not impose an output-level autoregressive chain, and a single forward pass still produces the full forecast.Source: 6) SAR: causal self-attention over… · p. 4
  • The fused patch grid is read in chronological order, concatenated into a forecast sequence, and truncated to the requested horizon Tf.The reconstructed forecast is produced in RevIN-normalized space before inverse RevIN is applied when original-scale values are required.Source: 7) Output Reconstruction · p. 4

7) Output Reconstruction:

SACQ uses batch-adaptive scaled log-cosh to reduce the influence of extreme residuals while retaining MSE-like sensitivity for typical errors. Its transition scale is estimated from each minibatch rather than tuned per dataset.

D. Training objective

  • MSE can give disproportionate updates to large residuals in SACQ’s staged, gated refinement, reducing training robustness under long horizons and unreliable supervision.The paper notes that this does not invalidate MSE, but motivates a more robust objective for this head.Source: D. Training objective · p. 4
  • Batch-adaptive scaled log-cosh rescales residuals before applying log-cosh, whose fixed transition otherwise does not track dataset or training-stage residual scale.The loss is locally quadratic and asymptotically L1-like.Source: D. Training objective · p. 4
  • The transition width β makes errors much smaller than β MSE-like and errors much larger than β approximately linear with damped penalties.The β^2 prefactor restores local curvature, and typical errors retain MSE sensitivity while extreme target steps receive suppressed gradients.Source: D. Training objective · pp. 4–5
  • In the final decoder layer, a linear projection maps width D to patch width p before self-attention and the FFN, which output Δ for Eq. (4).Earlier layers remain at width D.Source: Figure 2 · p. 5
  • The method estimates β from the minibatch median residual magnitude, using β = k m with k=0.5 by default and stopping gradient flow through β.The median reflects typical residual scale while resisting sporadic batch outliers; β can widen early and narrow as predictions improve, without dataset-specific calibration.Source: D. Training objective · p. 5

IV. EXPERIMENTS

The experiments assess clean benchmark performance, mechanisms, robustness, and deployment cost under a unified protocol. They use seven long-horizon datasets and specify model and optimization settings, with PatchTST as the default backbone.

IV. EXPERIMENTS

  • The evaluation covers clean benchmark performance, component and objective ablations, corruption and label-noise robustness, and context sensitivity and deployment cost.Unless otherwise stated, SACQ uses PatchTST with scaled log-cosh.Source: §IV · p. 5

1) Datasets.:

  • The benchmarks are ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Traffic, and Weather, using usual long-horizon train, validation, and test splits.The datasets range from seven ETT variates to 321 Electricity channels and 862 Traffic series; comparisons emphasize relative ranking under identical preprocessing.Source: 1) Datasets · p. 5

2) Implementation Details.:

  • Experiments use Adam, full precision, validation-based early stopping, and step-wise learning-rate decay; SACQ uses two cross-attention layers and k=0.5.Batch sizes are 128 for ETT/Weather and 8 for Traffic/ECL; learning rates are 5×10^-4 and 10^-3, respectively.Source: 2) Implementation Details · p. 5

3) Metrics and Baselines.:

SACQ is evaluated on normalized test MSE and MAE against published baselines, then tested across alternative backbones and through component ablations. It ranks near the top across benchmarks, and removing refinement components worsens both metrics.

3) Metrics and Baselines.:

1) Standard benchmark comparison.:

  • 18/24 best counts and average ranks of 1.55/1.16 on MSE/MAE place SACQ in the top tier across benchmarks, with most cells best or second-best.Weather and some ETT comparisons are close; SACQ retains favorable ranks on difficult Traffic and Electricity horizons at Tf=720.Source: 1) Standard benchmark comparison · p. 6

2) Generalization across backbones.:

  • SACQ lowers both metrics versus flatten readout on every reported patch-Linear and patch-Mamba cell, supporting its use as a plug-in head beyond PatchTST.The gains are largest on difficult ETTh2 long-horizon patch-Linear settings; patch-Mamba wins uniformly, with a wider gap at Tf=720.Source: 2) Generalization across backbones · p. 6

1) Component-wise analysis.:

  • Removing cross-attention or fixing the gate raises both metrics, while dropping concat–linear fusion hurts least but remains positive across datasets.The ablation varies one mechanism at a time while holding other settings fixed.Source: 1) Component-wise analysis · p. 6
  • Skipping the cross-self stack produces the largest error increase on every dataset, with the widest gap on high-dimensional ECL.This coarse-only variant omits the cross-self stack entirely.Source: 1) Component-wise analysis · p. 6

2) Training objective.:

Under inference-time spike and level-shift corruption, PatchSACQ retains the most best or tied-best cells as severity rises, while PatchTST degrades fastest. Objective comparisons favor scaled log-cosh in some settings, and the default loss multiplier yields stable results across decoder depths.

2) Training objective.:

  • SACQ with scaled log-cosh is strongest on Weather and long-horizon MAE, while plain log-cosh can edge it out on MSE in some ETT cells.Both log-cosh variants improve on flatten with scaled log-cosh and on SACQ with MSE, indicating complementary rather than interchangeable robust loss and structured readout.Source: 2) Training objective · p. 6

3) Hyperparameter sensitivity.:

  • At k=0.5, ETTh1 and Weather MSE vary by at most 0.001 across decoder depths L∈{1,2,3}, with MAE similarly stable.The study therefore uses L=2 and k=0.5 as its default; increasing k widens the spread across depths.Source: 3) Hyperparameter sensitivity · p. 6

D. Robustness Evaluation

  • The robustness evaluation examines inference-time input corruption, training-set defects, tail-shifted supervision, and gate dynamics.For the input-corruption test, models train on clean data and are evaluated with combined spike and level-shift corruptions while targets remain clean; severity levels are 0–5 and lookback N=512.Source: D. Robustness Evaluation · p. 6, 1) Inference-time input corruptions · p. 6

1) Inference-time input corruptions.:

  • As corruption severity rises, PatchSACQ retains the most best or tied-best cells, while PatchTST degrades fastest; the margin widens markedly at severity ≥3.ETTm1 is the main exception: PatchLGA sometimes has lower MSE at mid-to-high severity, while PatchSACQ retains the best MAE.Source: 1) Inference-time input corruptions · p. 6

2) Training-set corruptions.:

Training-set corruption tests compare flatten and structured readouts with different losses on clean test data. Structured readouts degrade more slowly under input masking, and SAR with scaled log-cosh is most robust to target spikes.

3) Causal bias under tail-shifted supervision.:

Under training-set tail shift, full SACQ outperforms the bidirectional-attention variant, with larger gains as shift severity increases. Gate behavior also indicates greater reliance on attention-derived corrections for spiked input histories.

3) Causal bias under tail-shifted supervision.:

4) Gate dynamics as adaptive correction.:

  • Spiked test histories raise mean gate α above clean-input levels, and the gap widens with training-set target-spike ratio.When test inputs are clean, curves for different training spike ratios nearly overlap; the authors link the gate increase to stronger attention-derived correction.Source: Figure 5 · p. 8, 4) Gate dynamics as adaptive correction · pp. 8–9

E. Context Sensitivity

The conclusion reports consistent accuracy improvements across baselines and encoder families. Lookback tests show SACQ+scaled log-cosh remains stable at the longest context, while the added decoder incurs a width-dependent deployment cost.

1) Lookback length.:

  • At Tf=720, flatten readouts spike near N=1440 while SACQ+scaled log-cosh remains stable.At Tf=96, configurations gain history up to N ∈ [512, 720] and then plateau; SACQ+scaled log-cosh is lowest for N ≥336.Source: 1) Lookback length · p. 9

F. Efficiency Analysis

  • At dmodel=256, the incremental decoder adds roughly 2.7M parameters and 2.6 ms per forward pass, versus ∼8.2M parameters and ∼9.2 ms for the full model.The SACQ-only curves rise more slowly than total model parameters and latency as backbone width increases.Source: F. Efficiency Analysis · pp. 9–10, Figure 7 · p. 9

V. CONCLUSION AND FUTURE WORK

  • SACQ consistently improves accuracy over strong baselines, with controlled readout swaps effective across encoder families.The conclusion identifies added inference cost from attention and gating as a limitation.Source: §V · p. 10
Loading 2610.11170v1…