PaperlayerView exact revision on arXiv

Source-linked AI summary

HART: Hybrid Attention-RWKV Time series Foundation Model

Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai

arXiv:2610.12883v1cs.LG

TL;DR

NAF predicts a full future horizon in one pass, but RWKV-7 degrades at long horizons when forecasting without new observations. HART alternates attention and RWKV blocks to combine access to historical tokens with recurrent temporal modeling. On GIFT-Eval and TIME-Benchmark, it improves over pure RWKV at medium and long horizons and performs comparably at short horizons, with lower aggregate MASE than a similarly sized pure Transformer.

  • Problem

    NAF supplies no observed inputs in the prediction region, and RWKV-7 shows systematic long-horizon degradation in the authors’ experiments.

    Source: §1 · p. 1

  • Method

    HART alternates Transformer and RWKV blocks to combine direct access to historical tokens with recurrent temporal modeling.

    Source: §1 · p. 2

  • Results

    HART improves over pure RWKV at medium and long horizons, maintains comparable short-horizon performance, and achieves lower aggregate MASE than a comparably sized pure Transformer.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Controlled results across GIFT-Eval and TIME-Benchmark support HART’s performance across the reported horizons against pure RWKV and a comparably sized pure Transformer.

    Source: §5 · p. 9

    The mechanistic analysis is restricted to RWKV-7, and its norm diagnostics do not directly measure retained historical information or isolate PID from other architectural effects.

    Source: §6 · p. 9

Abstract

from arXiv · show

Non-autoregressive multi-step forecasting (NAF) enables time series foundation models to predict an entire future horizon in a single forward pass. Despite its strong sequence modeling capabilities, RWKV-7 exhibits long-horizon degradation in our NAF experiments. In this setup, future patches are represented as missing inputs, requiring the recurrent state to evolve across the prediction horizon without new observations. Our analysis of RWKV-7's generalized delta rule suggests a mechanism we term Placeholder Idling Decay (PID): repeated state transitions can attenuate historical information when placeholder-driven writes provide insufficient compensation. Layer-wise analysis reveals reduced state-injection norms in most layers, while transition-matrix norms remain similar between historical and prediction regions, consistent with PID. Motivated by this analysis, we propose HART, a Hybrid Attention-RWKV Time-series Foundation Model that alternates attention and RWKV blocks, combining direct access to historical tokens with recurrent temporal modeling. Controlled experiments on GIFT-Eval and TIME-Benchmark show that HART improves over pure RWKV at medium and long horizons while maintaining comparable short-horizon performance, and achieves better aggregate performance than a comparably sized pure Transformer. With 36M parameters, HART achieves long-horizon MASE comparable to Toto-2.0-1B on GIFT-Eval (0.752 vs. 0.754), using approximately 1/28 of its parameters.

1 INTRODUCTION

NAF predicts the full future horizon without observed inputs, yet RWKV-7 degrades at long horizons. The paper proposes HART, which combines recurrent modeling with attention, and reports stronger medium- and long-horizon results than pure RWKV.

1 INTRODUCTION

  • NAF predicts the future horizon in one pass, but its prediction region has no observed inputs, leaving models to rely on internal state or historical retrieval..Source: §1 · p. 1
  • RWKV-7 exhibits systematic long-horizon degradation in the authors’ NAF experiments.Source: §1 · p. 1
  • The authors propose Placeholder Idling Decay (PID): repeated state transitions may attenuate historical information when placeholder-driven writes do not compensate sufficiently.They characterize PID through generalized delta-rule analysis and layer-wise norm measurements.Source: §1 · p. 2
  • HART alternates RWKV blocks for temporal compression and local modeling with Transformer blocks that retrieve historical information during prediction.This design combines recurrent modeling with direct access to historical tokens.Source: §1 · p. 2
  • HART improves over pure RWKV at medium and long horizons, maintains comparable short-horizon performance, and outperforms a controlled pure Transformer on aggregate metrics.The paper also reports better long-horizon and overall MASE than comparably sized baselines.Source: §1 · p. 2

2 RELATED WORK

The related work situates HART among non-autoregressive time-series foundation models, recurrent approaches, and hybrid recurrent-attention architectures.

2 RELATED WORK

  • Forecasting research spans applications including energy, transportation, finance, and healthcare, with deep learning described as the mainstream approach.Source: §2 · p. 3

2.1 TIME SERIES FOUNDATION MODELS

  • NAF generates all future steps in one pass, alleviating horizon-wise error accumulation and becoming widely adopted by time-series foundation models.Source: §2.1 · p. 3
  • TiRex is a closely related xLSTM baseline comparable in parameter count to HART’s 36M model, using the same CPM pre-training task and Pinball loss; prior RWKV time-series work targets specific applications rather than foundation models.Source: §2.1 · p. 3

2.2 HYBRID RECURRENT-ATTENTION ARCHITECTURES

  • Hybrid recurrent-attention architectures are established in language modeling and have also been applied to time series; HART motivates its hybrid design through PID analysis under NAF.In HART, attention bypasses recurrent stepwise decay during prediction, while RWKV retains a local temporal inductive bias in the historical region.Source: §2.2 · p. 3

3 METHOD

The method formulates NAF and analyzes PID in RWKV-7, then introduces HART and describes its training objective and inference pipeline.

3.1 NON-AUTOREGRESSIVE FORECASTING AND PLACEHOLDER IDLING DECAY

  • NAF uses historical patches and zero-valued, missingness-marked prediction placeholders; a recurrent backbone still updates its state across the forecast horizon in one pass.Source: §3.1 · pp. 3–4
  • PID is the potential attenuation of useful historical information when prediction transitions are not sufficiently compensated by placeholder-driven writes, without requiring faster prediction-region decay.The norm measurements characterize matrix magnitudes but do not directly establish historical information loss or transition-product contraction; the mechanistic analysis is restricted to RWKV-7.Source: §3.1 · p. 4

3.2 HART: A MECHANISM-DRIVEN HYBRID ARCHITECTURE

  • HART alternates Transformer and RWKV blocks, allowing attention to retrieve historical tokens directly and provide attention-conditioned prediction representations to recurrent updates.The backbone uses L Transformer-then-RWKV pairs; RWKV blocks model recurrent temporal structure while attention bypasses the stepwise state-decay path for historical retrieval.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 TRAINING OBJECTIVE AND PIPELINE

  • Training uses contiguous patch masking and next-patch prediction, reconstructing masked regions from available context while unmasked positions also contribute to prediction.Masked patches are placeholders whose ground-truth values supervise reconstruction, exposing the model during training to the NAF inference mode.Source: §3.3 · p. 6
  • The model predicts nine quantiles with Pinball loss on observed positions; inference supplies historical tokens and prediction placeholders in one pass, then reverses z-score normalization.The loss is computed on the standardized scale, and the historical sequence’s mean and standard deviation restore outputs to the original scale.Source: §3.3 · p. 6

4 EXPERIMENTS

Controlled experiments compare HART with pure RWKV, pure Transformer, and external baselines on GIFT-Eval and TIME-Benchmark. HART improves over RWKV at medium and long horizons, beats the pure Transformer on aggregate MASE, and shows task-dependent results in additional comparisons.

4.1 EVALUATION SETUP

  • Evaluations use GIFT-Eval and TIME-Benchmark, with geometric-mean MASE relative to seasonal naïve forecasts and horizon-grouped task sets.The three controlled models—HART, pure RWKV, and pure Transformer—use the same NAF inputs and identical components apart from their core blocks.Source: §4.1 · p. 6

4.2 MECHANISTIC EVIDENCE OF PID

  • In most layers, prediction-region Kt norms are lower while Mt norm ratios remain 1.00–1.03, a pattern consistent with PID but not proof of equal retention.Kt ratios range from 0.60 to 0.87 in the first seven layers and 0.84 to 0.97 in Layers 8–11; Layer 12 reverses to 1.34.Source: §4.2 · p. 7

4.3 MAIN RESULTS

  • HART improves over pure RWKV on Medium and Long horizons while maintaining comparable reported MASE on Short across both benchmarks.On GIFT, MASE falls from 0.749 to 0.727 on Medium and from 0.770 to 0.752 on Long; TIME gains are 0.011 on Medium and 0.032 on Long.Source: §4.3 · p. 7, Table 2 · p. 8
  • HART lowers aggregate MASE versus the pure Transformer from 0.721 to 0.710 on GIFT All and from 0.670 to 0.666 on TIME All.Both paired-median comparisons have 95% bootstrap confidence intervals excluding zero.Source: §4.3 · p. 8, Table 2 · p. 8
  • The Transformer comparison is significant at aggregate level, but horizon-level differences are significant only on GIFT Long and TIME Medium.Synthetic rankings also vary by signal: RWKV beats HART and Transformer on SE4, while HART and Transformer beat RWKV on SE3 under NAF.Source: §4.3 · p. 8
  • With unchanged pre-trained checkpoints, RWKV has the largest NAF–teacher-forced AR gap on SE1, SE2, and SE3, while gaps are signal-dependent.This control changes available information and does not isolate PID or represent deployable autoregressive forecasting.Source: §4.3 · p. 8

4.4 BENCHMARK COMPARISON

  • On GIFT Long, HART’s MASE of 0.752 is below TiRex’s 0.767 and RWKV’s 0.770, and comparable to Toto-2.0-1B’s 0.754.HART-36M uses approximately 1/28 of Toto-2.0-1B’s parameters; Xihe-1.5B has lower Medium MASE on GIFT (0.718 vs 0.727), while HART has lower Long MASE (0.752 vs 0.763).Source: §4.4 · pp. 8–9, Table 3 · p. 9
  • The leaderboard comparison shows TiRex has the lowest Short MASE among TiRex, RWKV, and HART, while HART’s Short-to-Long MASE gap is smaller.These results use official leaderboard comparisons and do not isolate architecture, training data, or inference-design effects.Source: §4.4 · p. 8, Table 3 · p. 9

5 CONCLUSION

The analysis proposes Placeholder Idling Decay (PID) as a possible attenuation of historical information under repeated state transitions with insufficient placeholder-driven writes. HART combines recurrent temporal modeling with direct access to historical tokens, and controlled benchmarks show gains over pure RWKV at medium and long horizons with comparable short-horizon performance.

5 CONCLUSION

  • PID describes how repeated state transitions may attenuate historical information when placeholder-driven writes provide insufficient compensation.Layer-wise measurements show lower write norms in most layers and similar transition norms across historical and prediction regions, consistent with the hypothesis.Source: §5 · p. 9
  • HART improves over pure RWKV on Medium and Long horizons, maintains comparable Short performance, and achieves lower aggregate MASE than a comparably sized pure Transformer.The model alternates Transformer and RWKV blocks to combine direct access to historical tokens with recurrent temporal modeling.Source: §5 · p. 9

6 LIMITATIONS AND FUTURE WORK

The mechanistic analysis is limited to RWKV-7, and norm diagnostics do not directly measure retained historical information. The appendix reports per-dataset MASE scores for all listed models on GIFT-Eval, with datasets grouped by forecast horizon.

6 LIMITATIONS AND FUTURE WORK

  • The PID analysis is restricted to RWKV-7; norm diagnostics do not directly measure retained historical information, and forecasting comparisons do not isolate PID from all architectural effects.Whether other recurrent architectures share the proposed write-compensation mechanism requires separate analysis; other block ratios and orderings remain open.Source: §6 · p. 9

A DETAILED PER-DATASET RESULTS

  • The appendix reports per-dataset MASE scores for GIFT-Eval and TIME-Benchmark; HART, RWKV, and Transformer use identical evaluation settings.External baselines are taken from the benchmarks’ official leaderboards, and lower MASE indicates better accuracy.Source: A DETAILED PER-DATASET RESULTS · p. 12

A.1 GIFT-EVAL

  • GIFT-Eval’s per-dataset tables cover 21 long-term datasets, 21 medium-term datasets, and 55 short-term datasets.The respective prediction lengths are 720–900, approximately 500, and approximately 30 time steps.Source: §A.1 · p. 12

A.2 TIME-BENCHMARK

The TIME-Benchmark appendix reports per-task MASE across long-, medium-, and short-term horizons. It also supplies synthetic-signal setup and temporal-norm measurement details for the PID analysis.

A.2 TIME-BENCHMARK

B MECHANISTIC EVIDENCE OF PID

  • The mechanistic appendix presents temporal-evolution figures for the Kt and Mt norms near the history–prediction boundary, extending the region-averaged evidence in the main text.The two figures show the corresponding norms across the boundary.Source: B MECHANISTIC EVIDENCE OF PID · p. 15

B.1 EXPERIMENTAL SETUP

  • The synthetic signal uses three sinusoidal segments with periods of 100, 250, and 80 time steps, plus Gaussian noise with standard deviation 0.1.With patch size P = 16, each sequence contains 1,024 historical patches and 64 prediction patches.Source: §B.1 · p. 15
  • The norm analysis uses 4,096 independently generated sequences and records Kt and Mt at every patch position across 12 layers.The visualization displays the last 64 historical patches and all 64 prediction patches; norms are averaged across eight heads and all sequences.Source: §B.1 · p. 15

B.2 TEMPORAL EVOLUTION OF Kt AND Mt

Across the displayed prediction region, write-norm averages are lower in most layers while transition-matrix norms remain similar, with layer-specific dynamics. These magnitudes are consistent with PID but do not directly measure net information loss.

B.2 TEMPORAL EVOLUTION OF Kt AND Mt

  • Kt prediction-to-history mean ratios are 0.60–0.66 in Layers 2, 3, and 5, and 0.84–0.97 in Layers 8–11.Layer 12’s prediction-region mean rises from approximately 3.2 to 7.2 before settling around 5.8; the norms do not establish whether it restores information.Source: §B.2 · p. 15
  • The displayed Mt norms remain similar across the boundary and prediction interval, with Layers 10–12 rising gradually from about 7.3 to 7.5.These norm differences do not determine transition-product contraction or the amount of historical information retained.Source: §B.2 · pp. 15–16
  • Prediction-region write-norm averages are lower in Layers 1–11 while transition norms remain similar; transient recoveries and Layer 12’s reversal qualify the pattern.The figures show temporal evolution around the history–prediction boundary.Source: §B.2 · p. 16, Figure 3 · p. 16, Figure 4 · p. 17

C SYNTHETIC EVIDENCE AND TEACHER-FORCED COMPARISONS

  • The NAF–teacher-forced comparison is used as a diagnostic of sensitivity to available observations, not as a comparison with free-running autoregressive forecasting.The appendix reports the full results in Table 12.Source: C SYNTHETIC EVIDENCE AND… · pp. 17–18

C.1 EXPERIMENTAL SETUP

Six synthetic experiments compare trained architectures across signal structures at three forecast horizons, using a shared 16k-step context and MASE evaluation.

C.1 EXPERIMENTAL SETUP

  • The six experiments use 16,384 historical steps, prediction horizons of 32, 480, or 720 steps, 256 sequences each, and MASE; only NAF matches the main benchmark setup.The teacher-forced AR control supplies real future values, and the models use pre-trained checkpoints without additional synthetic-signal fine-tuning.Source: §C.1 · p. 18
  • SE1–SE3 test regime changes, an ultra-long cycle with local oscillation, and amplitude modulation with phase jumps, respectively.Their structures combine locally changing periodic patterns, long-span drift plus high-frequency variation, or fast oscillations with slow envelopes and discrete phase shifts.Source: §C.1 · p. 18
  • SE4 combines eight frequencies, a chirp, amplitude modulation, structural breaks, and a linear trend under noise, making it the most complex signal.The design includes both predictable deterministic structure and non-stationary components.Source: §C.1 · p. 18
  • SE5 tests a weak period-200 sinusoid obscured by strong noise, while SE6 tests a constant level with noisy observations.These designs contrast a low-signal-to-noise periodic component with a signal having no underlying temporal variation.Source: §C.1 · p. 18

C.2 SIGNAL-DEPENDENT ARCHITECTURE PERFORMANCE

Architecture rankings vary across synthetic signal regimes: HART leads on SE3, while RWKV leads on SE4. On real benchmarks, HART improves aggregate MASE over Transformer and medium- and long-horizon results over RWKV.

C.2 SIGNAL-DEPENDENT ARCHITECTURE PERFORMANCE

  • On SE4, RWKV has the lowest Long MASE at 1.058, versus HART at 1.148 and Transformer at 1.151.RWKV also beats Transformer under teacher-forced AR, 0.631 vs. 0.641; HART slightly improves over Transformer under NAF but remains behind RWKV.Source: §C.2 · p. 19
  • SE3 couples fast oscillations, a slow envelope, and phase jumps, requiring local details and global shape to be modeled together.RWKV’s Long NAF–AR gap is 0.316 on SE3, compatible with both task-dependent architecture differences and sensitivity to placeholders.Source: §C.2 · p. 19
  • On SE3 under NAF, HART has the lowest Long MASE at 0.265, followed by Transformer at 0.306 and RWKV at 0.581.RWKV also has higher Long MASE under teacher-forced AR than HART and Transformer, 0.265 vs. 0.169 and 0.172.Source: §C.2 · p. 19
  • On real benchmarks, HART improves aggregate MASE over pure Transformer and outperforms pure RWKV at Medium and Long horizons.The differing SE3 and SE4 rankings motivate evaluating hybrid architectures across signal regimes rather than expecting universal dominance.Source: §C.2 · p. 19

C.3 INTERPRETING SENSITIVITY TO MISSING FUTURE OBSERVATIONS

The section compares NAF with teacher-forced AR across synthetic regimes, describes the pre-training data, and explains how contiguous patch masking exposes models to placeholder inputs during training.

C.3 INTERPRETING SENSITIVITY TO MISSING FUTURE OBSERVATIONS

  • On SE5, absolute NAF–AR gaps are at most 0.001, and on SE6 they are zero, so the structured-signal loss is not uniform across tested regimes.The controls do not by themselves exclude overfitting, distribution shift, or other mechanisms; the NAF–AR comparison also changes available information and input pattern, so it does not isolate PID from all forecasting error.Source: §C.3 · p. 19
  • On Long, RWKV has larger NAF–AR gaps than HART and Transformer on each structured signal: 0.060 on SE1, 0.095 on SE2, and 0.316 on SE3.This ranking indicates greater evaluated RWKV sensitivity to removing future observations, consistent with PID motivation but not ruling out task-difficulty or input-distribution effects.Source: §C.3 · p. 19, Table 12 · p. 20

D PRE-TRAINING SETUP

  • The three architectures share pre-training data, optimization strategy, and random seed; training uses a corpus of approximately 86.7M sequences, roughly half real and half synthetic.The corpus includes about 43.7M real sequences and 43M synthetic sequences.Source: D PRE-TRAINING SETUP · p. 20, §D.1 · p. 20

D.1 PRE-TRAINING DATA

  • Real pre-training data come from GIFT-Eval, Chronos, and Code15, with temporal overlap against GIFT-Eval test sets removed from Chronos data.Real series are truncated to at most 32k steps, using sliding windows for longer series.Source: §D.1 · p. 20
  • Synthetic pre-training data comprise Chronos-style Gaussian-process sequences, long-range GP-interpolated sequences, and non-uniformly sampled sequences with roughly 10% of points dropped.The families cover multi-scale fluctuations, long-range continuous patterns, and incomplete observations, complementing real data.Source: §D.1 · pp. 20–21

D.2 CONTIGUOUS PATCH MASKING

  • Contiguous Patch Masking samples contiguous groups of patches for masking, replacing masked inputs with placeholders while retaining their ground truth for loss supervision.Unmasked positions also contribute to the loss, and the task trains reconstruction of contiguous futures from arbitrary historical positions rather than only extrapolation at the sequence end.Source: §D.2 · p. 21

D.3 OPTIMIZER AND TRAINING SCHEDULE

The appendix specifies shared optimization settings and the computational components of HART’s patch embedding, attention, and RWKV blocks, noting the limited changes from the original architectures.

D.3 OPTIMIZER AND TRAINING SCHEDULE

  • Training runs for 220k steps, one pass over the corpus, with a linear warmup then cosine learning-rate decay; HART-36M, RWKV-36M, and Transformer-35M share data, strategy, and seed.The schedule warms from zero to 1 × 10^-3 over 8k steps, then decays to 3.3 × 10^-6 over 212k steps; the setup does not itself guarantee absence of overfitting.Source: §D.3 · p. 21

E DETAILED COMPUTATIONAL FORMULAE FOR EACH BLOCK

  • Patch embedding normalizes the input series using historical mean and standard deviation, then projects patch values and observation and missingness masks into the hidden dimension.Historical statistics are retained for inverse normalization; shared embedding weights process historical patches and prediction placeholders.Source: E DETAILED COMPUTATIONAL FORMULAE FOR… · p. 22

E.1 ATTENTION BLOCK

  • The Attention Block combines RMSNorm, gated self-attention, and a SwiGLU feed-forward network with residual connections.Queries and keys receive RoPE encoding, causal self-attention is computed, and a sigmoid gate modulates its output.Source: §E.1 · p. 22, §E.1.1 · p. 22, §E.1.2 · p. 22

E.2 RWKV BLOCK

  • The RWKV Block retains RWKV-7 TimeMix and its generalized delta rule, with LayerNorm and a SwiGLU channel mixer connected residually.Its stated modification from RWKV-7 is replacing the RWKV-FFN activation from ReLU2 with SwiGLU.Source: §E.2 · p. 23, §E.2.4 · p. 24, §E.3 · p. 24

E.2.1 TOKEN SHIFT

  • Token shift linearly interpolates each current token with the previous token after normalization and before TimeMix or the RWKV feed-forward submodule.The mixing factors are learnable, with separate factors for the RWKV-FFN.Source: §E.2.1 · p. 23

E.2.2 LORA PARAMETERIZATION IN TIMEMIX

  • TimeMix quantities are computed from shifted inputs using learned projections, LoRA adapters, and nonlinear transformations, with removal and replacement keys and value-residual interpolation.The replacement key interpolates between 1 and the attention-weight vector; value residual learning interpolates the final value in later layers.Source: §E.2.2 · p. 23

E.2.3 GDR STATE UPDATE AND ATTENTION READOUT

  • The GDR state update and readout operate per head, normalize the removal key within each head, and read the attention result from the state via the receptance vector.The resulting heads are concatenated before gating and output projection.Source: §E.2.3 · p. 23

E.3 SUMMARY OF ARCHITECTURAL DIFFERENCES FROM THE ORIGINALS

  • HART assembles attention and RWKV blocks, using gated self-attention and SwiGLU in attention while changing only the RWKV-FFN activation from ReLU2 to SwiGLU.The RWKV TimeMix design is retained.Source: §E.3 · p. 24
Loading 2610.12883v1…