PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Windowing in Transformers via Regime Incorporation for Financial Time Series

Praveen, Prince Chouhan, Keshav Maheshwari, Aman Verma

arXiv:2609.05460v1cs.CE

TL;DR

Financial forecasting is challenged by non-stationary regimes in which the useful temporal horizon changes, while standard Transformers retain fixed attention structures. The paper introduces four price-derived regime signals and two ways to integrate them into Transformers, with experiments reporting consistent improvements across five S&P 500 stocks and five evaluation metrics.

  • Problem

    Financial time series have regime-dependent temporal dependencies, but standard Transformers use fixed attention windows and static positional encodings.

    Source: §I · p. 1, §II · p. 2, §V · p. 7

  • Method

    The framework derives four generic regime signals from prices and integrates them through regime-augmented inputs or regime-modulated attention.

    Source: §I · p. 1, §V · p. 7

  • Results

    Consistent improvements over the baseline Transformer are reported across multiple assets and evaluation metrics for both proposed architectures.

    Source: E. Discussion · pp. 6–7, §V · p. 7

  • Takeaways & Limitations

    Regime-aware mechanisms provide a simple, effective way to make Transformer forecasting models more robust to non-stationary financial behavior.

    Source: §V · p. 7

Abstract

from arXiv · show

Financial time series exhibit non-stationary behavior, where the strength and extent of temporal dependencies vary across market regimes. Trending, low-volatility phases typically require long-range contextual information, whereas mean-reverting, high-volatility periods rely more heavily on short-term dynamics. Standard Transformer architectures, with fixed attention windows and static positional encodings, are therefore unable to adapt to such variations. In this work, we propose a regime-aware dynamic windowing framework that incorporates market regime information directly into the Transformer. We construct four generic regime signals from price series: volatility ratio, trend strength, local predictability ratio (LPR), and rolling autocorrelation. We incorporate these signals into the model through two mechanisms: (i) regime-augmented inputs to a standard Transformer architecture, and (ii) a modified attention layer that modulates attention weights using regime embeddings. Experiments on five S&P 500 stocks show consistent improvements across five evaluation metrics, demonstrating that regime-aware dynamic windowing enhances both interpretability and predictive performance in financial forecasting tasks.

I. INTRODUCTION

Financial markets shift among regimes with different volatility, predictability, and temporal dependencies, making fixed Transformer context insufficient. The paper proposes regime-aware descriptors and architectures that adapt representations and attention to evolving market states.

  • Financial time series are non-stationary and exhibit regime-specific temporal structures, volatility profiles, and predictability patterns.Source: §I · p. 1
  • Trending, low-volatility phases favor long-range dependencies, whereas high-volatility mean-reverting periods are dominated by short-term fluctuations.Source: §I · p. 1
  • Fixed attention windows become suboptimal because the effective temporal footprint required for forecasting changes across regimes.Source: §I · p. 1
  • The framework derives four generic regime descriptors from raw prices: volatility ratio, trend strength, local predictability ratio, and rolling lag-1 autocorrelation.Source: §I · p. 1
  • Regime information enters through augmented positional encodings or attention weights adaptively scaled to expand or contract the receptive field.Source: §I · p. 1
  • The contributions emphasize interpretable price-derived regime signals and a dynamic-window Transformer for adaptive attention spans.Source: A. Key Contributions • · p. 1

II. MOTIVATION AND BACKGROUND

Financial forecasting must handle regime-dependent temporal horizons and heterogeneous time- and frequency-domain features whose useful resolutions vary over time. The paper addresses this with two regime-aware Transformer variants that incorporate market context into token representations or attention modulation.

II. MOTIVATION AND BACKGROUND

  • Trending markets require larger receptive fields, while volatile or shock-driven regimes are governed by shorter-lived dynamics.Source: §II · p. 2
  • Standard Transformers use static positional encodings and fixed attention spans, conflicting with market regimes whose useful context windows vary over time.Source: §II · p. 2
  • Globally fixed temporal and spectral resolutions, redundant coefficients, and implicit feature weighting can increase dimensionality, overfitting risk, and learning burden.Source: §II · p. 2
  • Appending regime indicators alone may entangle them with other inputs, reducing interpretability and robustness; explicit gating or adaptive attention offers direct control.Source: §II · p. 2
  • Architecture 0 adds regime embeddings to token representations, whereas Architecture 1 masks or modulates attention weights to dynamically change attention span.Source: §II · p. 2
  • The framework connects regime-switching econometric insights with deep sequence models while addressing dynamic dependencies and the time–frequency tradeoff.Source: §II · p. 2
  • The methodology begins with a baseline Transformer and extends it with input-level and attention-level regime integration using four regime features.Source: §III · p. 2

A. Regime Time Series Formulation

The paper formulates four regime signals from price and return data to characterize volatility, trend, local predictability, and return persistence or mean reversion. Each signal provides a distinct view of evolving market structure for regime-aware modeling.

Regime signals

  • The four signals are volatility ratio, trend strength, local predictability ratio, and rolling lag-1 autocorrelation.They are constructed from asset prices and log-returns, with each targeting a different structural property.Source: A. Regime Time Series Formulation · pp. 2–3

Volatility Ratio (VR)

  • Volatility Ratio compares short-term with long-term volatility, with rising values typically indicating market stress or rapid regime shifts.The short- and long-window volatilities are represented by σs and σl.Source: A. Regime Time Series Formulation · p. 3

Trend Strength (TS)

  • Trend Strength measures the magnitude of the deviation between short- and long-term moving averages, with higher values indicating stronger directional trends.The moving averages are denoted MAs and MAl.Source: A. Regime Time Series Formulation · p. 3

Local Predictability Ratio (LPR)

  • Local Predictability Ratio compares short- and long-term deviation errors, where positive values indicate closer alignment with short-horizon structure.The errors are Es(t) = |pt − MAs(t)| and El(t) = |pt − MAl(t)|.Source: A. Regime Time Series Formulation · p. 3

Rolling Lag-1 Autocorrelation

  • Rolling lag-1 autocorrelation quantifies persistence or mean reversion in recent returns, with positive autocorrelation suggesting momentum-driven regimes and negative values indicating mean reversion or choppy markets.The regime-feature series are illustrated using BTC-USD daily data from 2018 to 2023.Source: A. Regime Time Series Formulation · pp. 2–3

B. Baseline Transformer Architecture

The baseline uses an encoder–decoder Transformer with 60-token historical input windows and scaled dot-product attention. The decoder forecasts the next price autoregressively using masked self-attention and encoder–decoder cross-attention.

C. Architecture 0: Regime-Augmented Positional Encoding

Architecture 0 injects regime information at the input level by combining token embeddings, positional encodings, and regime embeddings. This preserves the standard attention mechanism while conditioning token representations on market structure.

D. Architecture 1: Regime-Modulated Attention

Architecture 1 modifies attention logits with regime-dependent information so the effective temporal receptive field can adapt to market conditions. The adjustment can shift attention toward or away from different temporal positions.

IV. RESULTS

The evaluation uses rolling-origin expanding-window backtesting with non-overlapping future test segments. This protocol supports comparison across multiple market regimes while avoiding temporal leakage.

B. Metrics

The study evaluates three Transformer architectures across five stocks using metrics covering directional accuracy, numerical error, correlation, and trading relevance.

Evaluation metrics

  • Five metrics assess directional prediction, numerical accuracy, correlation, and trading value across the evaluated architectures.The metrics are Directional Accuracy, MSE, MAPE, Pearson Correlation, and Spearman Correlation.Source: B. Metrics · p. 5
  • Directional Accuracy measures the percentage of correctly predicted next-day return directions.It captures trading relevance rather than raw error magnitude.Source: B. Metrics · p. 5
  • MSE and MAPE measure forecasting error, with MSE penalizing large errors and MAPE measuring proportional deviation.Source: B. Metrics · p. 5
  • Pearson and Spearman correlations measure linear agreement and rank-based monotonic association between predicted and actual prices.Spearman correlation is useful when ordinal trends matter more than exact magnitudes.Source: B. Metrics · p. 5

Evaluation setup

D. Visual Comparison of Key Metrics

Figures 13 and 14 aggregate results across stocks using Directional Accuracy and MAPE to provide complementary views of trend prediction and numerical precision.

Aggregated comparisons

E. Discussion

Regime-aware mechanisms improve forecasting consistency over the baseline across assets and metrics. Input-embedding augmentation primarily improves error and correlation measures, while attention modulation further improves directional accuracy by adapting the effective receptive field.

Overall findings

  • Across nearly all stocks, regime awareness improves model robustness and forecasting consistency.Source: E. Discussion · p. 6

Architecture 0

  • Architecture 0 typically produces lower MAPE and higher correlation scores than the baseline.The discussion attributes this pattern to structural market context in the embeddings.Source: E. Discussion · p. 6

Architecture 1

  • Architecture 1 further improves Directional Accuracy by modulating attention weights according to the current regime.Source: E. Discussion · p. 6
  • Regime-derived attention biases focus the model on short-term dependencies during high volatility and extend its horizon during persistent trends.Source: E. Discussion · pp. 6–7

Interpretation

  • The framework adapts the effective attention window so historical prices are interpreted in the context of prevailing market conditions.Source: §V · p. 7

Overall findings

  • Both proposed architectures consistently outperform the baseline across multiple assets and evaluation metrics.The results support regime-driven embeddings as a simple approach for improving robustness to non-stationary financial behavior.Source: §V · p. 7, E. Discussion · p. 7

Related papers

Loading 2609.05460v1…