Source-linked AI summary
Dynamic Windowing in Transformers via Regime Incorporation for Financial Time Series
Praveen, Prince Chouhan, Keshav Maheshwari, Aman Verma
TL;DR
Financial forecasting is challenged by non-stationary regimes in which the useful temporal horizon changes, while standard Transformers retain fixed attention structures. The paper introduces four price-derived regime signals and two ways to integrate them into Transformers, with experiments reporting consistent improvements across five S&P 500 stocks and five evaluation metrics.
Problem
Financial time series have regime-dependent temporal dependencies, but standard Transformers use fixed attention windows and static positional encodings.
Method
The framework derives four generic regime signals from prices and integrates them through regime-augmented inputs or regime-modulated attention.
Results
Consistent improvements over the baseline Transformer are reported across multiple assets and evaluation metrics for both proposed architectures.
Takeaways & Limitations
Regime-aware mechanisms provide a simple, effective way to make Transformer forecasting models more robust to non-stationary financial behavior.
Abstract
from arXiv · showhide
Financial time series exhibit non-stationary behavior, where the strength and extent of temporal dependencies vary across market regimes. Trending, low-volatility phases typically require long-range contextual information, whereas mean-reverting, high-volatility periods rely more heavily on short-term dynamics. Standard Transformer architectures, with fixed attention windows and static positional encodings, are therefore unable to adapt to such variations. In this work, we propose a regime-aware dynamic windowing framework that incorporates market regime information directly into the Transformer. We construct four generic regime signals from price series: volatility ratio, trend strength, local predictability ratio (LPR), and rolling autocorrelation. We incorporate these signals into the model through two mechanisms: (i) regime-augmented inputs to a standard Transformer architecture, and (ii) a modified attention layer that modulates attention weights using regime embeddings. Experiments on five S&P 500 stocks show consistent improvements across five evaluation metrics, demonstrating that regime-aware dynamic windowing enhances both interpretability and predictive performance in financial forecasting tasks.
I. INTRODUCTION
Financial markets shift among regimes with different volatility, predictability, and temporal dependencies, making fixed Transformer context insufficient. The paper proposes regime-aware descriptors and architectures that adapt representations and attention to evolving market states.
- Financial time series are non-stationary and exhibit regime-specific temporal structures, volatility profiles, and predictability patterns.
- Trending, low-volatility phases favor long-range dependencies, whereas high-volatility mean-reverting periods are dominated by short-term fluctuations.
- Fixed attention windows become suboptimal because the effective temporal footprint required for forecasting changes across regimes.
- The framework derives four generic regime descriptors from raw prices: volatility ratio, trend strength, local predictability ratio, and rolling lag-1 autocorrelation.
- Regime information enters through augmented positional encodings or attention weights adaptively scaled to expand or contract the receptive field.
- The contributions emphasize interpretable price-derived regime signals and a dynamic-window Transformer for adaptive attention spans.
II. MOTIVATION AND BACKGROUND
Financial forecasting must handle regime-dependent temporal horizons and heterogeneous time- and frequency-domain features whose useful resolutions vary over time. The paper addresses this with two regime-aware Transformer variants that incorporate market context into token representations or attention modulation.
- II. MOTIVATION AND BACKGROUND: Trending markets require larger receptive fields, while volatile or shock-driven regimes are governed by shorter-lived dynamics.
- II. MOTIVATION AND BACKGROUND: Standard Transformers use static positional encodings and fixed attention spans, conflicting with market regimes whose useful context windows vary over time.
- II. MOTIVATION AND BACKGROUND: Globally fixed temporal and spectral resolutions, redundant coefficients, and implicit feature weighting can increase dimensionality, overfitting risk, and learning burden.
- II. MOTIVATION AND BACKGROUND: Appending regime indicators alone may entangle them with other inputs, reducing interpretability and robustness; explicit gating or adaptive attention offers direct control.
- II. MOTIVATION AND BACKGROUND: Architecture 0 adds regime embeddings to token representations, whereas Architecture 1 masks or modulates attention weights to dynamically change attention span.
- II. MOTIVATION AND BACKGROUND: The framework connects regime-switching econometric insights with deep sequence models while addressing dynamic dependencies and the time–frequency tradeoff.
- II. MOTIVATION AND BACKGROUND: The methodology begins with a baseline Transformer and extends it with input-level and attention-level regime integration using four regime features.
A. Regime Time Series Formulation
The paper formulates four regime signals from price and return data to characterize volatility, trend, local predictability, and return persistence or mean reversion. Each signal provides a distinct view of evolving market structure for regime-aware modeling.
- Regime signals: The four signals are volatility ratio, trend strength, local predictability ratio, and rolling lag-1 autocorrelation.They are constructed from asset prices and log-returns, with each targeting a different structural property.
- Volatility Ratio (VR): Volatility Ratio compares short-term with long-term volatility, with rising values typically indicating market stress or rapid regime shifts.The short- and long-window volatilities are represented by σs and σl.
- Trend Strength (TS): Trend Strength measures the magnitude of the deviation between short- and long-term moving averages, with higher values indicating stronger directional trends.The moving averages are denoted MAs and MAl.
- Local Predictability Ratio (LPR): Local Predictability Ratio compares short- and long-term deviation errors, where positive values indicate closer alignment with short-horizon structure.The errors are Es(t) = |pt − MAs(t)| and El(t) = |pt − MAl(t)|.
- Rolling Lag-1 Autocorrelation: Rolling lag-1 autocorrelation quantifies persistence or mean reversion in recent returns, with positive autocorrelation suggesting momentum-driven regimes and negative values indicating mean reversion or choppy markets.The regime-feature series are illustrated using BTC-USD daily data from 2018 to 2023.
B. Baseline Transformer Architecture
The baseline uses an encoder–decoder Transformer with 60-token historical input windows and scaled dot-product attention. The decoder forecasts the next price autoregressively using masked self-attention and encoder–decoder cross-attention.
- Each input window contains 60 historical prices that are embedded and processed by stacked encoder self-attention and feedforward layers.
- The encoder extracts temporal dependencies across the full fixed input window.
- The decoder autoregressively generates the next price while masked self-attention prevents access to future values.
- Cross-attention lets the decoder use information from the encoder outputs for conditional forecasting.
- Scaled dot-product attention computes weights from projected queries, keys, and values, defining the baseline mechanism for comparison.
C. Architecture 0: Regime-Augmented Positional Encoding
Architecture 0 injects regime information at the input level by combining token embeddings, positional encodings, and regime embeddings. This preserves the standard attention mechanism while conditioning token representations on market structure.
- Architecture 0 enriches each token representation with regime descriptors before self-attention operations.
- The combined representation uses token embedding Xt, positional encoding PEt, and regime embedding eR_t.
- Positional encodings preserve temporal ordering, while regime embeddings distinguish structurally different market environments.
- The architecture modifies inputs without changing the Transformer’s attention mechanism.
D. Architecture 1: Regime-Modulated Attention
Architecture 1 modifies attention logits with regime-dependent information so the effective temporal receptive field can adapt to market conditions. The adjustment can shift attention toward or away from different temporal positions.
- Standard self-attention uses QK^T token similarity to determine which past positions each query attends to, producing fixed inductive behavior.
- A regime-derived bias term W_rR^T is added inside attention logits to alter token affinities context-dependently.
- During high volatility, the learned bias may suppress long-range attention and strengthen short-range interactions, shrinking the active receptive field.
- The regime term serves as a dynamic mask that shifts attention weights toward or away from different temporal positions.
IV. RESULTS
The evaluation uses rolling-origin expanding-window backtesting with non-overlapping future test segments. This protocol supports comparison across multiple market regimes while avoiding temporal leakage.
- Rolling-origin backtesting expands the training set forward while testing on a non-overlapping future segment.
- The procedure is designed to prevent temporal leakage and mimic practical forecasting deployment.
- Each fold pair (a, b) trains on the first a fraction of data and tests on the next b −a fraction.
- Concatenating predictions across folds enables computation of global evaluation metrics.
- The protocol compares models across trending, sideways, high-volatility, and reversal periods.
B. Metrics
The study evaluates three Transformer architectures across five stocks using metrics covering directional accuracy, numerical error, correlation, and trading relevance.
- Evaluation metrics: Five metrics assess directional prediction, numerical accuracy, correlation, and trading value across the evaluated architectures.The metrics are Directional Accuracy, MSE, MAPE, Pearson Correlation, and Spearman Correlation.
- Evaluation metrics: Directional Accuracy measures the percentage of correctly predicted next-day return directions.It captures trading relevance rather than raw error magnitude.
- Evaluation metrics: MSE and MAPE measure forecasting error, with MSE penalizing large errors and MAPE measuring proportional deviation.
- Evaluation metrics: Pearson and Spearman correlations measure linear agreement and rank-based monotonic association between predicted and actual prices.Spearman correlation is useful when ordinal trends matter more than exact magnitudes.
- Evaluation setup: Table I compares the Baseline Transformer, Architecture 0, and Architecture 1 across five S&P 500 stocks.The supplied figure captions identify rolling-origin evaluations for BTC-USD using each architecture.
D. Visual Comparison of Key Metrics
Figures 13 and 14 aggregate results across stocks using Directional Accuracy and MAPE to provide complementary views of trend prediction and numerical precision.
- Aggregated comparisons: Figures 13 and 14 visualize aggregated Directional Accuracy and MAPE results across all stocks.
- Aggregated comparisons: Directional Accuracy summarizes trend-prediction performance, whereas MAPE summarizes numeric precision.
E. Discussion
Regime-aware mechanisms improve forecasting consistency over the baseline across assets and metrics. Input-embedding augmentation primarily improves error and correlation measures, while attention modulation further improves directional accuracy by adapting the effective receptive field.
- Overall findings: Across nearly all stocks, regime awareness improves model robustness and forecasting consistency.
- Architecture 0: Architecture 0 typically produces lower MAPE and higher correlation scores than the baseline.The discussion attributes this pattern to structural market context in the embeddings.
- Architecture 1: Architecture 1 further improves Directional Accuracy by modulating attention weights according to the current regime.
- Architecture 1: Regime-derived attention biases focus the model on short-term dependencies during high volatility and extend its horizon during persistent trends.
- Interpretation: The framework adapts the effective attention window so historical prices are interpreted in the context of prevailing market conditions.
- Overall findings: Both proposed architectures consistently outperform the baseline across multiple assets and evaluation metrics.The results support regime-driven embeddings as a simple approach for improving robustness to non-stationary financial behavior.