Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis

Dat Thanh Tran; Alexandros Iosifidis; Juho Kanniainen; Moncef Gabbouj

Source-linked AI summary

Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis

Dat Thanh Tran, Alexandros Iosifidis, Juho Kanniainen, Moncef Gabbouj

arXiv:1712.00975v1Dec 4, 2017cs.CEcs.LGq-fin.CP

TL;DR

Financial forecasting is challenging because market data are noisy and stochastic, while HFT requires inference that is both accurate and fast. The paper proposes a bilinear layer with temporal attention to focus on important time instances and improve interpretability. On a large-scale LOB dataset, a two-hidden-layer network using the layer outperforms deeper state-of-the-art architectures while requiring fewer computations.

Problem
Financial time-series forecasting is difficult because financial data are noisy and non-stationary, and HFT additionally requires accurate and fast automated inference.
Method
The paper proposes a differentiable bilinear neural-network layer augmented with temporal attention for multivariate time-series data.
Results
A two-hidden-layer TABL network exceeds the previous best Setup1 average F1 result by nearly 25% and outperforms deeper CNN and LSTM architectures on the reported LOB experiments.
Takeaways & Limitations
Temporal attention improves bilinear-network performance while highlighting the contribution of different temporal instances for further analysis.

Abstract

from arXiv · show

Financial time-series forecasting has long been a challenging problem because of the inherently noisy and stochastic nature of the market. In the High-Frequency Trading (HFT), forecasting for trading purposes is even a more challenging task since an automated inference system is required to be both accurate and fast. In this paper, we propose a neural network layer architecture that incorporates the idea of bilinear projection as well as an attention mechanism that enables the layer to detect and focus on crucial temporal information. The resulting network is highly interpretable, given its ability to highlight the importance and contribution of each temporal instance, thus allowing further analysis on the time instances of interest. Our experiments in a large-scale Limit Order Book (LOB) dataset show that a two-hidden-layer network utilizing our proposed layer outperforms by a large margin all existing state-of-the-art results coming from much deeper architectures while requiring far fewer computations.

I. INTRODUCTION

Financial forecasting is difficult because market data are noisy and non-stationary, while HFT additionally requires accurate and fast automated inference. The paper proposes an interpretable, computationally efficient temporal-attention bilinear layer and reports strong LOB results with a shallow network.

Financial market data are highly non-stationary and noisy, making financial time-series forecasting particularly difficult.
The proposed layer combines bilinear projection with temporal attention for multivariate time-series data.Its attention formulation encourages competition among neurons representing the same feature at different time instances.
Temporal attention makes the learned model interpretable by identifying which specific time instances receive attention.
The attention mechanism is theoretically and experimentally efficient enough to support practical financial forecasting systems.
A two-hidden-layer network using the proposed layer outperforms deep CNN and LSTM results by a large margin on a LOB dataset containing more than 4 million limit orders.The model also highlights contributions from different temporal information for further temporal analysis.

II. RELATED WORK

Prior work spans statistical, machine-learning, recurrent, attention-based, and bilinear approaches to financial and sequential prediction. The paper motivates TABL by combining temporal modeling and interpretability while avoiding limitations of earlier bilinear and recurrent methods.

Attention-based sequence-to-sequence RNNs generate context vectors by combining encoder hidden states with learned attention weights.The decoder uses its previous state, previous output, and current context to produce each output.

III. PROPOSED METHOD

The bilinear layer operates directly on the natural matrix representation of multivariate time series, separately modeling feature and temporal dependencies. This preserves temporal structure while using substantially fewer parameters than an equivalent MLP transformation.

A multivariate time series is represented as X ∈ R^D×T, where columns are time instances and rows contain temporal variations of individual features.The history T is the span of past observations, while H denotes the prediction horizon.
The Bilinear Layer maps an input of size D × T to an output of size D′ × T′ using separate mappings for the two modes.W1 transforms feature representations and W2 performs the temporal mapping, followed by bias and nonlinearity.
The bilinear mapping estimates DD′ + TT′ + D′T′ parameters, compared with (DT + 1)D′T′ for an equivalent MLP layer.
W1 captures interactions among different features at each time instance, whereas W2 models the temporal progression of each feature.For LOB data, this separates cross-level interactions from the evolution of prices over time.

B. Temporal Attention augmented Bilinear Layer

TABL augments bilinear projection with a learned temporal attention step that identifies important time instances and suppresses less important elements. The layer remains differentiable and uses soft attention to support learning from initially noisy features.

TABL addresses the problem of determining which time instances matter for prediction by mapping X ∈ R^D×T to Y ∈ R^D′×T′.
TABL retains separate W1 and W2 dependencies while adding an intermediate attention step learned through W and λ.Its forward pass consists of five steps.
W1 transforms each temporal instance into a new feature space while preserving temporal order.
The structured matrix W combines temporal instances, and its fixed diagonal entries of 1/T ensure each instance contributes a baseline weight.The resulting values encode relative importance among elements in the transformed representation.
Softmax normalization produces an attention mask that drives many elements near zero while retaining a few positive values.The learnable λ provides soft rather than hard attention, which can avoid discarding useful information while early features remain noisy.
Attention induces competition among neurons representing different temporal steps of the same feature, and TABL is trained jointly by back-propagation.

C. Complexity Analysis

TABL retains BL’s computation and adds attention-specific costs, while remaining less memory- and computation-intensive than the referenced attention-based Seq-RNN.

TABL adds O(T^2) memory to BL for the attention mechanism.BL memory is O(DD′ + T T′ + D′T′).
ASeq-RNN has estimated memory complexity O(3D′D + 11D′^2 + 11D′) and computation complexity O(11T D′^2 + 20T D′ + 4T^2D′ + 3T D′D + T^2).The estimate assumes equal input and output sequence lengths and GRU units.
The paper reports that ASeq-RNN has much higher memory and computational complexity than TABL.The architectures are not directly comparable, and LSTM-based variants are even more computationally demanding than the GRU-based estimate.

IV. EXPERIMENTS

The experiments evaluate the proposed architecture on mid-price movement prediction using the large-scale FI-2010 high-frequency Limit Order Book dataset under two train-test setups.

The LOB represents supply and demand through bid and ask orders arranged across price levels.The experiments consider the 10 top price levels from both sides of the LOB.
The study predicts future mid-price movement from past bid and ask prices with their respective volumes.The mid-price is defined as the mean of the best bid and best ask prices, and its movement reflects LOB and market dynamics.
FI-2010 is evaluated in two setups: nine anchored forward daily folds in Setup1 and a seven-day train, three-day test split in Setup2.Both settings use the z-score normalized data provided by the database.

B. Network Architecture

The network study compares bilinear baseline configurations with versions whose final classification layer is replaced by the proposed temporal-attention layer.

Network Architecture: Three baseline configurations A, B, and C use d = {0, 1, 2} hidden layers composed of Bilinear Layers.These configurations evaluate the bilinear structure before introducing TABL.
Network Architecture: The input is a 40×10 matrix containing prices and volumes from the top 10 bid and ask orders across 100 historical events.The 40 values correspond to prices and volumes on both sides of the book.
Network Architecture: A(BL), B(BL), and C(BL) become A(TABL), B(TABL), and C(TABL) when their last BL classification layer is replaced by TABL.The experiments place attention in the last layer rather than validating all possible attention positions.
Network Architecture: The configurations include a 120 × 5-BL notation for a Bilinear Layer with output size 120×5.

C. Experiment Settings

The experiments evaluate training choices, predictive performance, interpretability, and computational efficiency across shallow bilinear, attention-augmented, and deeper architectures.

Training and regularization: Networks used SGD or Adam, trained for up to 200 epochs with mini-batches of 256 samples.The learning rate decreased according to a predefined schedule when training loss stopped decreasing.
Training and regularization: Dropout and max-norm regularization were combined, while weight decay was unsuitable for the bilinear structure in exploratory experiments.Dropout was fixed at 0.1, and max-norm was selected from {3.0, 5.0, 7.0}.
Evaluation protocol: Because the dataset was unbalanced, hyperparameters were tuned using average per-class F1, balancing precision and recall.Setup1 reports averages over 9 folds; Setup2 averages performance over 5 training runs.
Predictive performance: In Setup1, bilinear networks outperformed competing shallow models across prediction horizons, with two-hidden-layer TABL exceeding WMTR’s previous best average F1 by nearly 25%.C(TABL) also surpassed NBoF by nearly 30% on all horizons.
Predictive performance: In Setup2, one-hidden-layer TABL matched or exceeded LSTM results, while the seven-hidden-layer CNN was greatly inferior to the proposed shallow bilinear architectures.The authors relate this contrast to bilinear networks fusing global temporal information from the first layer.
Interpretability: Adding attention improved bilinear-network performance with only a small parameter increase and enabled analysis of temporal-instance contributions through the attention mask.For A(TABL) with H = 10, attention concentrated on the second, third, and fourth most recent events, with patterns differing for stationary movement.
Efficiency: The attention mechanism added relatively little computational cost, whereas CNN and LSTM architectures required around 3× and 10× longer training or inference despite inferior performance.Table III reports forward-pass, backward-pass, and single-sample training times measured on the same machine.

V. CONCLUSIONS

The paper introduces a bilinear-projection layer augmented with temporal attention for multivariate time-series analysis. On a large-scale LOB dataset, shallow networks using the layer outperformed state-of-the-art models while improving interpretability and computational efficiency.

Conclusion: The proposed layer combines bilinear projection with temporal attention for multivariate time-series data analysis.The paper also analyzes the layer’s complexity relative to attention mechanisms in recurrent structures.
Conclusion: Two-hidden-layer networks using the proposed layer surpassed existing state-of-the-art models by a large margin on a large-scale Limit Order Book dataset.The conclusion presents this result as evidence of the architecture’s effectiveness.
Conclusion: Temporal attention improved bilinear-structure performance and enhanced interpretability by exposing the contribution of temporal instances.The paper identifies quantitative attention-pattern analysis as an opportunity for future research.

APPENDIX A TABL DERIVATIVES

The appendix establishes notation and derivative identities used to derive gradients for TABL. It then applies the chain rule through TABL components, including the activation function, to obtain the required backward-pass derivatives.

Notation: TABL derivatives use matrix-shaped Jacobians, scalar-loss gradients, identity matrices, and all-ones matrices under explicitly defined notation.The notation specifies dimensions for derivatives such as ∂Y/∂X and ∂L/∂X.
Derivative identities: The derivation relies on the Kronecker product, column-wise vectorization, and diagonal matrices formed from vector elements.These operators provide the matrix calculus tools used in the appendix’s derivative formulas.
Component derivatives: The appendix differentiates the left-hand side with respect to each right-hand-side term across Eqs. (7)–(11).This proceeds component by component, including the derivative of A with respect to E.
Activation derivative: For the final layer, Eq. (11) is written as Y = φ(Ȳ), with derivatives determined by the chosen element-wise activation function.The activation derivative is evaluated with respect to the pre-activation Ȳ.
Backward pass: During backpropagation, the required TABL derivatives are obtained by combining the stated results with chain rules.The backward pass takes ∂L/∂Y and ∂φ(Ȳ)/∂Ȳ as inputs to the derivative calculations.

APPENDIX B COMPLEXITY OF ATTENTION-BASED RNN

This appendix decomposes an attention-based RNN into encoder, memory, and decoder modules and quantifies their resource requirements. The complete attention model has quadratic-in-sequence-length computation terms alongside parameter-dependent costs.

Model structure: The attention-based sequence-to-sequence model is organized into encoder, memory, and decoder modules.The sequence indices i and j refer to input and output positions, which are assumed to have equal length.
Encoder: The encoder estimates its parameters with O(3D′D + 3D′^2 + 3D′) memory and O(T(3D′D + 3D′^2 + 8D′)) computation.These costs correspond to the encoder’s parameter estimation and sequence processing.
Memory: The memory module uses vα, Wα, and Uα, requiring O(2D′^2 + D′) memory and O(2D′^2T + 4T^2D′ + T^2) computation.The memory cost includes terms quadratic in sequence length T.
Decoder: The decoder estimates its parameters with O(6D′^2 + 7D′) memory and O(T(12D′ + 6D′^2)) computation.The listed decoder parameters include Ud_r, bd, and wout.
Total complexity: Overall, the attention model requires O(3D′D + 11D′^2 + 11D′) memory and O(11TD′^2 + 20TD′ + 4T^2D′ + 3TD′D + T^2) computation.The total computation includes both T-dependent and T^2-dependent terms.