Source-linked AI summary
Autoregressive Convolutional Neural Networks for Asynchronous Time Series
Mikołaj Bińkowski, Gautier Marti, Philippe Donnat
TL;DR
The paper addresses prediction of noisy multivariate time series observed asynchronously, where assigning appropriate weights to past observations is difficult. It proposes Significance-Offset CNN, combining autoregressive weighting with convolutional, data-dependent functions. Across several asynchronous forecasting settings, the architecture outperforms popular convolutional and recurrent networks, while its scope includes assumptions that motivate further extensions.
Problem
Asynchronous multivariate series make it difficult to assign appropriate weights to past observations, and aligning them at a fixed frequency can lose information or enlarge the dataset.
Method
Significance-Offset CNN combines convolutional networks with an autoregressive-style weighting mechanism whose weights depend nonlinearly on the inputs.
Results
The proposed architecture successfully forecasts several asynchronous time series, outperforming popular convolutional and recurrent networks.
Takeaways & Limitations
The weighting mechanism provides a successful architecture for regression on asynchronous signals with high noise.
Takeaways & Limitations
The architecture assumes independent offset values for each past observation, and extending this assumption requires further empirical study.
Abstract
from arXiv · showhide
We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks. It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are datadependent functions learnt through a convolutional network. The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive series and UCI household electricity consumption dataset. The proposed architecture achieves promising results as compared to convolutional and recurrent neural networks.
1. Introduction
The paper studies convolutional networks for autoregressive prediction of noisy, multivariate time series, especially financial data observed asynchronously from multiple sources. It proposes a convolutional extension of autoregressive models with nonlinear weighting and compares it with convolutional and recurrent benchmarks.
- The paper examines CNNs for modeling the conditional mean of future observations in multivariate, noisy time series.This is framed as an autoregression problem.
- Financial time series are challenging because of low signal-to-noise ratios, heavy-tailed distributions, and unresolved return predictability.
- Asynchronous market observations combine different sources with distinct biases and noise, while strong correlations and lead-lag relationships complicate signal recovery.
- The proposed Significance-Offset CNN extends standard autoregressive models with a nonlinear weighting mechanism inspired by neural-network gating systems.The model is evaluated against multilayer CNNs, LSTMs, and Phased LSTM.
2. Related work
Related work spans econometric forecasting, machine-learning approaches for irregular time series, and neural gating or attention mechanisms. The paper positions its approach as weighting separate time-series inputs with learned CNN-based functions rather than recurrent layers or learned projections.
- Econometric forecasting commonly uses stochastic models such as AR, ARIMA, and GARCH, while machine learning emphasizes out-of-sample prediction.The paper notes unstable parameters and poor out-of-sample performance for econometric models on financial time series.
- Machine-learning research has used Gaussian processes for irregularly sampled series, while deep-learning research on financial time-series forecasting remains comparatively scarce.
- Neural gating mechanisms control how much candidate outputs pass through a layer and support recurrent architectures such as LSTM and GRU.
- The proposed gating system weights candidate predictors and is most closely related to softmax gating in MuFuRU.Its candidate outputs are linear functions of the inputs.
- Unlike attention networks, the approach weights separate time-series steps using learned multilayer CNN functions and does not use recurrent layers.
3. Motivation
Irregular, multivariate observations make appropriate autoregressive weighting difficult, while fixed-frequency alignment can lose information or greatly enlarge the dataset. The proposed model combines an autoregressive approach with a neural network to learn data-dependent weights.
- Challenges of asynchronous forecasting: Irregular sampling makes even simple autoregressive processes require highly nonlinear conditional expectations.For AR(2), the coefficients after irregular sampling depend on the sampling interval through rational functions of the original parameters.
- Challenges of asynchronous forecasting: Asynchronous multivariate data complicates weighting because separate dimensions are observed at different times and may be biased or noisy.The same underlying signal can arrive from multiple correlated sources with different biases, noise levels, and possible lead-lag relationships.
- Challenges of asynchronous forecasting: Fixed-frequency alignment can either lose information at low frequency or cause prohibitive dataset enlargement when observation durations vary widely.The enlargement problem is illustrated in Figure 2a and motivates an alternative representation for asynchronous series.
- Limits of existing approaches: LSTM-based handling of values and durations may require several nonlinear layers to compute weights even when past observations only need to be memorized.This creates an imbalance between the network's memory and nonlinearity requirements.
- Proposed direction: SOCNN combines a simple autoregressive approach with a neural network to learn meaningful data-dependent weights.The formulation extends the weighting idea by allowing neural-network transformations of past observations.
4. Model Architecture
SOCNN uses convolutional offset and significance subnetworks to form weighted sums of adjusted past observations while preserving the time dimension. Its representation incorporates durations and indicators so learned significance can reflect asynchronous timing.
- Inputs and outputs: SOCNN predicts selected features from a window of past multivariate observations represented as x_n and target values y_n.The model operates on a sequence of consecutive input pairs and predicts a subset of the current features.
- Weighted prediction: The architecture uses neural networks F and S together with row-wise normalized significance weights and element-wise multiplication.The normalization makes the significance weights sum to one across the M past observations for each row.
- Weighted prediction: Each output is a weighted sum across past steps, with F supplying adjusted regressors and S supplying learned significance across the input window.The summation is performed over matrix columns, so each output coordinate combines the corresponding row of F's outputs.
- Interpretation: SOCNN separates temporal dependence, local observation significance, and position-independent predictors through its weighting construction.The significance network is convolutional, while the offset predictors do not depend on the observation's relative time position.
- Asynchronous representation: For asynchronous data, durations and observation times are included as features so the significance network can learn their relevance to past observations.This avoids aligning observations by duplicating or interpolating them, which can increase input size and model complexity.
- Training objective: The training objective combines squared prediction error with an auxiliary mean-squared-error term for intermediate offset predictions.The auxiliary term is controlled by α, with its effect compared against α = 0 in the empirical analysis.
5. Experiments
The experiments evaluate SOCNN on artificial, household electricity, and proprietary asynchronous quote datasets, alongside CNN, recurrent, residual, and linear benchmarks. SOCNN generally performs best on asynchronous, electricity, and quotes data, while component analyses highlight the importance of its significance network and auxiliary loss.
- Datasets: The evaluation covers artificial synchronous and asynchronous series, UCI household electricity data, and proprietary credit-derivative quote data.The artificial datasets vary the number of sources, while the real-world datasets contain irregularly observed multivariate signals.
- Datasets: The quote dataset contains 2.1 million quotes from 28 market participants, with predictions made separately for selected source-and-direction tasks.Each quote includes price, source indicators, direction, and elapsed time from the previous quote.
- Results: SOCNN significantly outperforms benchmark networks on asynchronous, electricity, and quotes datasets, while nearly matching them on synchronous datasets.The authors attribute the synchronous parity to less nonlinear weighting of past values, which limits the significance network’s advantage.
- Component analysis: The significance network has a crucial performance role, whereas increasing offset-network depth has negligible or negative impact.This supports the interpretation that learning appropriate weights for past observations is harder than obtaining predictors from individual past values.
- Results: For the quotes dataset, SOCNN is best on 13 of 15 tasks and is the only model that always beats the VAR benchmark.Other networks often struggled to exceed the simple linear benchmark, with occasional unstable or very high test errors.
- Component analysis: Higher auxiliary-loss weights improve asynchronous-dataset test error, while SOCNN generally shows lower early-training and quote-dataset error variance.A small positive auxiliary weight also stabilizes test error in many cases, and SOCNN is the most robust model under added noise.
6. Conclusion and discussion
The proposed architecture combines a weighting mechanism with convolutional networks for noisy asynchronous time-series regression. It outperformed popular convolutional and recurrent networks on several asynchronous series, while further extensions and broader real-world testing remain open.
- The proposed architecture couples a weighting mechanism with convolutional networks for regression on asynchronous signals with high noise.
- The approach successfully forecast several asynchronous time series while outperforming popular convolutional and recurrent networks.
- The model could be extended with intermediate weighting layers of the same type.
- A further generalization would replace independent offset values with non-1x1 convolutional kernels in the offset sub-network, requiring additional empirical studies.
- The authors plan to test the architecture on additional real-life datasets with relevant characteristics.
A. Nonlinearity in the asynchronously sampled autoregressive time series
This section establishes, through an induction argument, how an asynchronously sampled AR(2) process can be represented with rational-function coefficients and accumulated innovation terms. It also describes a noise-robustness evaluation that perturbs selected input steps and compares trained models on unchanged targets.
- For an AR(2) series, the coefficients a_k and b_k are rational functions of the original parameters a and b.
- The proof proceeds by induction, showing the stated representation at the base case and then deriving the k+1 case from the k case.
- The residual term E_k(t) is a linear combination of innovations ε(t−i) for i from 0 through k−2.
- Robustness is evaluated by adding noise to selected input observations while assuming the output remains unchanged.
- The robustness procedure randomly selects 6,000 observations, split equally between training and test data, before perturbing inputs and evaluating each trained model.
C. Artificial data generation
The artificial datasets simulate noisy copies of autoregressive signals observed at synchronous or asynchronous times. The construction varies noise, sampling times, source selection, and the number of observed copies to test forecasting under asynchronous conditions.
- The construction also defines noisy observation functions and source-selection probabilities, including a possible dependence on q^K.
- The simulation begins with a stationary AR(10) series, which is copied K times and independently perturbed with additive or multiplicative Gaussian or Binomial noise.
- Asynchronous observations use a random time process whose consecutive event gaps are independent exponential variables.
- At each sampled time, one noisy copy is selected, producing a multivariate series containing values, durations, and source indicators.
- Both synchronous and asynchronous processes are simulated with N = 10,000 observations and K ∈ {16, 64}.
D. Household electricity dataset
The household electricity data are converted into an asynchronous, feature-subsampled dataset by retaining selected time intervals and one randomly available feature per timestep. Each resulting observation records temporal information, feature availability indicators, and the observed value.
- The original household electricity dataset contains seven power-related features plus date and time information.
- Deterministic sampling retains observations according to a periodic sequence of durations ranging from 1 to 7 minutes.
- At each retained timestep, one of seven features is selected with feature probabilities proportional to [1, 1.5, 1.52, 1.56].
- Each asynchronous observation is a 10-dimensional vector containing time, date, seven availability indicators, and the available feature value.
- The subsampled dataset exceeds 800,000 observations, representing 40% of the original dataset length.
E.2. Offset and significant weights in Electricity dataset
The Electricity analysis visualizes significance and offset activations for three input series, focusing on recent history within a 60-value input window. The accompanying appendix includes an AR(2) induction argument describing coefficients and error terms.
- Activation visualization: Figure 7 visualizes significance and offset activations for three Electricity input series across their past values.The figure shows the 25 most recent of 60 past values for three datapoints.
- Activation visualization: Each visualization row corresponds to activations for past values of one input feature.
- Activation visualization: The activation figure uses a log scale on its left graph.
B. Robustness of the proposed architecture
The robustness evaluation perturbs selected historical inputs with noise and tests trained models on unchanged-output observations. The supplementary experiments also construct synchronous and asynchronous multivariate series from noisy autoregressive signals and an asynchronously sampled electricity dataset.
- Robustness procedure: The robustness procedure adds noise of varying magnitude to selected past steps at one value dimension while assuming the output remains unchanged.Noise is applied to 20% of past steps.
- Robustness procedure: The evaluation samples 6,000 observations, split evenly between training and test sets, and evaluates each trained model separately on the perturbed data.
- Electricity data: The asynchronous Electricity dataset is created through periodic time-step subsampling and random feature sampling from the original seven-feature dataset.The resulting vectors contain time and date information, seven availability indicators, and the observed feature value.
E.2. Offset and significant weights in Electricity dataset
The Electricity analysis presents significance and offset activations for three input series, with rows corresponding to the past values of individual features. The accompanying figure covers recent values from a longer input history and uses a logarithmic scale on the left graph.
- Activation visualization: Figure 1 visualizes significance and offset activations for three input series from the Electricity-trained network.It presents the 25 most recent values out of 60 past input values for three datapoints.
- Activation visualization: Each row represents activations corresponding to past values of a single feature.
- Activation visualization: The figure’s left graph uses a log scale.