Source-linked AI summary
Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
Mathias Berglund, Tapani Raiko, Mikko Honkala, Leo Kärkkäinen, Akos Vetek, Juha Karhunen
TL;DR
Reconstructing gaps in high-dimensional time series is difficult because missing points depend on one another and inference must use context on both sides. This paper gives bidirectional RNNs two probabilistic interpretations, GSN and NADE, for that task. Both outperform one-way inference in the reported comparison, while Bayesian MCMC performs best and can be computationally infeasible for some data.
Problem
Filling gaps in high-dimensional binary time series requires inferring multiple dependent points from context before and after the gap.
Source: §4 · p. 4Method
The paper interprets bidirectional RNNs probabilistically through GSN and NADE to support efficient inference in both time directions.
Source: §6 · p. 8Results
GSN and NADE outperform one-way inference, while Bayesian MCMC seems to yield the best results; one-way inference performs very well on piano and JSB.
Source: §5.4 · pp. 6–7Takeaways & Limitations
The proposed methods can be used where Bayesian inference with a unidirectional RNN is computationally infeasible.
Source: §6 · p. 8For MIDI music, the model assumes independent Bernoulli outputs despite strong dependencies among output components.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Bidirectional recurrent neural networks (RNN) are trained to predict both in the positive and negative time directions simultaneously. They have not been used commonly in unsupervised tasks, because a probabilistic interpretation of the model has been difficult. Recently, two different frameworks, GSN and NADE, provide a connection between reconstruction and probabilistic modeling, which makes the interpretation possible. As far as we know, neither GSN or NADE have been studied in the context of time series before. As an example of an unsupervised task, we study the problem of filling in gaps in high-dimensional time series with complex dynamics. Although unidirectional RNNs have recently been trained successfully to model such time series, inference in the negative time direction is non-trivial. We propose two probabilistic interpretations of bidirectional RNNs that can be used to reconstruct missing gaps efficiently. Our experiments on text data show that both proposed methods are much more accurate than unidirectional reconstructions, although a bit less accurate than a computationally complex bidirectional Bayesian inference on the unidirectional RNN. We also provide results on music data for which the Bayesian inference is computationally infeasible, demonstrating the scalability of the proposed methods.
1 Introduction
RNNs have shown success in time-series modeling, but using them to reconstruct missing gaps raises both an inference challenge and questions about training and evaluation. The paper compares bidirectional generative approaches with unidirectional inference for binary time series, emphasizing scalability.
1 Introduction
- RNNs have been used for supervised recognition and unsupervised time-series learning, while prior sequential generation examples include bidirectional encoding or speech recognition.Bahdanau et al. [1] used a bidirectional RNN to encode a phrase but a unidirectional RNN to decode it; Maas et al. [18] used a deep bidirectional RNN in speech recognition.Source: §1 · p. 1
- Training to reconstruct longer gaps can push a model toward longer-term predictions, unlike the one-step prediction criterion commonly used in RNN training.Artificially missing values can also measure unsupervised-learning performance or serve as a training criterion.Source: §1 · pp. 1–2
- For unidirectional RNNs, inference for gaps larger than one becomes exponentially more expensive, and even sampling can scale exponentially with gap size.The challenge is smoothing: estimating missing values in the middle of a sequence using observations on both sides.Source: §1 · p. 2
- Explicitly training with missing data had not, to the authors’ knowledge, been thoroughly evaluated against other RNN inference strategies.The introduction identifies this comparison as an open evaluation question.Source: §1 · p. 2
- Bidirectional generative RNN methods are compared with unidirectional inference for reconstructing missing values in binary time series, with scalability a central consideration.The introduction frames reconstruction as useful for real missing data, unsupervised-learning evaluation, and training models to focus on longer-term predictions.Source: §1 · pp. 1–2
2 Recurrent Neural Networks
The section describes RNNs as sequence-to-sequence models and uses next-input prediction to apply them to unsupervised sequence modeling. It contrasts a unidirectional network’s past-dependent outputs with a bidirectional network’s use of past and future context.
2 Recurrent Neural Networks
- For unsupervised sequence modeling, the RNN predicts the next input by setting each output y_t to x_{t+1}.The general setup models P(Y|X), with input and output vectors at each time step.Source: §2 · p. 2
2.1 Unidirectional Recurrent Neural Networks
- A unidirectional RNN’s output depends on information propagated through its hidden layer from observations up to the current time step.The hidden-to-output, hidden-to-hidden, and input-to-hidden connections use separate weight matrices; output distributions may use softmax or independent Bernoulli variables with sigmoid functions.Source: §2.1 · p. 2
2.2 Bidirectional Recurrent Neural Networks
- A bidirectional RNN uses separate forward and backward hidden streams, so its predictions can use information from both past and future time steps.In the proposed formulation, y_t does not receive information from x_t, allowing the network to predict one time step given all the others.Source: §2.2 · p. 3
3 Probabilistic Interpretation for Unsupervised Modelling
The section turns sequence prediction into probabilistic modeling by contrasting a straightforward unidirectional factorization with two ways to define a joint distribution from bidirectional RNN predictions. GSN uses iterative resampling, while NADE constructs the sequence in a random order.
3 Probabilistic Interpretation for Unsupervised Modelling
- A unidirectional RNN defines the sequence joint distribution as the product of its successive predictions.Each prediction is conditioned on the preceding observations.Source: §3 · p. 3
- A bidirectional RNN predicts each output from all other time steps, so its predictions cannot simply be multiplied to obtain the joint distribution.The section introduces GSN and NADE as two solutions to this problem.Source: §3 · p. 3
- GSN defines the joint implicitly as the stationary distribution of a process that repeatedly resamples one time step from its conditional distribution given the others.This uses a denoising auto-encoder interpretation, with Gibbs sampling from a random initialization in practice.Source: §3 · p. 3
- NADE defines the time-series joint by drawing a random order of time indices and revealing data points one at a time from a fully missing sequence.The bidirectional RNN is trained with some inputs marked missing while all outputs are observed.Source: §3 · p. 3
4 Filling in gaps with Recurrent Neural Networks
The section frames gap filling as reconstructing consecutive missing points using context from both before and after the gap, then describes three RNN-based inference approaches.
4 Filling in gaps with Recurrent Neural Networks
- Gap filling must reconstruct consecutive, interdependent points in the middle of a sequence using both past and future context.One-way inference instead reconstructs points in order without using future context.Source: §4 · pp. 4–5
- The GSN approach trains a bidirectional RNN to estimate each point given all other points, then repeatedly samples gap values in random order.This approximates the stationary distribution; test-time complexity is O((dc + c^2)(T + gM)).Source: §4 · p. 4
- The NADE approach trains a bidirectional RNN with a separate missing-value token, then reconstructs each missing point once until the gap is filled.Its test-time complexity is O((dc + c^2)(T + g)), without repeatedly sampling each reconstructed point.Source: §4 · p. 4
- Bayesian MCMC uses a unidirectional RNN and Bayes’ theorem to evaluate gap-point probabilities by scoring outputs from t through T for each proposal.Its test-time complexity includes a factor a for the number of possible values; for binary vectors, a is 2^d, making evaluation infeasible as dimensionality grows.Source: §4 · p. 4
5 Experiments
Experiments compare bidirectional GSN and NADE gap reconstruction with unidirectional inference on Wikipedia text and polyphonic music. GSN and NADE outperform one-way inference overall, while Bayesian MCMC performs best but is computationally complex.
5 Experiments
- The experiments fill five-character gaps in Wikipedia text and five-time-step gaps in four polyphonic music data sets.Each music time step is an 88-dimensional binary vector representing pressed piano keys.Source: §5 · p. 5, §5.2 · p. 6
5.1 Training details for categorical prediction task
- Text BRNNs use a missing-value token and train on masked gaps, while the compared unidirectional and bidirectional networks use roughly similar parameter counts.Training uses 96-dimensional one-hot character vectors; the text experiments do not optimize hyperparameters or repeat runs for confidence intervals because training is time-consuming.Source: §5.1 · p. 5
5.2 Training Details for the Binary Structured Prediction Task
- Music experiments omit unidirectional RNNs because Bayesian MCMC is computationally prohibitive for 2^88 possible binary values.The music data sets contain at least seven hours of music each; models are trained for 50,000 updates, with the initial learning rate selected using validation cost.Source: §5.2 · p. 6
5.3 Evaluation of Models
- Evaluation uses five-step test gaps, with GSN and Bayesian MCMC sampled for 100 MCMC steps and NADE evaluated across all reconstruction orders.For the text task, GSN and NADE are tested on 50,000 gaps, while Bayesian MCMC is limited to 1,000; music evaluation uses all eligible five-step gaps.Source: §5.3 · p. 6
5.4 Results
- Bayesian MCMC yields the best NLL results, while GSN and NADE outperform one-way inference on five-step gaps.Table 1 reports negative log likelihood, for which lower is better.Source: §5.4 · p. 6, Table 1 · p. 7
- For most methods, gap-edge characters are easier to predict than middle characters; one-way inference performs well at the leftmost position but struggles later without future context.The position-wise NLL comparison covers Wikipedia and Piano data.Source: Figure 2 · p. 7, §5.4 · p. 7
- GSN performance quickly saturates on music data, whereas Wikipedia performance may improve with more than M = 100 MCMC steps.The authors assess performance as a function of GSN MCMC steps against NADE.Source: §5.4 · p. 7
- For gap size one, BRNNs outperform previously published unidirectional RNN results, serving as a check that the models trained successfully.These likelihood results are reported in supplementary Table 2.Source: §5.4 · p. 7
6 Conclusion and Discussion
Bidirectional RNNs receive two probabilistic interpretations, GSN and NADE, for time-series generation and efficient inference in both time directions. The discussion qualifies this approach with training and model assumptions, and identifies open choices for extending and comparing the methods.
6 Conclusion and Discussion
- GSN and NADE interpret bidirectional RNNs as generative time-series models and support efficient inference in both time directions, including when unidirectional Bayesian inference is infeasible.The paper frames this as addressing the difficulty of inferring missing values with recurrent generative models.Source: §6 · pp. 7–8
- NADE’s training differs from the basic BRNN in gap masking, training targets, and an added missingness input, and reconstruction-only training performs worse on single-value reconstruction.The authors say these training differences significantly affect the quality of the trained probabilistic model.Source: §6 · p. 8
- The RNN and BRNN comparison used equal parameter counts, leaving open which approach works best with optimally sized hidden layers.The RNN can focus learning on forward prediction and reuse those dependencies for computationally heavy backward Bayesian inference.Source: §6 · p. 8
- Future directions include exploring LSTM structures, gating the missingness-indicator connection in NADE, and using deeper architectures.Source: §6 · p. 8
- For structured MIDI prediction, the model assumes independent Bernoulli outputs despite strong dependencies among output components.Stochastic hidden units are suggested as one way to account for those dependencies; related work reconstructed motion-capture gaps using maximum-a-posteriori selection.Source: §6 · p. 8
- The proposed methods could extend to continuous-valued data, while recurrent time-series reconstruction has been shown effective in speech recognition under impulsive noise.Source: §6 · p. 8