Source-linked AI summary
Variational Recurrent Auto-Encoders
Otto Fabius, Joost R. van Amersfoort
TL;DR
The paper addresses efficient unsupervised learning and representation of time-series data by combining RNNs with SGVB in a generative VRAE. The model maps sequences to latent vectors, supports generation from latent samples, and can initialize supervised RNN training; its current approach captures dependencies over around 100 time steps.
Problem
Efficiently learning useful latent representations and generative models for time-series data, while leveraging unlabeled data to support RNN training, remains the paper’s central motivation.
Method
VRAE combines an RNN encoder and decoder with SGVB, using a variational posterior over latent variables and recurrent processing of sequence inputs.
Results
The trained model maps sequences to latent vectors, generates data from latent-space samples, and provides weights and network states for initializing standard RNNs.
Takeaways & Limitations
VRAE provides a generative latent-vector representation for time sequences and a way to use unlabeled data when initializing supervised RNN training.
Takeaways & Limitations
The current approach captures time dependencies over around 100 time steps, and the paper identifies input reversal as a possible improvement.
Abstract
from arXiv · showhide
In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data, mapping the time series data to a latent vector representation. The model is generative, such that data can be generated from samples of the latent space. An important contribution of this work is that the model can make use of unlabeled data in order to facilitate supervised training of RNNs by initialising the weights and network state.
1 INTRODUCTION
The paper introduces VRAE, an RNN-based variational auto-encoder that maps time sequences to distributions over latent representations. It supports efficient large-scale unsupervised learning and can initialize standard RNN training.
- Model motivation: RNNs are suited to temporal data because their dynamic temporal behaviour captures time dependencies.The paper cites applications in handwriting recognition and music modelling.
- Model motivation: VRAE combines recurrent neural networks with Variational Bayes to encode time-series data into a distribution over latent variables.Its encoder maps data to a latent representation, while its decoder maps latent representations back to data.
- Contributions: VRAE enables efficient, large-scale unsupervised variational learning by mapping time sequences to latent representations.This provides a compact latent vector representation for sequential data.
- Contributions: A trained VRAE provides weights and a network state for initializing a standard RNN.The paper states this initialization will likely improve training efficiency and possibly avoid exploding gradients or enable better scores.
2 METHODS
The method uses SGVB to optimize a variational lower bound, then implements the VRAE with recurrent encoder and decoder components. Reparameterization enables gradient-based training through the latent variables.
- SGVB: SGVB approximates the intractable posterior p(z|x) with q(z|x) and optimizes a lower bound on the log-likelihood.This avoids computationally expensive sampling-based methods.
- SGVB: The log-likelihood decomposes into a variational lower bound and a KL divergence between the true posterior and its approximation.The model parameters are denoted by θ.
- SGVB: Because KL divergence is non-negative, the variational objective is a lower bound on the log-likelihood.The lower bound is the quantity optimized during training.
- SGVB: The reparameterization trick expresses z sampled from q(z|x) as the deterministic transformation z = µ + σϵ, with ϵ ∼ N(0, 1).This makes gradient computation through the latent variable possible for the decoder.
- VRAE architecture: The encoder recurrently updates its state using successive inputs, and derives the latent distribution from the final state hend.The initial encoder state h0 is initialized as a zero vector.
- VRAE architecture: After sampling z, the decoder’s initial recurrent state is computed from z and then updated as in a traditional RNN.The decoding RNN uses one set of weights to compute its initial state.
3 EXPERIMENTS
The experiments train VRAE models on segmented video-game songs and examine both training bounds and latent-space organization. Two- and twenty-dimensional models support clustering, transition modeling, and generation from latent vectors.
- Dataset: The dataset contains 8 MIDI songs, sampled at 20Hz, with 49 retained pitch dimensions and 520 parts per song.Songs were divided into short parts, with equal numbers of data points retained from each song.
- Training: Adam and gradually reduced learning rates were important for learning a useful representation with the two-dimensional VRAE.The model used 500 hidden units, an initial learning rate of 1 · 10^-3, and a final rate of 5 · 10^-6.
- Latent organization: Two-dimensional latent representations placed parts of some songs in distinct regions, although short parts could not capture every song characteristic.Figure 1 visualizes each encoded datapoint at the two-dimensional mean of its encoding; Mario Underworld, Mario, and Mariokart were the most distinct.
- Latent organization: The twenty-dimensional model used overlapping 40-step sequences to learn transitions between parts and support generating music of arbitrary length.The model used overlap so each sequence began halfway through the previous one.
- Latent organization: Figure 2 projects twenty-dimensional latent means into two dimensions with t-SNE, showing that parts of each song occupy only part of the space and some songs form clear groups.The apparent grouping depends on song homogeneity, inter-song similarity, and spatial information lost during t-SNE.
- Generation: The decoder can generate data from encoded datapoints or randomly sampled latent vectors, producing short interpolations and song medleys.The first model was trained on non-overlapping 50-step sequences, so longer generated sequences were not expected to follow the training distribution.
4 DISCUSSION
The discussion presents VRAE as an effective SGVB-trained RNN model that maps time sequences to one latent vector. It identifies limits in captured temporal dependence and several extensions and applications.
- Discussion: VRAE maps time sequences to one latent vector rather than to latent state sequences.
- Limitations: The current approach captures temporal dependencies of around 100 time steps.
- Future work: Proposed improvements include denser segmentation, reversing input order, and incorporating LSTM to train on longer sequences.
- Applications: Potential applications include recognition, denoising, feature extraction, and combination with other sequential models for tasks such as music genre tagging.