PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective

Apratim Bhattacharyya, Bernt Schiele, Mario Fritz

arXiv:1806.07772v2cs.LGcs.CVstat.ML

TL;DR

Multi-modal future sequences challenge point-estimate predictors, even though uncertainty matters for real-world prediction. The paper introduces a “Best of Many” objective for Gaussian latent-variable sequence models; it reports consistently outperforming baselines and state of the art across three diverse tasks.

  • Problem

    Point-estimate predictors do not capture the multi-modal future sequences that arise in partially observed or stochastic settings.

    Source: §1 · p. 1

  • Method

    The paper introduces a “Best of Many” sample objective for Gaussian latent-variable models of structured sequence prediction.

    Source: §1 · p. 1

  • Results

    The models consistently outperform baselines and state of the art on MNIST strokes, Stanford Drone, and HKO weather data.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The objective yields more accurate and diverse samples and a better match between training and test latent-variable distributions in the reported tasks.

    Source: §1 · p. 1, §5 · p. 9

    A hybrid objective showed modest improvement in some cases but no significant improvement over standard CVAE on the structured sequence prediction tasks studied.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

For autonomous agents to successfully operate in the real world, anticipation of future events and states of their environment is a key competence. This problem has been formalized as a sequence extrapolation problem, where a number of observations are used to predict the sequence into the future. Real-world scenarios demand a model of uncertainty of such predictions, as predictions become increasingly uncertain -- in particular on long time horizons. While impressive results have been shown on point estimates, scenarios that induce multi-modal distributions over future sequences remain challenging. Our work addresses these challenges in a Gaussian Latent Variable model for sequence prediction. Our core contribution is a "Best of Many" sample objective that leads to more accurate and more diverse predictions that better capture the true variations in real-world sequence data. Beyond our analysis of improved model fit, our models also empirically outperform prior work on three diverse tasks ranging from traffic scenes to weather data.

1. Introduction

Sequence prediction uses past observations to anticipate plausible futures, but point estimates cannot represent the multi-modal outcomes common in partially observed or stochastic settings. The paper proposes a “Best of Many” objective for Gaussian latent-variable models and reports more accurate, diverse predictions across three tasks.

1. Introduction

  • Point-estimate sequence models can produce unrealistic predictions when partially observed or stochastic futures are multi-modal.For example, a pedestrian may turn left or right after crossing, with no information about their intention.Source: §1 · p. 1
  • The proposed “Best of Many” sample objective addresses limitations in CVAE objectives and latent-variable distribution matching to improve sample accuracy and diversity.The paper also analyzes the objective and reports improved latent-variable fit compared with prior approaches.Source: §1 · p. 1
  • The models consistently outperform baselines and state of the art on MNIST strokes, Stanford Drone, and HKO weather data.The authors report improved accuracy and diversity of generated samples across all three tasks.Source: §1 · pp. 1–2

2. Related Work

Related work covers stochastic and Gaussian latent-variable approaches to structured prediction, conditional generative models, and recurrent sequence models. The paper positions its objective as a way to encourage diverse samples while improving the match between training and test latent-variable distributions.

Variational Autoencoders

  • Gaussian latent variables enable stochastic optimization for structured prediction, while CVAEs extend VAEs to conditional structured outputs; the training and test latent distributions can still mismatch.IWAE uses multiple importance-weighted samples for richer latent representations, but does not model conditional structured-output distributions [3].Source: §2 · p. 2

Structured Output Prediction

  • Gaussian hidden variables support stochastic optimization through the re-parameterization trick, unlike binary-latent SFNNs described as hard to train on large datasets.Source: §2 · p. 2

Related multi-sample objective

  • A prior multi-sample conditional objective draws from the prior, whereas this work uses importance sampling with a jointly learned proposal to address high-variance updates [8].Source: §2 · p. 2

Recurrent Neural Networks

  • Earlier recurrent methods include point-estimate pedestrian prediction and recurrent CVAE trajectory sampling [17]; this work targets the sampling module with a multi-sample objective.The paper also extends convolutional RNN sequence prediction to condition on Gaussian latent variables.Source: §2 · p. 2

3. Structured Sequence Prediction with Gaussian Latent Variables

The section develops conditional sequence prediction with Gaussian latent variables and a Best-of-Many-Samples objective, then describes LSTM and Conv-LSTM architectures for trajectory and image sequences.

3. Structured Sequence Prediction with Gaussian Latent Variables

  • The model conditions on a Gaussian latent variable to map an input sequence to multiple plausible outputs and represent modes of the future-sequence distribution.The section assumes the latent variable is independent of the input, with a standard normal prior.Source: §3 · p. 3

3.1. Conditional Variational Auto-encoder Based Training Objective

  • The CVAE training objective uses recognition-network samples to estimate a variational lower bound after the data log-likelihood integral proves computationally intractable.The authors describe direct Monte Carlo estimation as unbiased but high variance, motivating importance sampling from the recognition network.Source: §3.1 · p. 3
  • Equal weighting requires every sample to assign high probability to the data, constraining the recognition network to trade off likelihood fit against the training-test latent-distribution gap.A hybrid objective showed no significant improvement over standard CVAE on the authors’ structured sequence prediction tasks.Source: §3.1 · p. 3

3.2. Best of Many Samples Objective

  • The objective derives a lower bound by separating integral terms, then approximates a difficult minimum with a KL divergence over the full distribution.The KL penalty discourages the recognition network from assigning high probability where the prior probability is low.Source: §3.2 · pp. 3–4
  • The Best-of-Many-Samples objective gives the recognition network multiple chances to draw a high-posterior sample, encouraging diversity while loosening its constraints.The authors approximate the many-sample objective with a best-sample objective to avoid numerical stability issues in regression tasks.Source: §3.2 · p. 4

3.3. Model Architectures for Structured Sequence Prediction

  • For trajectories, an encoder LSTM summarizes the input and a decoder LSTM predicts each step conditioned on that summary and a latent-variable sample.A CNN-derived visual summary can additionally condition trajectory samples on scene appearance.Source: §3.3 · pp. 4–5
  • Figure 4 shows diverse LSTM-BMS samples clustered by k-means, with the cluster count manually set to the expected digits inferred from the initial stroke.Source: Figure 4 · p. 5
  • For image sequences, stacked Conv-LSTMs use spatially structured latent tensors so neighboring states model local correlations while one latent sample maintains temporal consistency.Spatially neighboring decoder states are conditioned on neighboring latent subtensors, supporting spatially consistent outputs.Source: §3.3 · pp. 5–6

4. Experiments

The experiments evaluate sequence prediction on MNIST strokes, Stanford Drone trajectories, and HKO radar sequences. Across these tasks, the reported results favor the BMS objective on fit or sample accuracy, and describe generated samples as diverse.

4. Experiments

  • Experiments use MNIST stroke completion, Stanford Drone trajectories, and HKO radar sequences, with training sample counts T = 10, 10, and 5, respectively.Models were trained with ADAM; batch sizes were 32 for trajectory data and 4 for radar data.Source: §4 · p. 6

4.1. MNIST Sequence

  • On MNIST strokes, LSTM-BMS achieves the best CLL, lower training-time KL divergence than CVAE, and more accurate, diverse, multimodal samples.The KL comparison indicates reduced mismatch between recognition-network latents at training and the prior distribution used at testing.Source: §4.1 · pp. 6–7, Figure 6 · p. 6

4.2. Stanford Drone

  • On Stanford Drone, LSTM-BMS performs best on both CLL and Euclidean distance, while qualitative examples are described as plausible, diverse, and scene-consistent.The Euclidean distance is measured at 1/5 resolution; the table caption identifies the evaluation as Stanford Drone.Source: Table 2 · p. 7, §4.2 · p. 8

4.3. Radar Echo

  • On HKO, Conv-LSTM-CVAE and Conv-LSTM-BMS both outperform, and BMS has superior CLL with accurate, diverse samples.The comparison uses precipitation nowcasting metrics and CLL; the caption identifies Table 3 as evaluating HKO radar image sequences.Source: §4.3 · pp. 8–9, Table 3 · p. 8

5. Conclusion

The paper presents a best-of-many sample objective for Gaussian latent variable models and reports improved latent matching, accuracy, and performance across three sequence-prediction datasets.

  • The proposed best-of-many sample objective is designed for Gaussian latent variable models and conditional models on multimodal distributions.Source: §5 · p. 9
  • The learned latent representation better matches training and testing distributions, which the authors say leads to more accurate samples.Source: §5 · p. 9
  • Across MNIST strokes, Stanford Drone, and HKO weather data, the approach consistently outperforms baselines and state of the art.Source: §5 · p. 9

A. Additional Details of our “Best of Many” Sample Objective

The supplement derives the “Best of Many” objective by applying the First Mean Value Theorem of Integration [4] to the data log-likelihood. It also explains the many-sample approximation and the model-layer details.

A. Additional Details of our “Best of Many” Sample Objective

  • The First Mean Value Theorem of Integration [4] is applied to derive the main-paper likelihood bound from an integral over a bounded interval.The derivation sets f1(z) = p(z|x)/qφ(z|x,y) and f2(z) = pθ(y|z,x)×qφ(z|x,y).Source: A. Additional Details of our “Best of… · p. 10
  • The lower bound’s unknown maximizing point is approximated by maximizing log(f1) over N points in the interval.The passage notes that the minimum is difficult to estimate and uses the theorem’s interior point to motivate this approximation.Source: A. Additional Details of our “Best of… · p. 10
  • The recognition network uses the re-parameterization trick [14] to sample z and predicts the Gaussian mean and variance used to estimate the KL divergence.The mean and variance are denoted µ and σ in the passage.Source: A. Additional Details of our “Best of… · p. 11
  • Approximating the data log-likelihood yields the “Best of Many” objective, while Monte Carlo estimation yields the “Many Sample” objective.The second integral is identified as the KL divergence between qφ(z|x,y) and p(z|x).Source: A. Additional Details of our “Best of… · pp. 10–11

B.1. Model for Structured Trajectory Prediction

The structured trajectory prediction model and its recognition network are specified in Tables 4 and 5. The text defines the layer terminology used in those specifications.

B.1. Model for Structured Trajectory Prediction

  • Table 4 gives the structured trajectory prediction model, and Table 5 gives its training recognition network qφ.The passage refers to fully connected layers as Dense and defines Size as the number of neurons in a layer.Source: §B.1 · p. 11, Table 4 · p. 11, Table 5 · p. 11

B.2. Extension with Visual Input

The visual-input extension conditions the trajectory decoder on a CNN encoder’s output. Tables 6 and 7 describe the encoder and extended model, and the recognition network remains the one described for the prior model.

B.2. Extension with Visual Input

  • The extended trajectory model additionally conditions LSTMdec on the CNN encoder output and uses the previously described recognition network.Table 6 details the encoder, Table 7 the extended model, and Table 8 its recognition network.Source: §B.2 · p. 11, Table 6 · p. 11, Table 7 · p. 11

B.3. Model for Structured Image Sequence Prediction

The structured image sequence model uses Convolutional LSTMs and Convolutional Embedding layers, with its model and recognition-network details given in Tables 9 and 10.

B.3. Model for Structured Image Sequence Prediction

  • The image sequence model uses Convolutional LSTMs and Convolutional Embedding layers, unlike the structured trajectory model.Table 9 gives model details and Table 10 gives recognition-network details; the table terminology includes CLSTM, Conv, MaxPool, and UpSample.Source: §B.3 · p. 11, Table 9 · p. 12, Table 10 · p. 12

C. Additional Results

Figure 9 presents additional HKO samples at t + 5, t + 10, and t + 15, illustrating sample closeness to ground truth and diversity across predictions.

C. Additional Results

  • At t + 5, t + 10, and t + 15, the HKO examples show samples close to ground truth and diverse predictions.The text assesses closeness by comparing the Best sample with ground truth, and diversity by comparing the mean of all samples with the Best sample.Source: C. Additional Results · p. 12
  • For each example, the model generates T = 50 samples and displays the Best sample, the mean of all samples, and per-pixel variance.Source: C. Additional Results · p. 12
  • Figure 9 reports statistics of samples generated by the LSTM-BMS model on the HKO dataset.Source: Figure 9 · p. 13
Loading 1806.07772v2…