PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Autoencoder with Arbitrary Conditioning

Oleg Ivanov, Michael Figurnov, Dmitry Vetrov

arXiv:1806.02382v3stat.MLcs.LG

TL;DR

The paper addresses learning conditional distributions for arbitrary observed-feature subsets, so remaining real-valued or categorical features can be sampled in one shot. It proposes VAEAC, a VAE-like latent-variable model trained by stochastic variational Bayes. Experiments find effective feature imputation and diverse, realistic image inpainting, with performance competitive with state-of-the-art methods.

  • Problem

    Existing approaches address joint or conditionally specified generation, while this paper seeks one model for conditional distributions given any arbitrary subset of observed features.

  • Method

    VAEAC conditions the prior over latent Gaussian variables on observed features and uses stochastic gradient variational Bayes to generate unobserved features.

  • Results

    Experiments show effective conditional sampling, feature-imputation performance competitive with current state-of-the-art methods, and diverse realistic image inpaintings.

  • Takeaways & Limitations

    VAEAC supports single-shot sampling of diverse inpaintings and applies one arbitrary-conditioning model to feature imputation and image inpainting.

  • Takeaways & Limitations

    For image inpainting, training-mask distributions need to be similar to test-mask distributions, although the paper describes this limitation as not very strict in practice.

Abstract

from arXiv · show

We propose a single neural probabilistic model based on variational autoencoder that can be conditioned on an arbitrary subset of observed features and then sample the remaining features in "one shot". The features may be both real-valued and categorical. Training of the model is performed by stochastic variational Bayes. The experimental evaluation on synthetic data, as well as feature imputation and image inpainting problems, shows the effectiveness of the proposed approach and diversity of the generated samples.

1 INTRODUCTION

The paper targets learning conditional distributions for arbitrary subsets of features, proposing VAEAC to sample unobserved features from observed ones. It evaluates the model on feature imputation and image inpainting, finding effective conditional sampling and diverse, realistic outputs.

  • Problem and model: VAEAC learns conditional distributions p(x_I|x_U\I) for arbitrary subsets of features, generalizing joint and ordinary conditional distribution learning.Conditioning features affect the prior over latent Gaussian variables used to generate unobserved features.
  • Problem and model: The model is a latent-variable neural probabilistic model trained with stochastic gradient variational Bayes.Its design is similar to a variational autoencoder while supporting arbitrary conditioning subsets.
  • Applications: Feature imputation restores missing features from observed ones and can improve downstream machine-learning models.The paper identifies imputation as an application where imputed values may be useful directly or for subsequent algorithms.
  • Applications: Image inpainting fills unobserved image regions with artificial content for object removal or completion of partially closed or corrupted objects.The application concerns generating realistic content in missing image regions.
  • Evaluation: VAEAC successfully samples conditional distributions, with sample distributions close to true conditionals, and produces diverse, realistic inpaintings.The evaluation covers synthetic data, UCI feature imputation, and MNIST, Omniglot, and CelebA image inpainting.

2 RELATED WORK

Related work includes marginal, autoregressive, Markov-chain, decision-making, and neural inpainting methods. VAEAC is distinguished by diverse single-shot inpainting with low test-time computational cost.

  • Missing-feature methods: Universal Marginalizer approximates marginals of unobserved features conditioned on observable values using a feed-forward neural network.Autoregressive models of joint probability are also identified as related work.
  • Missing-feature methods: Other missing-data methods use noise-filled Markov chains or LSTM-based sequential decision making, but are computationally expensive at test time.These approaches approximate conditional distributions through learned transitions or sequential decisions.
  • Image inpainting: Neural image-inpainting methods combine adversarial, reconstruction, texture, or semantic parsing losses to generate completed images.The cited methods include general image inpainting and face-specific approaches.
  • Image inpainting: Unlike methods targeting a single realistic inpainting, VAEAC can sample diverse inpaintings.This distinction is presented as a central difference from the described alternatives.
  • Image inpainting: VAEAC is a single-shot inpainting method with low computational cost, whereas several competing methods require test-time optimization.The optimization requirement is associated with Yeh et al. (2016), Yang et al. (2017), and Yeh et al. (2017).

3 BACKGROUND

VAEs model data through latent variables and optimize a variational lower bound because posterior inference is generally intractable. Conditional VAEs extend this framework by conditioning the latent prior and generated distribution on observed variables.

  • Variational autoencoder: A VAE samples a latent variable z from a standard Gaussian prior and generates data x from pθ(x|z), inducing pθ(x).The generative distribution is modeled by a neural network with parameters θ.
  • Variational autoencoder: VAE parameters are trained by maximizing training-data likelihood, while posterior intractability motivates optimizing a variational lower bound.The lower bound can be optimized with backpropagation and stochastic gradient descent.
  • Variational autoencoder: The proposal distribution qφ(z|x) approximates the posterior, and reparameterization expresses z as µφ(x) + εσφ(x) for gradient estimation.The resulting gradient uses Monte Carlo estimation for one term and analytic computation for the other.
  • Conditional variational autoencoder: Conditional VAEs approximate p_d(x|y) and can sample diverse, realistic objects when multiple x values are probable for a given y.This avoids the blurry average that a deterministic mean-squared-error regression model may predict for multimodal targets.
  • Conditional variational autoencoder: CVAE variational objectives condition the distributions on y, including a neural-network prior pψ(z|y), and therefore use three trainable networks instead of VAE’s two.The objective is optimized using the reparameterization trick.

4 VARIATIONAL AUTOENCODER WITH ARBITRARY CONDITIONING

VAEAC models conditional distributions over arbitrary subsets of unobserved features given the observed features and their mask. It uses a latent-variable generative process, variational optimization, and extensions for mixed data and missing training features.

  • Problem formulation: The model targets pψ,θ(x_b|x_1−b, b) for arbitrary unobserved-feature masks, covering joint and conditional distribution learning as special cases.The mask distribution p(b) lets users specify which conditioning patterns receive emphasis.
  • Generative model: VAEAC samples latent z from a conditioning-dependent Gaussian prior, then generates unobserved features from pθ(x_b|z, x_1−b, b).The latent prior depends on observed features and the mask, while the generative network supports real-valued and categorical features.
  • Generative model: The implementation masks observed inputs into fixed-length vectors and computes likelihood only over unobserved output components.This accommodates architectures such as multilayer perceptrons and convolutional neural networks despite mask-dependent feature lengths.
  • Optimization: Training maximizes a variational lower bound combining expected reconstruction log-likelihood with a KL divergence between proposal and prior distributions.A fully factorized Gaussian proposal enables reparameterization and analytic KL computation.
  • Optimization: A Normal-Gamma prior on the latent-prior parameters is added to prevent their divergence and potential numerical instability during optimization.The added distribution is chosen to have little effect near zero while constraining excessively large prior parameters.
  • Missing training features: For training data with missing values, the mask distribution is conditioned on x, missing features are omitted from reconstruction loss, and an additional missingness mask is supplied to the proposal network.These modifications distinguish genuinely observed object features from values missing in the training data.

5 EXPERIMENTS

Experiments evaluate VAEAC for missing-feature imputation and image inpainting, including comparisons with established imputation and inpainting methods. Results indicate competitive imputation quality and diverse, realistic inpaintings, while highlighting limitations in training-mask assumptions and adversarial-loss design.

  • 5.1 MISSING FEATURES IMPUTATION: VAEAC imputations are evaluated on UCI datasets after randomly removing 50% of values from training and test sets.The study compares VAEAC with MICE, MissForest, and GAIN.
  • 5.1 MISSING FEATURES IMPUTATION: Multiple imputation uses n = 10 sampled completions per incomplete object, increasing the dataset size tenfold.VAEAC, MICE, and GAIN support this procedure, whereas MissForest does not.
  • 5.1 MISSING FEATURES IMPUTATION: VAEAC imputations are competitive in NRMSE, PFC, post-imputation regression R2-score, and classification accuracy, although MICE or MissForest outperform it on some datasets.NRMSE evaluates continuous datasets and PFC evaluates categorical datasets; post-imputation performance is reported using R2-score or accuracy.
  • 5.2 IMAGE INPAINTING: VAEAC image inpainting omits adversarial losses used for sharper and more realistic samples, leaving that adaptation for future work.The current experiments nevertheless report diverse and realistic inpaintings.
  • 5.2 IMAGE INPAINTING: Image-inpainting experiments visualize VAEAC samples on binarized MNIST, Omniglot, and CelebA under several mask settings.The evaluation compares PSNR across masks with Context Encoder and other generative inpainting models.
  • 5.2 IMAGE INPAINTING: For most masks, one VAEAC sample has higher PSNR than competing methods; for highly diverse cases, the best of 10 samples has higher PSNR.The authors note that PSNR does not fully reflect visual quality and may favor blurry VAE samples over realistic GAN samples.

6 CONCLUSION

The paper presents a scalable probabilistic model for learning all conditional distributions of a vector and applies it to feature imputation and image inpainting. Experiments find VAEAC competitive with state-of-the-art methods in both tasks.

  • VAEAC simultaneously learns all conditional distributions for a vector using a neural probabilistic model with Gaussian latent variables.The model is described as scalable and efficient for inference and learning.
  • The model is successfully applied to feature imputation and image inpainting.
  • Experimental results show competitive performance with state-of-the-art methods for missing-feature imputation and image inpainting.

A.1 NEURAL NETWORK ARCHITECTURES

The implementation combines convolutional residual architectures, skip-connections, mask-based conditioning, and task-specific preprocessing for imputation and inpainting experiments. Training uses repeated data splits, randomly dropped features, and validation-based model selection.

  • Architecture: Image-inpainting networks use convolutional neural networks based on ResNet blocks and skip-connections between the prior and generative networks.The main architecture is illustrated in Figure 5.
  • Architecture: Skip-connections route local information around the latent variables while the latent variables retain global information, improving log-likelihood and image realism.This makes the boundary between the observed image and inpainting less conspicuous.
  • Architecture: The architecture can use different hidden-layer widths and structures, including simpler one-hidden-layer fully connected networks that still produce good MNIST inpaintings.
  • Training and evaluation: Experiments split data 3:1, randomly drop 50% of values, repeat runs five times, and average results with standard deviations.
  • Training and evaluation: For multiple imputation, each incomplete object is replaced by n = 10 sampled imputations and predictions are averaged for regression or combined by mode for classification.The imputed target is excluded from downstream models for fair feature-imputation comparison.

A.4 GAIN IMPLEMENTATION DETAILS

The GAIN implementation uses one-hot categorical encoding, modified reconstruction losses, validation-based selection, and a tuned hint distribution. The paper reports that these modifications improve or preserve imputation quality relative to the original implementation.

  • Feature representation and losses: Categorical features use one-hot encoding, which performs better than treating categories as continuous values and rounding them.The comparison is reported in terms of NRMSE and PFC.
  • Feature representation and losses: The categorical reconstruction loss gives equal contribution to each categorical feature by averaging cross-entropy over its one-hot components.
  • Feature representation and losses: Binary and categorical features use a mixture of L2 and cross-entropy losses with weights 0.8 and 0.2, respectively.The paper reports better NRMSE and PFC than with the original model.
  • GAIN modifications: The modified GAIN configuration uses α = 10 and a Bernoulli-based hint with p = 0.01, outperforming the original model across tested α values in NRMSE and PFC.The authors note that this hint distribution is theoretically inconsistent but works well empirically.
  • GAIN modifications: The authors report that their GAIN modifications are consistently no worse or better than original GAIN on the considered datasets.For missing-feature imputation, the paper reports results for the modified GAIN implementation.

B.3 UNIVERSAL MARGINALIZER: TRAINING PROCEDURE MODIFICATION

The training procedure must match the distribution of masks encountered in use, because mask distributions induced by training and testing requests can differ. The section also reviews CVAE-derived training objectives and their limitations for multimodal targets.

  • Mask distribution: The mask distribution p(b) induces a request distribution ˆp(b) that can differ from it and also depends on the permutation of unobserved components.
  • Mask distribution: Training with fixed-size masks can omit masks with intermediate numbers of unobserved components, causing drastically low test likelihood and unrealistic samples.
  • Mask distribution: For randomly ordered unobserved components, the procedure samples b0 from p(b), samples u uniformly, and forms b by elementwise multiplication with Bernoulli(u) variables.
  • Training objectives: The CVAE training review distinguishes Monte-Carlo and importance-sampling estimators, with the latter also called IWAE and generally requiring fewer samples for comparable accuracy.Both estimators underestimate log-likelihood at small sample counts, with stronger underestimation for Monte Carlo.
  • Training objectives: GSNN and hybrid models have drawbacks when the target distribution p(x|y) has multiple local maxima.

C.1 THEORETICAL STUDY

The theoretical study explains why VAEAC can represent multimodal conditional distributions, whereas GSNN and hybrid variants tend toward averaged, blurry distributions when modes differ.

  • C.1 THEORETICAL STUDY: At α = 0, optimization sets all generator parameters equal to the same likelihood-maximizing value, while α = 1 optimizes the initial model likelihood.This formalizes the transition between multimodal modeling and averaging behavior.
  • C.1 THEORETICAL STUDY: GSNN can produce blurry samples because its latent variable is independent of the mode being reconstructed, forcing each generator output to suit all modes.For Bernoulli and Gaussian distributions, the resulting parameters equal an average across modes.
  • C.1 THEORETICAL STUDY: When a conditional distribution is multimodal, VAEAC can fit all modes while GSNN learns their average; for unimodal distributions, the two models are equal.GSNN may learn faster in the unimodal case because it has fewer parameters.
  • C.1 THEORETICAL STUDY: The hybrid model trades off VAEAC and GSNN behavior: decreasing α makes its learned distribution more blurry and closer to the average.The dependence on α can be derived analytically for simple distributions or evaluated experimentally.

C.2 SYNTHETIC DATA

Synthetic and downstream experiments compare VAEAC with GSNN, classical imputers, and neural baselines across distribution learning, inpainting, and feature imputation. VAEAC preserves multimodality and diverse inpaintings, while its imputation advantage is dataset-dependent.

  • C.2 SYNTHETIC DATA: VAEAC learns complex multimodal synthetic distributions, whereas GSNN and hybrid models lose local modes even when GSNN has a small weight.The experiments visualize joint and conditional distributions under different mixture coefficients.
  • C.2 SYNTHETIC DATA: VAEAC produces more diverse and realistic MNIST inpaintings than GSNN, whose outputs are smooth and blurry.The comparison is illustrated with sampled inpaintings and ground-truth images.
  • C.2 SYNTHETIC DATA: VAEAC learns the inpainting distribution better than GSNN in test log-likelihood, although GSNN can score better under small-sample Monte Carlo estimates.The latter pattern is interpreted as reflecting fewer local modes and greater blurriness in GSNN samples.
  • C.2 SYNTHETIC DATA: VAEAC has similar MNIST convergence speed to VAE in iterations, but each iteration is about 1.5 times slower because it uses three networks instead of two.The same convergence behavior was observed for other datasets in the experiments.
  • C.2 SYNTHETIC DATA: On some datasets, MICE and MissForest outperform VAEAC, GSNN, and NN because random-forest structure is more natural than neural-network structure there.Across the reported imputation metrics, the three neural methods otherwise show similar performance.

D.4 GAIN FOR IMAGE INPAINTING

The GAIN comparison examines image inpainting under a horizontal-line mask and contrasts it with VAEAC and Universal Marginalizer results. GAIN fails for this mask distribution, while its applicability depends on the training mask distribution.

  • D.4 GAIN FOR IMAGE INPAINTING: Under the horizontal-line MNIST mask, GAIN fails to learn the conditional distribution, whereas VAEAC uses unobserved pixels during training.The comparison uses released GAIN code with a different mask generator and VAEAC samples for the same setting.
  • D.4 GAIN FOR IMAGE INPAINTING: GAIN is not unsuitable for all inpainting settings: it can learn conditional distributions when masks are pixel-wise independent Bernoulli variables with probability 0.5.The limitation is therefore tied to the mask distribution examined in the horizontal-line experiment.
  • D.4 GAIN FOR IMAGE INPAINTING: For Gaussian-parameterized marginals, Universal Marginalizer cannot learn a simple distribution that VAEAC can represent.The paper presents this comparison as an example of a limitation of UM’s marginal parameterization.
  • D.4 GAIN FOR IMAGE INPAINTING: The cited UM result reports a log-likelihood of -5.2 for the compared distribution.The passage identifies this value as the UM result in the comparison.
Loading 1806.02382v3…