Source-linked AI summary
Handling Incomplete Heterogeneous Data using VAEs
Alfredo Nazabal, Pablo M. Olmos, Zoubin Ghahramani, Isabel Valera
TL;DR
Existing VAEs do not directly handle heterogeneous and incomplete data, despite these settings being common in real-world databases. HI-VAE provides a VAE framework for such data and outperforms competitors on heterogeneous data completion while achieving comparable classification accuracy to deep supervised methods.
Problem
Existing VAEs assume complete data and typically target homogeneous observations, whereas real-world databases can contain mixed numerical and nominal attributes with random missingness.
Method
HI-VAE is a general VAE framework that incorporates incomplete data and heterogeneous observations using likelihood models suited to individual attribute types.
Results
HI-VAE outperforms competitors on heterogeneous data completion and provides comparable classification accuracy to deep supervised methods that cannot handle missing input values.
Takeaways & Limitations
The framework supports generative modeling of incomplete heterogeneous datasets for estimating missing values and making predictions on unseen data.
Takeaways & Limitations
Learning separate networks for attributes can increase computational cost and overfitting while losing amortized inference across attributes; factorization also loses shared attribute correlations.
Abstract
from arXiv · showhide
Variational autoencoders (VAEs), as well as other generative models, have been shown to be efficient and accurate for capturing the latent structure of vast amounts of complex high-dimensional data. However, existing VAEs can still not directly handle data that are heterogenous (mixed continuous and discrete) or incomplete (with missing data at random), which is indeed common in real-world applications. In this paper, we propose a general framework to design VAEs suitable for fitting incomplete heterogenous data. The proposed HI-VAE includes likelihood models for real-valued, positive real valued, interval, categorical, ordinal and count data, and allows accurate estimation (and potentially imputation) of missing data. Furthermore, HI-VAE presents competitive predictive performance in supervised tasks, outperforming supervised models when trained on incomplete data.
1. Introduction
The paper addresses the gap between deep generative models designed for structured homogeneous data and real-world datasets that are heterogeneous and incomplete. HI-VAE provides a general VAE framework for mixed data types and missing entries, supporting unsupervised completion and supervised prediction.
- Motivation: Real-world databases are often large, heterogeneous, noisy, and incomplete, motivating models that capture latent structure and estimate missing values.Such models may also support outlier detection and prediction on unseen data.
- Research gap: Deep generative models have largely focused on highly structured homogeneous data such as images, text, video, and speech.These datasets provide strong dependencies that can be exploited by convolutional or recurrent architectures.
- Research gap: Existing approaches provide limited support for incorporating missing data during training, often assuming complete data or missingness only in a target dimension.These assumptions are unsuitable when missing entries occur throughout an unsupervised dataset.
- Proposed framework: HI-VAE combines likelihood models for mixed numerical and nominal attributes with a recognition model that handles Missing Data Completely at Random.Its design also includes a normalization input/output layer to prevent a few dimensions from dominating training.
- Proposed framework: The ELBO is computed only on observed data, allowing the resulting fully unsupervised VAE to perform density estimation, missing-data completion, classification, and regression with incomplete inputs.The framework includes practical design guidelines compatible with modern VAE and GAN efforts.
- Results: HI-VAE outperforms competitors on heterogeneous data completion and achieves classification accuracy comparable to deep supervised methods that require imputing missing inputs.The comparison includes the GAIN method for data completion.
2. Problem statement
The paper studies datasets whose objects contain mixed numerical and nominal attributes, with missing values appearing across entries. These datasets can have weak inter-attribute correlations, requiring models that accommodate varied data types and random missingness without overfitting.
- Data setting: A heterogeneous dataset contains N objects with D attributes, where attributes are numerical or nominal variables.Each object is represented as a D-dimensional vector.
- Data setting: Numerical attributes may be real-valued, positive real-valued, or discrete counts, while nominal attributes may be categorical or ordinal.Categorical values are unordered; ordinal values belong to a finite ordered set.
- Missingness: Missing values appear ubiquitously across the mixed attributes in incomplete heterogeneous datasets.Figure 1(a) illustrates this setting using sliced vectors containing only observed elements.
- Modeling challenge: Unlike images or other highly structured data, these databases contain scalar mixed attributes whose correlations are often weak.The model must therefore remain flexible while avoiding overfitting on relatively low-dimensional observed data.
3. Generalizing VAEs for Heterogeneous and Incomplete Data
HI-VAE extends VAEs to incomplete heterogeneous data by separating observed-data likelihood contributions and making recognition depend only on observed attributes. Its factorized decoder and input-dropout encoder support marginalization and missing-data inference under MCAR.
- HI-VAE extends the vanilla VAE to handle incomplete and heterogeneous data.
- The decoder factorizes across attributes, making it easier to marginalize missing attributes from the ELBO.Each attribute’s likelihood parameters are produced from the latent representation by an attribute-specific DNN.
- The recognition model uses only observed attributes, with missing inputs replaced by zeros in an input-dropout DNN.The architecture ensures zeroed missing entries do not affect encoder outputs or parameter derivatives.
- The ELBO is computed on observed data, with reconstruction restricted to observed entries and a KL term regularizing the latent posterior.
- An alternative requiring one independent DNN per attribute can increase computation and overfitting while losing amortization across missing-data patterns.
- The framework supports estimating missing values from the incomplete-data VAE.
1. Real-valued data. For real-valued data, we assume a Gaussian likelihood model, i.e.,
HI-VAE accommodates heterogeneous attributes with distinct likelihood models and normalization strategies. Its hierarchical extension shares parameters across attributes while preserving attribute-specific observation models.
- HI-VAE supports real-valued, positive real-valued, count, categorical, and ordinal variables with appropriate likelihood models.The listed models include Gaussian, log-normal, Poisson, multinomial logit, and ordinal logit likelihoods.
- Ordinal likelihood thresholds are constrained to remain ordered using a cumulative sum of positive network outputs.
- Positive likelihood parameters are produced with the softplus function.The function used is f(x) = log(1 + exp(x)).
- Batch normalization handles numerical attributes with different value ranges, including normalization of logarithms for positive-valued data.Real-valued inputs are shifted and scaled, while positive-valued data are normalized after taking natural logarithms.
4. The Heterogeneous-Incomplete VAE (HI-VAE)
HI-VAE extends a VAE for incomplete heterogeneous data with a structured latent prior and hierarchical generative network. Its recognition model handles observed inputs, latent mixture assignments, and differentiable ELBO optimization.
- Latent prior: A Gaussian mixture prior structures z_n through a categorical component variable s_n, helping prevent the KL term from dominating the ELBO.The mixture is assumed uniform, with π_ℓ = 1/L.
- Hierarchical generative model: A shared DNN maps z_n to a homogeneous representation Y, while attribute-specific networks convert each y_nd and s_n into heterogeneous likelihood parameters.The shared network captures dependencies among attributes; independent bottom-level networks support different attribute likelihoods.
- Hierarchical generative model: The HI-VAE generative model supports the heterogeneous likelihood parameterizations described for the model.These parameterizations are summarized in Table 1, although the supplied table excerpt does not list them individually.
- Recognition network: The recognition network models q(s_n|x_n^o) categorically and constructs q(z_n|x_n^o,s_n) from their combined representation.The categorical posterior uses a DNN with softmax output, followed by concatenation with the observed-input representation to parameterize the Gaussian posterior.
- Recognition network: The recognition model assumes missing attributes are conditionally independent of observed attributes given the latent variables s_n and z_n.The ELBO uses the Gumbel-softmax reparameterization trick to draw differentiable samples from the variational distribution.
5. Experiments
The experiments evaluate HI-VAE for heterogeneous-data imputation and classification under missingness, including design choices, imputation strategies, and comparisons with existing methods.
- Imputation strategy: HI-VAE imputes missing values through MAP inference of latent variables, generative-model evaluation, and taking the mode of each attribute distribution.Sampling from the generative model is also considered as an alternative imputation strategy.
- Evaluation: Average imputation error is computed across attributes using variable-specific metrics, including NRMSE for numerical and accuracy error for categorical variables.Ordinal variables use displacement error, reflecting the data type when evaluating imputation.
- HI-VAE design: With 20% missing data, deeper networks and larger latent dimensions tend to improve performance, although substantial gains are observed mainly for the Letter database.For Letter, more complex architectures lead to lower imputation error.
- HI-VAE design: Normalization significantly improves imputation error for Adult, Spam, and Wine and prevents infinite ELBO gradients during inference for Default.The normalization layer is applied to numerical variables.
- Comparison with existing methods: HI-VAE consistently outperforms mean imputation across datasets, while achieving its strongest relative advantage on nominal variables, where it is best in four of six cases.For numeric variables, its error is comparable to other methods; GLFM performs best on the Breast and Wine datasets.
- Predictive task: HI-VAE is evaluated for classification by treating target labels as missing entries and training with 50% of labels removed, under input-missingness rates of 0%, 10%, and 50%.Supervised baselines receive mean or mode imputations because they cannot handle missing input data directly.
6. Conclusions
HI-VAE is designed to model heterogeneous, incomplete data by learning from observed values, handling missingness in recognition, and capturing dependencies across attributes. Empirically, it outperforms competitors for heterogeneous data completion and matches deep supervised methods in classification accuracy despite missing inputs.
- Incomplete-data inference: HI-VAE derives a data-marginal-likelihood lower bound that depends only on observed values, enabling training with ubiquitous missing data.It also proposes methods for handling missing values in the recognition network.
- Empirical results: The evaluation includes average imputation error across missing-data percentages under missing completely at random and classification error with 10% and 50% missing inputs.
- Heterogeneous dependencies: HI-VAE enforces correlations among heterogeneous attributes by using a joint DNN to construct each attribute’s output-distribution parameters.The model addresses the complex optimization landscape in which local optima may capture only subsets of attribute correlations.
- Heterogeneous dependencies: The hierarchical HI-VAE captures correlations among attributes through a latent space spanned by a Gaussian mixture.
- Empirical results: HI-VAE outperforms competitors on heterogeneous data completion and achieves comparable classification accuracy to deep supervised methods.Those supervised methods cannot handle missing input values and therefore require imputation.