Source-linked AI summary
GAIN: Missing Data Imputation using Generative Adversarial Nets
Jinsung Yoon, James Jordon, Mihaela van der Schaar
TL;DR
Missing data is widespread, and complete training data may be unavailable for imputation. GAIN adapts GANs so a generator imputes missing components while a hinted discriminator distinguishes observed from imputed components; experiments report significant improvements over state-of-the-art methods.
Problem
Missing data is pervasive, and some existing imputation approaches require complete data or use only observed components for representation learning.
Method
GAIN adapts GANs by training a generator to impute missing components and a discriminator to distinguish observed from imputed components using a hint about missingness.
Results
GAIN significantly outperforms state-of-the-art imputation methods and achieves the best post-imputation prediction accuracy, with 8.9% to 79.2% performance improvements in mean bias and mean square error.
Takeaways & Limitations
GAIN provides a generative imputation architecture that can operate when complete data is unavailable and can support imputation across real-world datasets.
Takeaways & Limitations
If the hint H is independent of the mask M, the optimization does not generally define a unique density.
Abstract
from arXiv · showhide
We propose a novel method for imputing missing data by adapting the well-known Generative Adversarial Nets (GAN) framework. Accordingly, we call our method Generative Adversarial Imputation Nets (GAIN). The generator (G) observes some components of a real data vector, imputes the missing components conditioned on what is actually observed, and outputs a completed vector. The discriminator (D) then takes a completed vector and attempts to determine which components were actually observed and which were imputed. To ensure that D forces G to learn the desired distribution, we provide D with some additional information in the form of a hint vector. The hint reveals to D partial information about the missingness of the original sample, which is used by D to focus its attention on the imputation quality of particular components. This hint ensures that G does in fact learn to generate according to the true data distribution. We tested our method on various datasets and found that GAIN significantly outperforms state-of-the-art imputation methods.
1. Introduction
Missing data arises across medical and other applications, motivating imputation methods that infer unobserved values from observed information. GAIN adapts GANs to imputation without requiring complete training data, using adversarial discrimination and hints.
- Motivation: Missing data can result from uncollected, lost, difficult, dangerous, or intentionally omitted measurements.Medical examples include unmeasured respiratory rate and unavailable biopsy information.
- Motivation: Imputation algorithms estimate missing values from observed measurements and support applications including medical analysis, image concealment, compression, and counterfactual estimation.The paper describes imputation as useful beyond medical data.
- Missingness mechanisms: Missingness is classified as MCAR, MAR, or MNAR according to whether missingness depends on no variables, observed variables, or observed and unobserved variables.The paper notes that MAR is formally defined in the supplementary materials.
- Related work: Existing imputation methods include discriminative approaches such as MICE, MissForest, and matrix completion, alongside generative methods based on expectation maximization and deep learning.The related methods span classical and neural approaches.
- Related work: Some prior approaches require complete data for discriminator training or use only observed components to learn data representations.These constraints are identified for deep convolutional GANs and denoising autoencoders.
- Contribution: GAIN generalizes GANs for imputation when complete data is unavailable, with a generator imputing missing components and a discriminator distinguishing observed from imputed components.The networks are trained adversarially, while hints provide partial missingness information to guide the discriminator toward the desired distribution.
2. Problem Formulation
The paper represents partially observed vectors with a data-dependent mask and defines imputation as sampling the conditional distribution of complete data given each partial observation. This formulation supports multiple imputations that retain uncertainty.
- 2. Problem Formulation: X is a d-dimensional random data vector, while M is a binary mask indicating which components are observed.The formulation allows continuous or binary data variables.
- 2. Problem Formulation: Each component space is extended with a special symbol ∗ to represent an unobserved value, producing the partially observed variable ˜X.The construction forms a product space for partially observed vectors.
- 2. Problem Formulation: The mask M can be recovered from the partially observed vector ˜X.Observedness is encoded by whether a component equals the unobserved-value symbol.
- 2. Problem Formulation: The notation uses lower-case letters for realizations and 1 for an all-ones vector whose dimension is context-dependent.The dimension is most often d.
- 2.1. Imputation: The imputation dataset contains n independent copies of ˜X paired with their recovered mask realizations.The dataset is written as D = {(˜x_i, m_i)}^n_i=1.
- 2.1. Imputation: The goal is to generate samples from P(X|˜X = ˜x_i) to fill each sample’s missing values.The target is the conditional distribution of complete data given the observed partial vector.
- 2.1. Imputation: Modeling the conditional distribution rather than only its expectation enables multiple imputations that capture uncertainty in missing values.Multiple draws can be made for each partially observed sample.
3. Generative Adversarial Imputation Nets
GAIN adapts GANs to partially observed data by having a generator produce imputations and a discriminator identify observed versus imputed components. A hint mechanism supplies missingness information so the adversarial objective targets the desired distribution.
- Architecture: GAIN is motivated by GANs and models P(X|˜X = ˜x_i) for imputation.The architecture is depicted in Figure 1.
- Generator: The generator takes ˜X, M, and masked noise (1−M)⊙Z as inputs and outputs an imputation vector ¯X.The generator is defined as a function from the partially observed space, binary masks, and noise to X.
- Generator: The completed vector ˆX replaces each ∗ in ˜X with the corresponding generator output, while G also outputs values for observed components.Thus, G produces a value for every component before completion.
- Generator: Masked noise matches the dimension of the conditional target distribution, whose dimension equals the number of missing components.This differs from passing an unrestricted noise vector as in a standard GAN.
- Discriminator: The discriminator predicts the mask componentwise, distinguishing observed components from imputed components rather than classifying the entire vector as real or fake.Each output estimates the probability that the corresponding completed-vector component was observed.
- Hint: The hint mechanism gives the discriminator additional information about M through H and changes it into a function of the completed vector and hint.The i-th discriminator output is conditioned on both ˆX and H.
- Hint: Without enough information about M in H, multiple generator distributions can be optimal, so hinting is needed for identification.The paper specifically notes that omitting hinting can leave several distributions equally optimal.
- Objective: The discriminator maximizes correct mask prediction while the generator minimizes the probability that the discriminator predicts M.GAIN defines this adversarial training through a minimax objective and associated loss function.
4. Theoretical Analysis
The theoretical analysis characterizes the discriminator and shows that an appropriately informative hint makes the generator’s target distribution unique and equal to the data distribution under MCAR. Without sufficient hint information, this guarantee can fail.
- Setup: Under the MCAR assumption, the analysis decomposes completed-data density into observed-data, generator-imputed, and hint components.The generator determines the conditional density of imputed components, while the hint distribution is chosen by design.
- Discriminator analysis: The optimal discriminator component is the posterior probability that the corresponding completed-data component was observed.For each component, D*(x,h)_i is defined by the observed-component joint density divided by the sum of observed and imputed-component joint densities.
- Hint necessity: If the hint lacks enough information about the mask, the generator’s optimal density need not be unique.In particular, when H is independent of M, the criterion can admit multiple solutions.
- Hint mechanism: The proposed hint reveals all but one mask component to the discriminator while leaving the unrevealed component ambiguous.Known hint values imply the corresponding mask values, whereas 0.5 provides no direct information about that component.
- Guarantee: With this hint, the theoretical solution is unique, invariant across mask vectors, and yields completed data with the same distribution as the original data.The result relies on the stated MCAR assumption that M is independent of X.
5. GAIN Algorithm
GAIN alternates discriminator and generator optimization: the discriminator predicts which components are observed, while the generator fools it on missing components and reconstructs observed ones. The hint and the two generator losses are central to the algorithm’s reported gains.
- Optimization: GAIN alternates discriminator and generator updates using mini-batches and stochastic gradient descent.The discriminator is optimized with fixed G, followed by generator optimization with the updated D.
- Discriminator optimization: The discriminator is trained only on outputs whose hint entries do not already determine the corresponding mask entries.Training determined outputs would provide no information about G and could cause D to overfit the hint vector.
- Generator optimization: The generator minimizes an adversarial loss for missing components and a reconstruction loss for observed components.The adversarial term rewards making imputed values difficult to identify, while the reconstruction term keeps outputs for observed features close to their actual values.
6. Experiments
Experiments across real-world datasets and varied settings show that GAIN improves imputation and downstream prediction performance, while remaining robust to missingness, sample size, and feature dimension changes.
- 6.1. Source of gain: 15% improvement over the simple auto-encoder was achieved by full GAIN, while the hint vector additionally provided 10% improvement.The comparison isolates contributions from the GAN-like architecture and hint mechanism.
- 6.2. Quantitative analysis of GAIN: GAIN was evaluated on five UCI datasets against five state-of-the-art imputation methods using RMSE.The benchmarks were MICE, MissForest, Matrix completion, Auto-encoder, and Expectation-maximization.
- 6.2. Quantitative analysis of GAIN: GAIN significantly outperformed every benchmark in imputation performance.
- 6.3. GAIN in different settings: GAIN consistently outperformed MissForest and Auto-encoder across the entire range of missing rates.All algorithms degraded as missing rates increased.
- 6.3. GAIN in different settings: As the number of samples increased, GAIN’s performance improvements over the benchmarks also increased, although it still outperformed them with relatively few samples.GAIN was also robust to feature dimensions, whereas MissForest had difficulty when dimensions were small.
- 6.4. Prediction Performance: GAIN achieved the best post-imputation prediction accuracy, but large imputation gains did not always produce significant prediction gains.Prediction comparisons used AUROC and a common logistic-regression model, excluding the multiclass Letter dataset.
- 6.5. Congeniality of GAIN: GAIN achieved 8.9% to 79.2% performance improvements in mean bias and mean square error for feature-label congeniality.Lower values indicate that an imputation method better preserves feature-label relationships.
7. Conclusion
The paper presents GAIN as a generative imputation model that extends GANs to missing-data settings. Experiments on real-world datasets show significant improvements over state-of-the-art imputation techniques, while future work targets additional applications.
- GAIN generalizes GANs into a generative model that can handle the characteristics of missing-data imputation.
- Experiments on real-world datasets showed that GAIN significantly outperformed state-of-the-art imputation techniques.
- Future work will investigate GAIN for recommender systems, error concealment, and active sensing.Preliminary MNIST error-concealment results are reported in the Supplementary Materials.