PaperlayerView exact revision on arXiv

Source-linked AI summary

EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE

Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato, Sebastian Nowozin, Cheng Zhang

arXiv:1809.11142v4cs.LGstat.ML

TL;DR

EDDI addresses costly information acquisition when only part of the data is observed and the next measurement must be chosen dynamically. It combines Partial VAE inference with an information-theoretic acquisition function, showing improvements across benchmark and health-care applications.

  • Problem

    Information acquisition requires inference from variable subsets of observed data and selecting valuable next measurements, while exact acquisition-value computation is intractable.

  • Method

    EDDI combines a scalable Partial VAE for missing-data inference with an information-theoretic acquisition function for selecting unobserved variables.

  • Results

    EDDI shows clear improvements in two real-world health-care applications and demonstrates its behavior across six UCI datasets against multiple baselines.

  • Takeaways & Limitations

    The framework supports dynamic active variable selection, while Partial VAE can also serve as a computationally efficient nonlinear imputation method.

  • Takeaways & Limitations

    The acquisition calculation is intractable for high-dimensional variables because entropy estimation can be difficult, and it uses a Partial VAE approximation.

Abstract

from arXiv · show

Many real-life decision-making situations allow further relevant information to be acquired at a specific cost, for example, in assessing the health status of a patient we may decide to take additional measurements such as diagnostic tests or imaging scans before making a final assessment. Acquiring more relevant information enables better decision making, but may be costly. How can we trade off the desire to make good decisions by acquiring further information with the cost of performing that acquisition? To this end, we propose a principled framework, named EDDI (Efficient Dynamic Discovery of high-value Information), based on the theory of Bayesian experimental design. In EDDI, we propose a novel partial variational autoencoder (Partial VAE) to predict missing data entries problematically given any subset of the observed ones, and combine it with an acquisition function that maximizes expected information gain on a set of target variables. We show cost reduction at the same decision quality and improved decision quality at the same cost in multiple machine learning benchmarks and two real-world health-care applications.

1 Introduction

EDDI addresses costly, dynamic information acquisition by combining scalable partial-data inference with task-aware variable selection. It introduces Partial VAE and an information-theoretic acquisition function, then evaluates the framework across benchmarks and healthcare applications.

  • Motivation: Dynamic information acquisition must balance decision quality against the cost of obtaining additional questions, measurements, or tests.The framework targets settings where information is acquired sequentially for a task.
  • Technical challenges: Missing data and variable-sized observed subsets require probabilistic inference that can reason from any currently available answers.The acquisition problem also requires choosing which unobserved variable to query next, whose exact value is generally intractable to compute.
  • Framework: EDDI is a scalable framework that dynamically selects the next information item for a given task while accounting for acquisition cost.The framework supports information in forms including questionnaire answers and laboratory results.
  • Technical contributions: Partial VAE extends variational autoencoders to partially observed data, providing scalable probabilistic inference and missing-data imputation.The method uses partial observations as the basis for inference and can also be used independently.
  • Technical contributions: An information-theoretic acquisition function selects unobserved variables by their task-relevant contribution and uses a novel efficient approximation.The selection criterion is based on mutual information for tasks such as surveys and health assessments.
  • Evaluation: EDDI is evaluated on multiple machine-learning settings and two healthcare applications, where the paper reports clear improvements in both healthcare applications.The healthcare evaluations cover intensive-care risk assessment and public-health assessment using a national survey.

2 Related Work

The related work spans missing-data modeling, amortized inference, traditional and variable-wise active learning, and cost-sensitive active feature acquisition. EDDI is positioned as combining scalable partial-observation inference with instance-specific variable selection.

  • Missing-data modeling: Missing-data research includes probabilistic imputation methods, while traditional approaches often rely on linear models and do not scale to large datasets.The related work describes matrix factorization and other probabilistic frameworks for missing-value imputation.
  • Amortized inference: Amortized inference improves the scalability of deep generative latent-variable models, but partially observed inputs remain a central challenge.Existing approaches include training with fully observed data and zero imputation for missing entries.
  • Active learning: Traditional active learning selects data points to label, whereas EDDI performs variable-wise active learning for each instance.This distinction requires new information-theoretic acquisition functions and non-trivial approximations.
  • Active feature acquisition: Active Feature Acquisition addresses sequential feature selection, particularly in applications where obtaining features has a cost.The cited work includes objectives for selecting features across instances.

3 Method

EDDI combines Partial VAE inference for arbitrary partial observations with Bayesian-experimental-design acquisition of task-relevant variables. It uses permutation-invariant encoding, Partial VAE sampling, and an approximation to an otherwise intractable information reward.

  • Problem formulation: The active variable selection objective is to query as few unobserved variables as possible while accurately predicting a quantity of interest.The reward can incorporate both prediction quality and acquisition cost.
  • Partial VAE: Partial VAE extends VAE amortized inference to arbitrary subsets of observed variables and variable-sized inputs.It approximates p(z|xO), after which unobserved variables can be inferred through the latent representation.
  • Partial VAE: The PNP specification constructs each observed-variable representation as ed ∗ xd, while PN uses concatenation [ed,xd].The paper reports that the multiplication formulation generalizes the zero-imputation VAE.
  • Partial VAE: Permutation-invariant aggregation produces a fixed-size representation from observed-variable elements, allowing the encoder to handle observations in any order and of arbitrary length.The representation is transformed into statistics of a multivariate Gaussian approximation to the latent posterior.
  • Acquisition: EDDI selects the next unobserved variable by maximizing an information-theoretic reward for target variables within an adaptive Bayesian experimental-design formulation.The framework repeatedly acquires the selected variable until a stopping criterion such as a time budget is reached.
  • Acquisition: The Partial VAE approximates predictive sampling by sampling z from q(z|xo) and then sampling xi from p(xi|z).The same procedure is used for joint samples involving the queried variable and target variables.
  • Efficient approximation: The information reward is difficult to compute because the relevant conditional distributions are intractable, and entropy estimation becomes difficult for high-dimensional target variables.The paper derives an approximation and notes that analytic KL solutions are available for specific variational families.
  • Efficient approximation: The approximation uses a Partial VAE posterior in place of p(z|xφ,xi,xo), while Equation (8) is described as exact.This substitution is explicitly presented as an approximation in the acquisition computation.

4 Experiments

The experiments evaluate Partial VAE for MNIST image inpainting and EDDI for active variable selection across UCI datasets and healthcare applications. Results show strong imputation, efficient personalized acquisition, improved AUIC rankings, and substantially lower computational cost than DRAL.

  • Partial VAE image inpainting: PNP-based Partial VAE performs best among Partial VAE methods for randomly missing MNIST pixels and contiguous-region inpainting.For contiguous regions, PN and PNP recover multiple possible digit modes while remaining consistent with observed pixels.
  • Comparison with DRAL: EDDI is 103 times more computationally efficient than DRAL for active variable selection per test point.The reported comparison measures test CPU time in seconds per test point.
  • Comparison with DRAL: EDDI significantly outperforms DRAL on Boston Housing test RMSE at every acquisition step.The comparison uses ten randomly selected test points; the paper attributes the advantage to EDDI’s more flexible Partial VAE model.
  • UCI active variable selection: EDDI achieves the same test RMSE with less than half as many selected variables as random selection on three UCI datasets.EDDI selects variables separately for each data instance, whereas single optimal ordering is less efficient despite improving over random ordering.
  • UCI active variable selection: EDDI outperforms other variable-selection orders across all Partial VAE settings on six UCI datasets, with PNP- and PN-based settings ranking above ZI-based settings.Average ranking is based on area under the information curve, where smaller AUIC indicates better performance.
  • Healthcare applications: In healthcare evaluations, EDDI significantly outperforms other variable-selection strategies on MIMIC III risk assessment, while PNP performs best among Partial VAE settings.The experiments also apply grouped active selection to NHANES public-health assessment, with AUIC weighted by selected group size.

5 Conclusion

The paper presents EDDI for dynamic active variable selection, using Partial VAE amortized inference to handle missing data and a variable-wise acquisition function. It reports effectiveness across multiple real-world applications while identifying extensions for missingness, time-series, and cold-start settings.

  • EDDI performs dynamic active variable selection for each instance.
  • Partial VAE provides amortized inference for missing data and can independently serve as a nonlinear probabilistic imputation method.
  • The framework includes a variable-wise acquisition function and an approximation method.
  • EDDI is effective on active variable selection tasks across multiple real-world applications.
  • Future extensions target missing-not-at-random data, time series, and cold-start situations.

A Additional Derivations

The derivation formulates EDDI’s information reward using partial VAE distributions and approximates it with shared samples and tractable divergence calculations. The supplied implementation details describe sampling and model-training procedures, alongside a qualitative comparison of generated images.

  • Information reward: The experimental design problem maximizes an information reward conditioned on observed variables using a partial VAE model.
  • Information reward: KL-divergence chain-rule decompositions rewrite the reward into terms that can be approximated with partial VAE inference.
  • Information reward: The approximate objective rewards latent-belief shifts while penalizing information not absorbed by the target variables.
  • Information reward: Shared samples and exact KL calculations for common encoder parameterizations improve computational efficiency.
  • Implementation: Partial VAE models are trained with random missingness, Adam optimization, batch size 100, and 3K iterations.
  • Implementation: Figure 8 compares random images generated by naive zero imputing, masked zero imputing, PN, and PNP.

B.2 UCI datasets

The UCI experiments apply EDDI to six datasets with target variables defined as variables of interest. Models use normalized data, repeated test splits, Partial VAE architectures, and sampling-based evaluation during active learning.

  • Datasets: EDDI is evaluated on six UCI datasets, with each dataset’s target variables serving as the variables of interest.
  • Preprocessing: All data are normalized and scaled between 0 and 1, with 10% held out for testing in each of 10 repetitions.
  • Model details: The Partial VAE models use 10-dimensional diagonal Gaussian latent variables and shared encoder-decoder architecture sizes.
  • Training: Training uses Adam, batch size 100, random missingness, and 3K iterations.
  • Evaluation: During active learning, 50 samples estimate expectations for information curves and target-variable predictive likelihoods.

B.2.2 STATISTICAL SIGNIFCANT TEST RESULTS

The statistical analysis tests whether EDDI-PNP-Partial VAE’s AUIC performance differs from other settings across six UCI datasets. All 15 reported comparisons are significant at the standard α = 0.05 cutoff.

  • Statistical tests: Wilcoxon signed-rank tests compare EDDI-PNP-Partial VAE with 15 other settings using RMSE-based AUIC.
  • Results: All 15 comparisons are statistically significant under the standard α = 0.05 cutoff.
  • Results: Table 6 reports p-values for EDDI-PNP versus 11 other settings across six UCI datasets.

B.2.3 ADDITIONAL PLOTS OF PN, ZI AND ZI-M ON UCI DATASETS

Additional RMSE information curves compare active variable-selection strategies across Boston Housing, Energy, and Wine using PointNet, Zero Imputing, and masked Zero Imputing.

  • Figure 9 compares RMSE during active variable selection across three UCI datasets and three partial-observation approaches.The approaches are PointNet, Zero Imputing, and Zero Imputing with mask; lower RMSE is better.

B.2.4 NEGATIVE TEST LOG LIKELIHOOD PLOTS OF PN, ZI AND ZI-M ON UCI DATASETS

Additional negative test log-likelihood curves compare active variable-selection strategies across Boston Housing, Energy, and Wine using three partial-observation approaches.

  • Figure 10 displays negative test log likelihood during active variable selection for Boston Housing, Energy, and Wine.It compares random selection, EDDI, and single best ordering under PointNet, Zero Imputing, and masked Zero Imputing; lower values are better.

B.2.5 COMPARISONS BETWEEN EDDI AND LASSO-BASED METHOD

This section compares EDDI with LASSO and presents related experimental and model details across UCI and health-care settings. The comparison favors EDDI while highlighting that the methods target different selection settings.

  • EDDI versus LASSO: EDDI significantly outperforms LASSO in all reported Partial VAE model comparisons.The comparison uses Wilcoxon signed-rank tests across the UCI results summarized in Table 7.
  • EDDI versus LASSO: EDDI performs sequential variable selection from partially observed data, whereas LASSO selects a global feature set using fully observed data.The authors describe the latter setting as often unrealistic for practical applications.
  • UCI comparisons: Figure 11 reports test RMSE information curves for EDDI and single best ordering across Boston Housing, Energy, and Wine.The figure uses PNP-Partial VAE and compares EDDI with the single best ordering baseline.
  • Health-care applications: The MIMIC-III target variable is the binary mortality indicator, with other data normalized and categorical variables converted to real-valued representations.The active-learning experiments use sampled estimates for the information-reward expectation and target losses.
  • Health-care applications: Figure 14 reports Bernoulli negative test likelihood curves for active variable selection on the MIMIC-III risk-assessment task.Results are shown for Zero Imputing, PointNet, and masked Zero Imputing.
  • Model details: The model family includes zero imputation, masked zero imputation, PointNet, and related inference-network parameterizations for partially observed inputs.Zero imputation sets unobserved values to zero, while PointNet uses permutation-invariant feature aggregation; the masked variant distinguishes missingness from observed zeros.

C.2 Approximation Difficulty of the Acquisition Function

The acquisition function is difficult to approximate because its information terms are intractable and common variational approximations can have the wrong objective direction. The paper relates its reward to conditional mutual information under a correct-model assumption.

  • Traditional variational approximations can produce an upper bound when the acquisition objective requires maximizing a lower quantity.The paper identifies this direction mismatch as a justification problem for applying standard variational approximation here.
  • Direct entropy estimation is problematic for high-dimensional target variables.This is listed as one limitation of common approaches to approximating the information reward.
  • Reversing the information reward and applying an ELBO also yields an upper-bound approximation in the wrong direction.The paper states that this reversed objective does not make mathematical sense for the intended maximization.
  • Ranganath’s entropy bound likewise gives an upper bound of the objective, which is the wrong direction.The paper lists this as another unsuccessful approximation strategy.
  • The common alternatives additionally require latent-space samples, introducing a second level of approximation.This adds computational and uncertainty costs to the approximation procedure.
  • Assuming q = p, the paper connects its reward formulation to the conditional mutual information used in BALD.The connection is derived through the relationship between entropy and KL-divergence reduction.
Loading 1809.11142v4…