PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperImpute: Generalized Iterative Imputation with Automatic Model Selection

Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, Mihaela van der Schaar

arXiv:2206.07769v1stat.MLcs.LG

TL;DR

Missing-data imputation requires methods that handle incomplete records without demanding unsuitable model specifications or stronger missingness assumptions. HyperImpute generalizes iterative imputation by automatically selecting flexible column-wise models and hyperparameters, and it consistently performs at or above benchmark levels across evaluated settings. The paper therefore presents a strong defense of well-configured iterative imputation, while restricting its formal attention to MCAR and MAR settings.

  • Problem

    Missing-data imputation must handle records with different variables missing, while existing iterative and deep generative approaches impose different specification, optimization, or data-assumption burdens.

  • Method

    HyperImpute automatically selects and configures column-wise models and hyperparameters within a generalized iterative imputation framework.

  • Results

    HyperImpute performs at or above all benchmarks on 10 of 12 datasets under MAR at 30% missingness for both RMSE and WD, and performs better across broader tested settings.

  • Takeaways & Limitations

    The findings support well-configured iterative imputation as a strong alternative to more complex methods and provide an accessible, extensible platform for imputation research.

  • Takeaways & Limitations

    The paper limits its attention to MCAR and MAR settings rather than MNAR, where missing-data distributions generally require additional assumptions for identification.

Abstract

from arXiv · show

Consider the problem of imputing missing values in a dataset. One the one hand, conventional approaches using iterative imputation benefit from the simplicity and customizability of learning conditional distributions directly, but suffer from the practical requirement for appropriate model specification of each and every variable. On the other hand, recent methods using deep generative modeling benefit from the capacity and efficiency of learning with neural network function approximators, but are often difficult to optimize and rely on stronger data assumptions. In this work, we study an approach that marries the advantages of both: We propose *HyperImpute*, a generalized iterative imputation framework for adaptively and automatically configuring column-wise models and their hyperparameters. Practically, we provide a concrete implementation with out-of-the-box learners, optimizers, simulators, and extensible interfaces. Empirically, we investigate this framework via comprehensive experiments and sensitivities on a variety of public datasets, and demonstrate its ability to generate accurate imputations relative to a strong suite of benchmarks. Contrary to recent work, we believe our findings constitute a strong defense of the iterative imputation paradigm.

1. Introduction

Missing-data imputation balances flexible iterative conditional models against powerful but assumption-heavy deep generative approaches. HyperImpute addresses this trade-off by automatically configuring column-wise models and hyperparameters, with experiments showing accurate imputations against strong benchmarks.

  • Conventional approaches: Iterative imputation estimates each feature’s conditional distribution from the others and cycles through univariate models until convergence.This provides customizable multivariate modeling through feature-wise conditional specifications.
  • Deep generative approaches: Deep generative imputers estimate a joint model using neural function approximators but face more challenging optimization and stronger practical constraints.The supplied discussion specifically highlights difficulties with GAN-based adversarial training.
  • Contributions: HyperImpute automatically configures column-wise models and hyperparameters within a generalized iterative imputation framework.The framework is designed to combine conditional specification with flexible function approximators.
  • Contributions: The proposed implementation includes out-of-the-box learners, optimizers, simulators, and extensible interfaces.These components are intended to facilitate accessibility and reproducibility in imputation research.
  • Contributions: The paper evaluates HyperImpute through comprehensive experiments and sensitivities, reporting accurate imputations relative to strong benchmarks.The contribution statement identifies empirical evaluation as the third contribution.

2. Background

The paper distinguishes iterative, generative, and traditional imputation methods by their modeling assumptions and missingness regimes. It focuses on MCAR and MAR settings, where missingness is ignorable, while MNAR generally requires additional domain-specific assumptions.

  • Missingness assumptions: The paper restricts its discussion to MCAR and MAR settings, where missingness depends only on observed components or is independent of the data.MNAR, by contrast, allows missingness to depend on the missing components themselves.
  • Missingness assumptions: In MCAR and MAR settings, inference can ignore the missingness mechanism, whereas MNAR distributions are generally not identifiable without additional assumptions.The additional requirements may include domain-specific constraints or parametric missingness models.
  • Related methods: Iterative methods specify a univariate model for each feature and cycle through regressions, with methods such as MissForest and MICE valid in the MAR setting.This is the paper’s principal discriminative approach in the related-work comparison.
  • Related methods: GAN-based implicit imputers are valid only under MCAR, while many explicit deep latent-variable models require fully observed training data or are likewise limited to MCAR.MIWAE is identified as an exception adapted for MAR, though its bound is tight only with infinite computational power.
  • Related methods: The paper also situates mean substitution, hot deck imputation, k-nearest neighbors, EM-based models, matrix completion, and optimal transport among traditional alternatives.These methods rely on differing modeling choices, including joint-model or low-rank assumptions.

3. HyperImpute

HyperImpute generalizes iterative imputation by automatically selecting models and hyperparameters separately for each column, combining conditional flexibility with automatic configuration. Its inside-out search preserves the iterative procedure while avoiding the higher complexity of global search.

  • 3.1. Problem Formulation: The framework assumes missingness is MCAR or MAR and represents incomplete data with masks that identify observed and missing features.The imputation problem is to reconstruct missing values while retaining observed values.
  • 3.2. Generalized Iterative Imputation: HyperImpute searches over models and hyperparameters for each column, extending iterative imputation beyond a fixed learner specification.This permits flexible approximators such as deep neural networks and modern boosting methods.
  • 3.3. Automatic Model Selection: Inside-Out Search embeds model search within the iterative loop, achieving O(KDA) complexity instead of top-down search complexity O(KD A^D).The method searches each regression during the O(KD) iterative procedure, rather than evaluating complete combinations globally.
  • 3.3. Automatic Model Selection: The inside-out strategy combines iterative imputation’s MAR validity and convergence properties with automatic selection among flexible function approximators.The search is performed inside the iterative procedure while the outer loop retains the usual iterative structure.
  • 3.4. The HyperImpute Algorithm: Algorithm 1 initializes imputations with a baseline method, then repeatedly visits columns, selects or reuses models, and refines each column’s missing values until stopping.Disabling model selection recovers conventional iterative imputation.

4. Practical Implementation

HyperImpute is implemented as an extensible, reproducible imputation system with configurable learners, optimizers, imputers, simulation, evaluation, and benchmarking components. Its sklearn-transformer interface supports integration into existing pipelines.

  • 4. Practical Implementation: The implementation provides extensible learner classes for column-wise classification and regression, including linear models, neural networks, bagging, and boosting methods.Users define the global model and hyperparameter configuration space searched for each variable.
  • 4. Practical Implementation: Optimizers implement model search over heterogeneous learner configurations, with Hyperband as the default and RMSE or negative AUROC as objectives.The optimizer interface supports configuration selection and evaluation methods such as Bayesian optimization and adaptive computation.
  • 4. Practical Implementation: Imputer modules provide baseline initialization and benchmark algorithms through a fit-transform interface.The implementation defaults to mean substitution for initialization and includes comprehensive recent benchmark methods.
  • 4. Practical Implementation: Simulation, evaluation, and comparison modules support seeded, systematic experiments across missingness mechanisms and performance metrics.These modules are designed to facilitate accessibility and reproducibility in imputation research.
  • 4. Practical Implementation: HyperImpute is implemented as an sklearn transformer, enabling direct integration with sklearn pipelines and existing downstream workflows.The transformer can serve as a component in a larger pipeline.

5. Empirical Investigation

HyperImpute is evaluated against strong imputation benchmarks through overall performance, sensitivity, source-of-gains, model-selection, and convergence analyses. Across these experiments, it consistently performs strongly, with adaptive components and rapid convergence contributing to its behavior.

  • 5.1. Overall Performance: HyperImpute outperforms all benchmarks on both RMSE and WD across 10 of 12 UCI datasets under MAR with 30% missingness.It outperforms all benchmarks on at least one metric on all 12 datasets.
  • 5.1. Overall Performance: HyperImpute’s advantage increases with sample size, is more noticeable above five features, and remains significant across missingness rates.Its low WD across missingness rates suggests reduced overfitting on sparser datasets.
  • 5.2. Source of Gains: All four design aspects improve RMSE: HyperImpute gains 18% over the best ICE models, 11% over restricted learners, 4% over global selection, and 7% and 5% over two non-adaptive variants.These comparisons are measured under MAR at a 0.3 missingness rate.
  • 5.3. Model Selections: Learner selection shifts with missingness and sample availability, with more expressive neural networks and boosting selected less often at higher missingness rates than random forests and linear regression.The analysis tallies learner-class selections across columns and datasets.
  • 5.3. Model Selections: Across iterations, HyperImpute selects different learner types as imputations improve, with boosting and neural networks more commonly selected in later rounds.The authors associate later-round selection with stabilized imputations and harder conditional imputations.
  • 5.4. Convergence: HyperImpute generally reaches a convergence plateau within four iterations, and its internal objective tracks ground-truth RMSE and WD improvements.Convergence is illustrated on the iris and wine_white datasets, with additional results in the appendix.

6. Conclusion

HyperImpute automatically and adaptively configures column-wise models from expressive function approximators, backed by a practical implementation and comprehensive benchmarks. Its results support iterative imputation as a strong approach for varied missing-data problems.

  • HyperImpute automatically and adaptively configures column-wise models from expressive function approximators.
  • The paper provides a practical implementation, comprehensive benchmarks, and integrated tools for accessibility and reproducibility.
  • The findings argue that well-configured conditional specification can achieve state-of-the-art performance.
  • HyperImpute serves as an investigative platform for studying characteristics and solutions across imputation problems.

A.1. Dataset Details

The evaluation uses reproducible missingness procedures, multiple missingness mechanisms and rates, and a broad column-wise model-search configuration. HyperImpute selects models through task-specific objectives under computationally constrained search and iterative termination criteria.

  • Experiments simulate MCAR, MAR, and MNAR missingness at 10%, 30%, 50%, and 70% rates.
  • The model-search space includes linear and logistic regression, random forests, XGBoost, CatBoost, and neural networks.
  • Column-wise model selection minimizes RMSE for continuous labels or maximizes AUROC for categorical labels through cross-validation.
  • Search strategies trade speed, exploration, and computational cost across default evaluation, model-specific Bayesian optimization, and adapted HyperBand.
  • The iterative procedure stops when iteration limits, imputation changes, or objective improvements meet specified termination criteria.

A.6. Data Types

HyperImpute handles categorical and continuous variables by assigning task-specific candidates and objectives. Across evaluated UCI datasets and missingness scenarios, it outperforms most benchmarks on RMSE and WD.

  • HyperImpute separates categorical and continuous tasks, candidate model classes, and loss functions during automated search.
  • The evaluation covers 12 UCI datasets, three missingness scenarios, and four missingness rates.
  • HyperImpute outperforms the majority of benchmarks on both RMSE and WD across MCAR, MAR, and MNAR scenarios and missingness rates.

B.2. Sensitivity Analysis

Sensitivity analysis varies observed sample size, feature count, and missingness rate across scenarios. HyperImpute performs especially strongly with more features and higher missingness, while its advantage is limited with few features.

  • HyperImpute outperforms all benchmarks when fewer samples are available, with the improvement becoming more significant as data sizes increase.
  • With higher feature counts, HyperImpute demonstrates consistently superior performance.
  • With low feature counts, HyperImpute does not demonstrate markedly better imputation performance.
  • HyperImpute achieves superior performance across missingness scenarios and rates, with a clearer advantage at higher missingness rates.

B.3. Source of Gains

HyperImpute’s gains arise from combining column-wise imputation, automatic model and hyperparameter tuning, adaptive selection across iterations, and a flexible base-imputer catalogue. Across missingness mechanisms and datasets, model choices vary with data conditions, while convergence is usually reached after four iterations.

  • Source of Gains: 18% average performance gain over the best ICE models demonstrates HyperImpute’s overall improvement in the source-of-gains study.The comparison uses RMSE across five UCI datasets under MAR at 0.3 missingness.
  • Source of Gains: 40% average WD improvement over the best ICE models and 27% over restricted base imputers show stronger distance-based gains.WD measures distance between imputed data and the underlying ground truth.
  • Model Selection: Boosting and neural networks are selected more often with low missingness and larger datasets, whereas linear and bagging models dominate lower-data regimes.HyperImpute also tends to choose more powerful algorithms in later iterations as it addresses more challenging imputations.
  • Convergence: Convergence is generally achieved after four iterations, although 3 of 12 MAR experiments remain nonconvergent after more than 10 iterations.The internal objective generally corresponds well to ground-truth imputation performance.

D. Running Time

HyperImpute is not the most time-intensive method in the reported comparisons, and its runtime scales roughly linearly with feature count. Model training dominates runtime, while model reuse can make selection competitive.

  • Running Time: HyperImpute is far from the most time-intensive method, with all algorithms converging in seconds to minutes.Selecting and reusing models from multiple learner classes can offset the cost of broader model selection.
  • Running Time: Runtime grows roughly as O(D) with feature count, consistent with model-training costs dominating execution.The feature-dimension experiment uses the largest dataset, libras, across multiple feature counts and missingness levels.
Loading 2206.07769v1…