Source-linked AI summary
MissForest - nonparametric missing value imputation for mixed-type data
Daniel J. Stekhoven, Peter Bühlmann
TL;DR
Missing-value imputation is difficult for high-dimensional data containing both continuous and categorical variables, because many methods handle only one type or treat types separately. missForest uses iterative random-forest imputation and out-of-bag error estimates, outperforming established methods across diverse biological and medical datasets, especially with complex interactions or nonlinear relations.
Problem
Existing imputation methods often handle only continuous or categorical variables, despite high-dimensional biological and medical data combining both types.
Method
MissForest iteratively trains random forests on observed values to impute missing entries across mixed-type data without distributional assumptions or tuning parameters.
Results
MissForest outperformed or matched compared methods across datasets, with imputation-error reductions of up to 50% and OOB estimates deviating from true error by no more than 10–15% on average.
Takeaways & Limitations
MissForest is particularly useful for mixed-type, high-dimensional data with complex interactions or nonlinear relations, while providing an internal imputation-quality estimate.
Takeaways & Limitations
MICE results on the Children’s Hospital data require caution because ill-distributed and nearly dependent variables made the implementation sensitive to missingness patterns.
Abstract
from arXiv · showhide
Modern data acquisition based on high-throughput technology is often facing the problem of missing data. Algorithms commonly used in the analysis of such large-scale data often depend on a complete set. Missing value imputation offers a solution to this problem. However, the majority of available imputation methods are restricted to one type of variable only: continuous or categorical. For mixed-type data the different types are usually handled separately. Therefore, these methods ignore possible relations between variable types. We propose a nonparametric method which can cope with different types of variables simultaneously. We compare several state of the art methods for the imputation of missing values. We propose and evaluate an iterative imputation method (missForest) based on a random forest. By averaging over many unpruned classification or regression trees random forest intrinsically constitutes a multiple imputation scheme. Using the built-in out-of-bag error estimates of random forest we are able to estimate the imputation error without the need of a test set. Evaluation is performed on multiple data sets coming from a diverse selection of biological fields with artificially introduced missing values ranging from 10% to 30%. We show that missForest can successfully handle missing values, particularly in data sets including different types of variables. In our comparative study missForest outperforms other methods of imputation especially in data settings where complex interactions and nonlinear relations are suspected. The out-of-bag imputation error estimates of missForest prove to be adequate in all settings. Additionally, missForest exhibits attractive computational efficiency and can cope with high-dimensional data.
1 Introduction
The introduction motivates missForest as a nonparametric, iterative random-forest method for imputing mixed-type data with complex interactions and nonlinear relations while making few structural assumptions. It reports competitive or superior performance, computational efficiency, and accurate out-of-bag error estimates without tuning parameters.
- Motivation: Missing-value imputation is crucial because many analyses require fully observed data, which are uncommon in contemporary medical and biological research.These fields increasingly produce high-dimensional multivariate data, sometimes with more variables than observations.
- Motivation: Medical and biological data may combine variable types and contain complex interactions and nonlinear relation structures that parametric procedures struggle to capture.Variables can range from technical mass-spectrometer settings to diagnostic expert opinions.
- Method: MissForest uses random forest in an iterative imputation scheme to handle any input-data type while allowing interactive and nonlinear regression effects with few structural assumptions.It trains a random forest on observed values and then predicts missing values.
- Results: Up to 50% lower imputation error is reported in some cases, while out-of-bag error estimates deviate proportionally by no more than 10 - 15% on average.The method is typically effective within a few iterations, requires no tuning parameter, and is computationally attractive.
2 Approach
missForest imputes missing entries by iteratively fitting random forests to observed values and predicting missing values, while ordering variables by increasing missingness. It stops when successive imputations worsen for both variable types and evaluates accuracy with type-specific error measures and out-of-bag estimates.
- Iterative imputation: For a variable Xs, the method separates observed and missing response values and uses other variables’ observed and missing entries as predictors for direct random-forest prediction.This approach avoids the built-in proximity-based missing-value routine, which requires a complete response.
- Iterative imputation: missForest initializes missing values, orders columns by increasing missingness, and iteratively fits random forests to observed responses before predicting each variable’s missing entries.The procedure repeats until its stopping criterion is met and returns the imputed matrix.
- Stopping criterion: The stopping criterion is met when the difference between the newly and previously imputed matrices increases for the first time for both continuous and categorical variables, when present.Continuous-variable differences use N, while categorical-variable differences use F and the number of missing categorical entries.
- Performance assessment: Imputation performance uses NRMSE for continuous variables and PFC for categorical variables, with values near 0 indicating good performance and values around 1 indicating bad performance.NRMSE is computed over continuous missing values, while PFC is computed over categorical missing values.
- Performance assessment: Out-of-bag error estimates from random forests are averaged across variables of the same type to approximate true imputation errors after convergence.Estimation quality is assessed by the absolute difference between true imputation error and the OOB imputation error estimate across simulation runs.
3 Methods
The methods compare missForest with four imputation approaches across continuous, categorical, and mixed-type data. Experiments use repeated random-missingness simulations, standardized or dummy-coded inputs where needed, and error-based comparisons including missForest’s OOB estimates.
- Comparative design: missForest is compared with four methods on ten data sets spanning continuous-only, categorical-only, and mixed variable types.
- Continuous-data methods: KNNimpute imputes missing variables from k nearest observed variables using a distance-weighted mean, with k selected by cross-validation.The distance is usually Euclidean, and the tuning parameter k can strongly affect performance.
- Continuous-data methods: KNNimpute inputs are standardized to unit standard deviation and centered at zero to prevent variance-based weighting across differently scaled variables.
- Continuous-data methods: MissPALasso models missing variables from observed ones with lasso-penalized regressions and partially updates the latent distribution in an E step.
- Evaluation protocol: Each experiment runs 50 independent simulations removing 10%, 20%, or 30% of values completely at random, then computes NRMSE, PFC, or both.Methods are compared with paired Wilcoxon tests against missForest, while its OOB error estimates are recorded in every simulation.
4 Results
Across continuous, categorical, and mixed-type data, missForest generally achieved the lowest imputation error, with especially large gains in several settings. Its out-of-bag estimates were usually accurate, while computational costs were higher than KNNimpute but lower than other compared methods.
- Continuous data: MissForest reduced continuous-data average NRMSE by up to 25% versus KNNimpute and by more than 50% for musk molecules.MissPALasso performed slightly better on gene-expression data, while its computation was infeasible for the high-dimensional Insulin data set.
- Computational cost: MissForest typically stopped after about 5 iterations, took about 10 times as long as cross-validated KNNimpute, and required about 2 hours for Insulin imputation.Despite being slower than KNNimpute, missForest ran considerably faster than MICE and MissPALasso and required no antecedent standardization or laborious dummy coding.
- Categorical data: MissForest consistently outperformed the compared categorical-data methods, reducing PFC versus MICE by up to 60% on SPECT data and around 10–20% elsewhere.The amount of missing data had only a minor influence on the performance of all methods, except for MICE on SPECT data.
- Mixed-type data: MissForest generally outperformed the other mixed-data methods, reducing imputation error by more than 50% in many cases.For GFOP, KNNimpute had slightly lower NRMSE but twice as much error on categorical variables; NRMSE was more variable than PFC as missingness increased.
- OOB error estimates: OOB imputation-error estimates differed from true error by only a few percents for Isoprenoid and Musk data and were comparably good on Parkinson’s data despite greater variability.For categorical data sets, estimation accuracy behaved similarly across scenarios, with a tendency to underestimate error as missingness increased.
5 Conclusion
MissForest provides broadly applicable imputation for mixed-type data without tuning parameters or distributional assumptions. Its advantages are especially pronounced for complex, nonlinear, unequal-scale relationships and high-dimensional data.
- Applicability: MissForest imputes essentially any data type, including multivariate data with continuous and categorical variables simultaneously.It is designed to handle mixed-type data in one imputation procedure.
- Methodological advantages: MissForest requires neither tuning parameters nor distributional assumptions about the data.The method is therefore nonparametric and avoids parameter calibration and distributional modeling requirements.
- Empirical performance: MissForest outperforms established imputation methods such as k-nearest neighbour on real data sets from biological and medical fields.The comparison was conducted across several real data sets.
- Empirical performance: MissForest is most useful when variables have complex interactions or nonlinear relations, particularly across unequal scales and different types.These conditions are identified as the settings in which the method deploys its full potential.
- High-dimensional data: MissForest handles high-dimensional data where the number of variables may greatly exceed the number of observations while still providing excellent imputation results.The conclusion specifically highlights applicability when variables substantially outnumber observations.