Source-linked AI summary
Missing Data: A Comparison of Neural Network and Expectation Maximisation Techniques
Fulufhelo V. Nelwamondo, Shakir Mohamed, Tshilidzi Marwala
TL;DR
Missing data affects real-time applications and the quality of decisions based on incomplete inputs. The paper compares maximum-likelihood/EM and neural-network approaches, finding that their performance varies across datasets, while EM could fail for one column.
Problem
Missing data in real-time processing can affect the quality of decisions based on the data.
Method
The paper compares maximum-likelihood methods, including EM for incomplete data, with neural-network methods for imputing missing values.
Results
Results vary across datasets: NN-GA achieves higher imputation accuracy in one comparison, while EM performs better in others and fails to predict column 1 in one dataset.
Takeaways & Limitations
Neither approach is uniformly superior across the reported datasets, and EM's applicability can be constrained when imputing column 1.
Takeaways & Limitations
A non-positive definite matrix when imputing column 1 prevented EM from being used to estimate the missing data.
Abstract
from arXiv · showhide
The estimation of missing input vector elements in real time processing applications requires a system that possesses the knowledge of certain characteristics such as correlations between variables, which are inherent in the input space. Computational intelligence techniques and maximum likelihood techniques do possess such characteristics and as a result are important for imputation of missing data. This paper compares two approaches to the problem of missing data estimation. The first technique is based on the current state of the art approach to this problem, that being the use of Maximum Likelihood (ML) and Expectation Maximisation (EM. The second approach is the use of a system based on auto-associative neural networks and the Genetic Algorithm as discussed by Adbella and Marwala3. The estimation ability of both of these techniques is compared, based on three datasets and conclusions are made.
1. INTRODUCTION
The paper addresses real-time estimation of missing input elements by comparing maximum-likelihood/EM techniques with an auto-associative neural-network and genetic-algorithm approach across three datasets.
- Missing input data can affect the quality of decisions based on that data.
- The study compares maximum-likelihood/EM estimation with an auto-associative neural-network and genetic-algorithm combination.The comparison uses three datasets.
2. BACKGROUND
The background characterizes missing-data mechanisms and introduces autoencoders, multilayer perceptrons, and genetic algorithms as tools for modeling relationships and reconstructing incomplete inputs.
- Missing Data: Real-time applications that depend heavily on data can suffer from missing input variables.
- Missing Data: Missingness is categorized as MCAR, MAR, or MNAR according to its dependence on observed and missing variables.
- Missing Data: The work assumes data is missing at random, so missing values are expected to be deducible from remaining data in a complex manner.
- Autoencoder Neural Networks: Autoencoders learn linear and nonlinear interrelationships, including correlation and covariance, by compressing inputs through a smaller hidden representation.
- Autoencoder Neural Networks: The autoencoder uses an MLP architecture with input, hidden, and output layers, trained by back-propagation to reproduce target vectors.
- Genetic Algorithms: The genetic algorithm searches a chromosome-based solution space through selection, crossover, and mutation.
3. NEURAL NETWORKS AND GENETIC ALGORITHM FOR MISSING DATA
The NN-GA estimator combines an autoencoder with genetic optimization: guessed and observed values form a complete input, and GA minimizes reconstruction error to estimate missing entries.
- The method combines auto-associative neural networks with genetic algorithms to approximate missing data.
- A complete vector containing observed and guessed values is fed into the autoencoder.
- Xk denotes known variables and Xu denotes unknown or missing variables in the full input space.
- The difference between target and actual output defines the reconstruction error used by the estimator.
- The error function is minimized with a genetic algorithm to estimate the missing values.
4. MAXIMUM LIKELIHOOD
The paper presents maximum likelihood and EM as model-based approaches for imputing incomplete data, using iterative parameter and missing-value estimation under a statistical data model.
- Maximum likelihood imputes missing values using a precise statistical model, commonly a multivariate Gaussian mixture model.
- The paper considers single imputation and therefore uses the EM algorithm.
- EM is described as assuming data is missing at random rather than missing completely at random.
- EM fits models to incomplete data by exploiting the relationship between missing values and unknown model parameters.
- EM iteratively predicts missing values, updates parameter estimates, and repeats until the parameter sequence converges to maximum-likelihood estimates.
5. EXPERIMENTAL EVALUATION
The study compares Expectation Maximization with a neural-network system coupled to genetic algorithms for approximating missing data across three datasets.
- Expectation Maximization and a neural network coupled with genetic algorithms were compared for approximating missing data.
- The comparison used three different datasets.
A) Power plant data
The first dataset contains measurements from a 120 MW French power plant under normal operating conditions. EM and NN-GA were evaluated using matched training and testing data after normalization.
- The dataset records five power-plant inputs: gas flow, turbine-valve opening, superheater spray flow, gas dampers, and air flow.
- 200 instances were recorded from the power plant, with measurements sampled every 1228.8 seconds.
- One seventh of the data was reserved for testing, with the remainder used for training.
- EM and NN-GA used the same amounts of testing and training data for comparison.
- The data were min-max normalized to [0,1] before use.
B) HIV database
The second dataset is a South African antenatal sero-prevalence survey database from questionnaires completed by eligible first-time participants. It contains reproductive, demographic, educational, and HIV-status attributes.
- The survey data came from questionnaires answered by pregnant women visiting selected public clinics in South Africa.
- The study used HIV status, education level, gravidity, parity, age group, and age gap as attributes.
- Parity counts birth events, while gravidity and parity describe reproductive activity and reproductive health state.
- The dataset contains 5776 instances and was divided into training and testing subsets.
- Testing used 776 instances within a Bayesian-framework training setup.
C) Data from an industrial winding process
The third dataset represents an industrial winding process with reel speeds, motor-current set points, and measured web tensions. The study evaluates missing-data estimation using this process data and correlation-based accuracy measures.
- The plant unwinds a plastic web from reel 1, passes it over the traction reel, and rewinds it on the rewinding reel.
- Reels 1 and 3 are driven by DC motors, while reel speeds and web tensions T1 and T3 are measured.
- The winding-process dataset contains 2500 instances sampled every 0.1 seconds.
- Testing used 500 instances, with the remaining data divided between training and validation.
- The process uses reel speeds S1, S2, and S3 plus motor-current set points I1 and I3 as inputs.
- Effectiveness was assessed using correlation between predicted and actual data and relative prediction accuracy.
- Relative prediction accuracy counts predictions within a specified tolerance; the study used a 10% tolerance chosen as an assumed maximum acceptable error margin.The authors state that the tolerance was chosen arbitrarily for the applications considered.
6. EXPERIMENTAL RESULTS AND DISCUSSION
Across three datasets, EM and NN-GA show dataset- and variable-dependent imputation performance. EM performs better for some cases, while NN-GA achieves higher accuracy in others, especially where input correlations can be exploited.
- Evaluation setup: The experiments compare EM and NN-GA using prediction accuracy and the proportion of estimates within 10% of actual values.The study reports comparative results for power plant, HIV, and industrial winding data.
- Power plant data: EM fails to predict column 1 when the prediction matrix is not positive definite, commonly because variables or their moment elements are linearly dependent.Suggested remedies include deleting dependent variables or replacing collinear variables with principal components.
- Power plant data: NN-GA imputes missing values with higher accuracy in most cases for the power plant dataset.The comparison is presented graphically in Figure 5.
- Interpretation: Neural-network estimation depends on discoverable linear or nonlinear correlations, while EM requires that variables not be linearly dependent.The study tested the data for correlation before experimentation.
- HIV data: EM performs better than NN-GA for predicting Education, Parity, Age, and Age gap in the HIV dataset.The paper attributes weaker NN-GA performance to potentially unrepresentative variables or unreliable questionnaire responses.
- Industrial winding data: Industrial winding results vary by variable: EM performs better for some variables, whereas NN-GA performs better for others.The predicted values are not very correlated with the actual missing variables, and EM is unavailable for column 1 because of a non-positive definite matrix.
7. CONCLUSIONS
The paper compares EM with an auto-associative neural network–genetic algorithm system for missing-data imputation, finding that neither method consistently outperforms the other. Performance depends strongly on the problem and variable dependencies.
- The study compares maximum likelihood and neural network–genetic algorithm approaches for approximating missing data.
- EM achieves better imputation accuracy for some variables, whereas the neural network–genetic algorithm system performs better for others.
- The relative imputation ability of the two methods is highly problem dependent.
- EM performs better when there is very little dependency among the variables.