PaperlayerView exact revision on arXiv

Source-linked AI summary

Processing of missing data by neural networks

Marek Smieja, Łukasz Struski, Jacek Tabor, Bartosz Zieliński, Przemysław Spurek

arXiv:1805.07405v3cs.LGstat.ML

TL;DR

Missing inputs are often handled by imputing single values, while some neural approaches require complete training outputs. This paper represents missing values with a probability density and uses expected neuron responses in the first hidden layer. Across the reported tasks, it outperforms imputation methods in many comparisons and is comparable to context encoders despite their access to complete training data.

  • Problem

    Neural networks need a way to process missing inputs, but some existing neural approaches require complete training outputs, which many datasets do not provide.

    Source: §1 · p. 1

  • Method

    The method represents missing attributes with a probability density and replaces first-hidden-layer neuron responses with their expected values under that density.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across reported tasks, the method outperformed imputation techniques in almost all cases and achieved results comparable to context encoders despite their access to complete training data.

    Source: §5 · pp. 7–9, §6 · p. 9

  • Takeaways & Limitations

    The approach provides a theoretically justified way to train varied neural architectures using only incomplete samples and minimal architectural modification.

    Source: §1 · p. 1, §6 · p. 9

    When removed image patches were smaller, all considered methods performed well and could not be visibly distinguished.

    Source: §5 · p. 7

Abstract

from arXiv · show

We propose a general, theoretically justified mechanism for processing missing data by neural networks. Our idea is to replace typical neuron's response in the first hidden layer by its expected value. This approach can be applied for various types of networks at minimal cost in their modification. Moreover, in contrast to recent approaches, it does not require complete data for training. Experimental results performed on different types of architectures show that our method gives better results than typical imputation strategies and other methods dedicated for incomplete data.

1 Introduction

The paper proposes representing missing inputs probabilistically and replacing first-hidden-layer responses with expected activations. The approach is designed for training on entirely incomplete datasets and requires only minimal network modification.

1 Introduction

  • The method models uncertainty in absent attributes with a probability density instead of completing each input with a single imputed value.The density is trained jointly with the network parameters.Source: §1 · p. 1
  • The first hidden layer computes each neuron's expected activation over the missing-data distribution.This is equivalent to averaging activations over imputations drawn from that distribution.Source: §1 · p. 1, Figure 1 · p. 2
  • It can train on datasets containing only incomplete samples, unlike cited approaches that require complete training outputs.The paper contrasts it with context encoders, denoising autoencoders, and modified GANs.Source: §1 · p. 1
  • The authors report theoretical justification without loss of information and practical usefulness across several network types.They state that the approach requires only minimal architectural modification.Source: §1 · p. 1

2 Related work

Prior work handles incomplete inputs through imputation, probabilistic modeling, or specialized learning methods. Some neural approaches require complete training targets, while others process incomplete inputs directly or address particular inference settings.

2 Related work

  • Common strategies fill absent attributes using observable values, predict missing features with separate models, or iteratively refine imputations [6–13].The cited methods include mean and k-NN imputation, iterative filling, GAN-based filling [12], and supervised imputation.Source: §2 · p. 2
  • Other work models incomplete data probabilistically or learns decision functions from visible inputs, with methods for particular learners and assumptions [16–23, 26–30].Examples include expected-risk modeling, low-rank kernel classification, and matrix completion.Source: §2 · p. 2
  • Specialized neural approaches include input-feedback imputation, rough-set networks for imprecise outputs, and a deep Boltzmann machine supporting inference with missing inputs.Source: §2 · p. 2
  • Context encoders, modified GANs, and denoising autoencoders were adapted from image tasks but require complete network outputs during training.The passage notes this requirement conflicts with datasets that lack complete samples, such as many medical datasets.Source: §2 · p. 2

3 Layer for processing missing data

The method represents an incomplete input as a probability density on the affine subspace fixed by its observed attributes, then processes that distribution through expected neuron responses. Only the first hidden layer needs modification; Gaussian-mixture assumptions support practical density estimation and efficient activation calculations.

3 Layer for processing missing data

  • An incomplete input is represented by a conditional density restricted to the affine subspace consistent with its observed coordinates.The resulting density is degenerate in the full input space and serves as a probabilistic representation of the missing point.Source: §3 · p. 3
  • The approach uses a diagonal-covariance Gaussian mixture, estimated with an adapted EM algorithm under MAR or learned with the network in the more general case.Diagonal covariance reduces parameters in high-dimensional problems, and regularization moves the conditional density toward the marginal density in the limit.Source: §3 · pp. 3–4
  • For a missing-data density, a neuron's generalized response is its mean output over that probability distribution.Source: §3 · p. 3
  • Only the first hidden layer needs generalized responses; the remaining architecture stays unchanged, provided the expected value under the density can be computed.The paper reports efficient calculations for ReLU and RBF neurons with Gaussian mixtures.Source: §3 · pp. 3, 5
  • For ReLU neurons, the Gaussian-mixture input induces a mixture of one-dimensional Gaussian weighted inputs used to calculate the generalized response.The stated theorem considers mixtures of possibly degenerate Gaussians.Source: §3 · p. 4
  • A network using the generalized response can also process complete points, but accurate missing-input testing requires incomplete training data to fit the density model.Source: §3 · p. 5

4 Theoretical analysis

The section argues that generalized first-layer responses do not lose information and establishes that generalized ReLU networks can distinguish probability measures satisfying condition (4). It also reports analogous results for RBF activations and a broader result for neuron families with UAP on compactly supported measures.

4 Theoretical analysis

  • Generalized ReLU networks distinguish any two probability measures satisfying condition (4), establishing the section’s central identifiability result.The theorem’s proof is presented as a sketch based on a modification of standard UAP proofs.Source: §4 · p. 5
  • The proof extends the response-generated function family to bounded continuous functions, then uses characteristic functions to conclude equality of measures.The paper presents this as justification that generalized first-layer responses do not lose information.Source: §4 · pp. 5–6
  • An analogous result holds for RBF activations, and neuron families satisfying UAP can identify any probability measure with compact support.Source: §4 · p. 6

5 Experiments

Experiments tested the approach on autoencoder reconstruction and MLP and RBFN classification; the reported comparisons favor it over imputation methods, with RBFN results also exceeding modified kernel methods in almost all cases.

Autoencoder

  • For MNIST reconstruction, outputs were sharper than competing methods, and reconstruction errors were lower than imputation methods over both the whole image and the area outside the mask.Context encoder was slightly better inside the mask but required complete training data.Source: Table 1 · p. 7, §5 · p. 7

Experiments

  • Evaluation covered MNIST image reconstruction, ESR multiclass classification, and small low-dimensional binary UCI tasks with internally missing attributes.The MLP used 25%, 50%, 75%, and 90% missingness; classification results used cross-validation and accuracy.Source: §5 · pp. 7–8

Multilayer perceptron

  • On ESR classification, the model was comparable to context encoder despite CE's access to complete training data; classical imputations were reasonable only with few missing values.Dropout was slightly less robust than the proposed model, while the complete-data MLP reached 0.836 accuracy.Source: Table 2 · p. 8, §5 · p. 8

Autoencoder

  • For the high-dimensional MNIST experiment, mice could not construct imputations and EM could not fit a GMM because of singular covariance matrices.Source: §5 · p. 8

Radial basis function network

  • The RBFN model outperformed imputation techniques in almost all cases and was more accurate than modified kernel methods that directly process incomplete data.The reported comparison is based on binary classification on UCI datasets with internally missing attributes.Source: Table 4 · p. 9, §5 · p. 9

6 Conclusion

The conclusion presents a general approach for neural networks trained on incomplete-only datasets, supported by a theoretical justification and experiments across tasks and architectures.

Conclusion

  • The approach adapts neural networks to incomplete data and can be trained using datasets containing only incomplete samples.Source: §6 · p. 9
  • The method introduces an input layer for missing data that applies across a wide range of networks without extensive architectural modification.Source: §6 · p. 9
  • The paper states that the generalized neuron response is theoretically justified and reports practical usefulness across diverse tasks and architectures, with results comparable to methods requiring complete training data.Source: §6 · p. 9

A Missing data representation

This section concerns regularizing conditional probability densities and presenting formulas for the Gaussian-mixture case.

Missing data representation

A.1 Regularized conditional density

The section represents a partially observed point with a conditional density on values consistent with its known coordinates. It regularizes this density for Gaussian models and develops theoretical results for neuron families.

A.1 Regularized conditional density

  • A missing point is represented by restricting the data density to the affine subspace of points matching its known coordinates, yielding a conditional density.The paper models this as a probabilistic representation of the missing point.Source: §A.1 · p. 11
  • Gaussian-mixture conditionals can become unreliable far from data clusters, so γ-regularization controls the influence of density away from the subspace.As γ approaches zero, the regularized density converges to the conditional density; as γ approaches infinity, it converges to the marginal density.Source: §A.1 · pp. 11–12

A.2 Gaussian model for missing data density

  • For high-dimensional Gaussian-mixture modeling, the paper uses diagonal covariance matrices to reduce parameter count and support more reliable estimation.It derives an analytical formula for the regularized conditional density in this setting.Source: §A.2 · p. 13

B. Theoretical analysis; B.1 Identification property for RBF

  • The theoretical analysis considers RBF neurons for arbitrary probability measures and defines an RBF as a Gaussian density parameterized by its mean and covariance.It also considers diagonal or rescaled-identity covariance matrices.Source: B Theoretical analysis · p. 13, §B.1 · p. 14

B.2 General identification property

For compactly supported probability measures, the paper proves that a neuron family with the universal approximation property identifies a measure from its neuron expectations. It also reports supplementary material on datasets and incomplete-image reconstructions.

B.2 General identification property

  • For compactly supported measures, equal expected responses across every neuron in a family with the universal approximation property imply that the measures are equal.The theorem applies when the family can uniformly approximate continuous functions on compact sets.Source: §B.2 · p. 14
  • Table 5 summarizes datasets after 50% of their values were randomly removed to create missing samples.Source: Table 5 · p. 15
  • Table 6 reports classification accuracy on UCI datasets with 50% of attributes removed; the supplied caption does not state the results.Source: Table 6 · p. 15

C Reconstruction of incomplete MNIST images

D Additional RBFN experiment

In an additional RBFN experiment with 50% of UCI dataset values removed, the method outperformed imputation techniques in almost all cases and was slightly better than the KARMA algorithm. The section also analyzes the computational cost of the missing-data layer.

D Additional RBFN experiment

E Computational complexity

  • With k Gaussian components and t first-layer neurons, the missing-data layer costs O(k|J′| + tk(D + |J|)); generalized activations can be about 2k times slower than complete-data processing.Complete-data processing requires O(tD) ReLU computation.Source: E Computational complexity · p. 17

F Learning missing data density

The network receives an initial Gaussian mixture whose parameters are tuned jointly with its weights. In a toy classification example, learning the missing-data density produced a better classification outcome than estimating the mixture directly with EM.

F Learning missing data density

  • The initial Gaussian-mixture distribution is passed to the network, and its parameters are tuned jointly with the network weights to minimize overall cost.Source: F Learning missing data density · p. 17
  • In the toy example, values were removed from randomly selected points with x1 < 0, leaving missing samples only from the two left-side Gaussian components.Source: F Learning missing data density · p. 17
  • After training, the first learned mixture component estimates class 1 density and the second matches class 2; classification was better than with a GMM estimated directly by EM.Source: F Learning missing data density · p. 17

Related papers

Loading 1805.07405v3…