PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Latent Variable Framework for Jointly Modeling Missingness, Measurement Error, and Heterogeneity

Yasin Khadem Charvadeh, Grace Y. Yi, Mithat Gönen, Pouya Faroughi

arXiv:2608.30040v1stat.MLcs.LG

TL;DR

Missingness, measurement error, and population heterogeneity often coexist, but are commonly modeled separately. HTree-VAE unifies imputation, denoising, heterogeneous missingness modeling, and adaptive subgroup sharing through a deep latent-variable architecture. Simulations show substantial improvement over competing methods, while the approach relies on a representative validation subset with paired surrogate and accurate measurements.

  • Problem

    Missingness and measurement error frequently co-occur, while heterogeneous populations have subgroup-specific mechanisms that existing approaches do not jointly model.

  • Method

    HTree-VAE combines hierarchical routing, selective parameter sharing, pattern-aware latent representations, and calibration-based denoising in one probabilistic framework.

  • Results

    Simulation studies demonstrate substantial improvement over existing competing methods under heterogeneous missingness and noisy-measurement settings.

  • Takeaways & Limitations

    A smaller validation subset with paired surrogate and accurate measurements can provide supervision for improving reconstruction quality across the cohort.

  • Takeaways & Limitations

    The measurement-error correction relies on a representative validation subset with paired surrogate and accurate measurements, which can be difficult to collect at one institution.

Abstract

from arXiv · show

Missing data, measurement error, and population heterogeneity are pervasive challenges in analyzing data arising from modern observational studies and machine learning applications. Although these problems frequently coexist and interact, they are often treated separately in existing works. We propose a unified probabilistic framework that jointly addresses these issues utilizing deep latent variable representation. The proposed method integrates a novel hierarchical tree-routed variational autoencoder with pattern-aware latent representations and calibration-based denoising. The framework accommodates missing data mechanisms, including MCAR, MAR, and MNAR, while simultaneously learning subgroup-specific and globally shared latent structure. The introduced reconvergent routing mechanism enables selective parameters to be shared across related subpopulations, which offers flexibility as well as improved statistical efficiency. Simulation studies demonstrate substantial improvements over existing deep generative imputation approaches under complex heterogeneous missingness and measurement-error settings. The proposed framework provides a principled approach for learning from noisy and incomplete data in modern healthcare and other high-dimensional applications.

1 Introduction

Missingness and measurement error often co-occur in observational and machine-learning data, while population heterogeneity creates additional modeling challenges. HTree-VAE addresses these interacting problems within a unified deep latent-variable framework.

  • Missing data and measurement error frequently interact, but methodological work addressing both within one inferential framework remains limited, especially in high-dimensional nonlinear settings.
  • Healthcare data combine heterogeneous, incomplete, and error-contaminated sources, and missingness may be informative about whether tests are ordered, repeated, or recorded.
  • Existing deep generative approaches generally treat recorded covariates as reliable, assume population homogeneity, and may enumerate missingness patterns for MNAR data.
  • Population heterogeneity creates a trade-off between pooled models that cannot specialize and separate subgroup models that sacrifice sample efficiency for shared structure.
  • HTree-VAE jointly imputes missing values, denoises corrupted measurements, models heterogeneous missingness, discovers latent subgroups, and selectively shares parameters across related subpopulations.

2 Problem Formulation

The problem formulation represents incomplete and noisy observations alongside validation data containing accurate measurements. It defines joint goals for imputation, denoising, subgroup discovery, and adaptive parameter sharing.

  • The feature vector is partitioned into continuous and categorical features, with categorical variables represented using one-hot encoded column groups.
  • A binary mask records observed and missing components, allowing the formulation to represent MCAR, MAR, and MNAR missingness mechanisms.
  • Noisy-surrogate features may be continuous or categorical, covering additive measurement noise and misrecorded diagnosis labels.
  • The main study contains surrogate measurements, while a validation subset provides both surrogate and accurate measurements for learning and assessing denoising.
  • The objective combines missing-value imputation, validation-based denoising, heterogeneous subgroup discovery, and sample-efficient sharing with specialization where mechanisms diverge.
  • The input mask determines what enters the model, whereas a second mask identifies where outputs can be assessed against known values, combining imputation and denoising.

3 Modeling and Representation

The framework uses pattern-aware latent variables and a hierarchical tree-routed encoder to model data, missingness, and heterogeneous subgroups. Reconvergent routing and skip connections support adaptive sharing while preserving missingness context.

  • The generative model factorizes data, missingness, and latent variables conditional on a subject-specific missingness-context representation.
  • The latent variable is lower-dimensional than the observed feature vector, controlling the capacity of the compressed representation.
  • The model assumes features are conditionally independent of missingness context given the latent state, while missingness depends on latent state and context rather than raw features.
  • 3.2.2 Hierarchical Tree of Decision Points: The hierarchical encoder combines a root gate, sequential decision points with branch layers, and a VAE head producing variational-posterior parameters.
  • 3.2.2 Hierarchical Tree of Decision Points: Data-dependent routing shares parameters among similar subgroups and allows differences when heterogeneity is substantial, avoiding the extremes of full pooling or separate models.
  • 3.2.3 Architectural Comparison: Routing decisions use evolving task-relevant representations, so subgroup differences that emerge after transformation can guide deeper routing.
  • 3.2.3 Architectural Comparison: Reconvergence lets subjects taking different earlier paths share the same downstream branch parameters, while skip connections preserve missingness context at every depth.
  • 3.2.4 Decoder: The decoder maps latent codes to continuous-feature means and categorical probabilities, using Gaussian observation modeling and categorical cross-entropy.

4 Estimation

The model is estimated with a variational objective because the nonlinear latent-variable likelihood is intractable. Its ELBO combines reconstruction, missingness prediction, and KL regularization, while routing regularization and calibration-based supervision support denoising and balanced subgroup routing.

  • Variational estimation: The nonlinear decoder prevents direct likelihood estimation, so the method approximates the latent posterior and maximizes a tractable lower bound on log-likelihood.The variational strategy uses qϕ and an empirical lower bound rather than directly evaluating the likelihood.
  • Evidence lower bound: The ELBO contains reconstruction, missingness-prediction, and KL-divergence terms, with latent-variable expectations for the first two components.The KL term compares the variational posterior with a pattern-dependent prior.
  • Reconstruction: Reconstruction compares decoder outputs with clean targets where validation supervision exists and with observed features elsewhere, enabling denoising alongside imputation.The target mask excludes unavailable clean references while leveraging observed non-noisy features.
  • Missingness prediction: The missingness predictor uses the latent representation together with global missingness context to estimate feature-wise observation probabilities through a logistic model.Its expected log-likelihood is approximated empirically using reparameterized samples.
  • Latent regularization: The pattern-dependent Gaussian prior maps missingness context to latent means and log-variances, while the posterior and prior yield a closed-form KL divergence.Both distributions are diagonal Gaussians.
  • Optimization and inference: Routing entropy regularization prevents a single branch from dominating, and stochastic gradient descent updates the model through reparameterization and Gumbel–Softmax routing.At test time, the encoder produces a latent code whose decoded output supplies imputations and denoised values, while retained clean observations remain unchanged.

5 Simulation Study

The simulation represents a heterogeneous clinical registry with subgroup-specific latent processes, differential measurement error, and MCAR, MAR, MNAR, or fully observed regimes. It uses held-out clean measurements for denoising supervision and compares HTree-VAE with established incomplete-data baselines.

  • Data-generating design: The synthetic longitudinal cohort contains 8,000 subjects in four latent subgroups with distinct generative parameters and jointly varying measurement-error severity and missingness mechanism.Subgroup sizes are 500, 3,500, 500, and 3,500, and membership is hidden during training.
  • Measurement error: Measurement error is introduced through a noisy baseline surrogate, with substantial contamination in subgroups 1 and 3 and greater precision in subgroups 2 and 4.The surrogate is unbiased but differentially reliable across subgroups.
  • Data-generating design: Each subject has five measurement occasions and 25 biomarker measurements driven by a two-dimensional latent process with subgroup-specific temporal structure.The observable data include baseline covariates and biomarker measurements, while the clean severity score is held out except for validation subjects.
  • Missingness mechanisms: Missingness combines MCAR, MAR, and MNAR mechanisms across subgroups, plus a fully observed subgroup that isolates measurement-error correction from imputation.The MNAR mechanism depends on unobserved biomarker values, whereas the MAR mechanism depends on observed age.
  • Missingness mechanisms: The design produces roughly 11% missing biomarker cells overall, including 30.2% in subgroup 1, 17.5% in subgroup 2, 29.0% in subgroup 3, and none in subgroup 4.In a representative simulation, the overall missingness rate is 11.4%.
  • Data splitting and evaluation: An 80:20 split yields 6,400 training and 1,600 test subjects, with an independent 1,280-subject validation subset providing simultaneous erroneous and accurate measurements.The validation subset supplies supervision for identifying measurement-error correction.
  • Data splitting and evaluation: HTree-VAE is compared with MIWAE and not-MIWAE, using the observed surrogate as input and the clean measurement as the evaluation target in validation subjects.MIWAE assumes MAR, whereas not-MIWAE explicitly models missingness.

6 Analysis Results

Across pooled, task-specific, and subgroup-wise evaluations, HTree-VAE generally achieves the strongest MSE performance, with especially consistent severity-column results. Its imputation advantage varies by subgroup and missingness setting, while simulation rankings show strong overall performance and stability.

  • Pooled performance: Across all evaluated cells, HTree-VAE has the lowest median MSE, about 31.5% below not-MIWAE, and the smallest MSE standard deviation across simulations.The pooled evaluation combines baseline severity values and missing biomarker measurements; PSMVAE has the largest standard deviation and a single-simulation MSE of 104.937 versus 2.987 for HTree-VAE.
  • Task-specific performance: For measurement-error correction on the severity column, HTree-VAE has the lowest median MSE, followed by PSMVAE, MIWAE, and not-MIWAE.This task evaluates recovery of true severity from its error-contaminated surrogate.
  • Task-specific performance: For missing biomarker imputation, HTree-VAE has the lowest median MSE, 30.3% below not-MIWAE, while PSMVAE and not-MIWAE reverse order relative to measurement-error correction.No single baseline is preferable across both tasks, whereas HTree-VAE leads on each.
  • Subgroup-wise performance: HTree-VAE has the lowest severity-column median MSE in all four subgroups, with margins varying according to measurement-error severity and missingness conditions.The reported medians are 0.227, 0.129, 0.324, and 0.136 in subgroups 1–4; the widest margin occurs under heavier contamination in subgroup 1.
  • Subgroup-wise performance: Imputation performance is not uniform across subgroups: ordering depends on whether missingness patterns are informative and whether subgroup size supports model capacity.In subgroup 1, MIWAE has the lowest median MSE at 3.036; in subgroup 2, HTree-VAE dominates with median MSE 2.938; in subgroup 3, HTree-VAE has the lowest mean MSE at 7.052 while not-MIWAE has the lower median at 5.397.
  • Within-simulation rankings: HTree-VAE ranks first in 69% of pooled simulations, is in the top two about 90% of the time, and is never worse than second on the severity column.MIWAE and PSMVAE each finish last in roughly 40–50% of pooled simulations, with PSMVAE showing rare but severe failures that may require diagnostic safeguards.

7 Discussion

The framework jointly models missingness, measurement error, and latent population heterogeneity through hierarchical routing, selective parameter sharing, pattern-aware priors, and calibration-based denoising. Simulations show substantial improvement over competing methods, while the discussion identifies limitations involving architectural benefits, evaluation objectives, time-varying mechanisms, and validation-subset representativeness.

  • HTree-VAE jointly models missingness, measurement error, and latent population heterogeneity within one deep latent variable framework.
  • Hierarchical routing, selective parameter sharing, pattern-aware latent priors, and calibration-based denoising are integrated into the architecture.
  • Limited validation supervision can improve reconstruction quality throughout the cohort without requiring perfectly clean measurements for every subject.
  • Architectural advantages are context-dependent: conditioning on the observed pattern does not reduce bias when subgroup 1 involves only MCAR.
  • The framework uses a standard ELBO, whereas competing MIWAE variants and PSMVAE use a tighter importance-weighted objective, leaving relative performance under a uniform evaluation protocol for further testing.
  • The denoising correction relies on a validation subset containing both surrogate and accurate measurements, whose representativeness may be difficult to obtain at one institution.
Loading 2608.30040v1…