Source-linked AI summary
Conditional-Independence-Regularized Distributional Autoencoders for Mixed-Type Data
Siyuan Tang, Gongjun Xu, Ji Zhu
TL;DR
Mixed-type representation learning must recover full conditional distributions while preserving relationships between numerical and categorical variables, which existing approaches do not reliably ensure. The paper introduces Conditional-Independence-Regularized Distributional Autoencoders, combining distributional losses with structural regularization, and reports improved categorical recovery with competitive overall recovery across synthetic and real-world data.
Problem
Existing representation-learning and generative methods often fail to recover full conditional distributions while preserving interpretable dependence structure in mixed-type data.
Method
The framework combines an energy-score loss for numerical variables, a likelihood-based loss for categorical variables, and auxiliary conditional-independence regularization.
Results
CI-Reg and CI-Reg-0 improve categorical conditional-distribution recovery and achieve competitive or superior overall recovery across synthetic and real-world datasets.
Takeaways & Limitations
The learned representations preserve conditional laws while encouraging structural disentanglement between numerical and categorical components.
Takeaways & Limitations
The method relies on population-level expressivity assumptions, and systematic downstream utility remains an open direction for future work.
Abstract
from arXiv · showhide
Mixed-type data containing both numerical and categorical variables arise in many scientific and real-world applications. Existing representation learning and generative modeling approaches typically focus either on reconstruction accuracy or unconditional data generation, but often fail to recover the full conditional distribution of the data while preserving interpretable structural relationships between heterogeneous variable types. In this work, we introduce Conditional-Independence-Regularized Distributional Autoencoders, a framework for learning low-dimensional representations of mixed-type data through conditional distribution matching and structural regularization. Our method combines an energy-score-based objective for numerical variables, a likelihood-based objective for categorical variables, and an auxiliary conditional independence regularization term encouraging the learned representation to capture the dependence between numerical and categorical components. We provide theoretical analysis showing that the optimal representation balances unexplained numerical variability, conditional entropy of categorical variables, and residual conditional dependence. Empirically, the proposed method achieves strong performance on both synthetic and real-world datasets, substantially improving categorical distribution recovery, achieving competitive overall conditional distribution recovery, and preserving mixed-type dependence structure. The code has been made available at GitHub.
1 Introduction
The paper targets low-dimensional representations for mixed numerical and categorical data while recovering full conditional distributions and heterogeneous dependencies. It addresses limitations of mean-focused autoencoders and existing generative approaches through a distributional, structurally regularized framework.
- Mixed-type data combine numerical and categorical variables with complex dependencies that challenge accurate low-dimensional representation learning.
- Mean-focused PCA and autoencoders can underestimate variability and distort higher-order properties such as tail behavior.
- VAE- and GAN-based methods do not guarantee recovery of P(X | Z = e(X)), while flows and diffusion models require latent dimensions matching the data dimension.
- DPA directly matches conditional distributions but is primarily tailored to numerical data and does not naturally extend to categorical variables.
- The proposed framework separates numerical and categorical modeling, combining energy-score, likelihood-based, and conditional-independence objectives.
2 Method
The method learns a low-dimensional encoder and decoder for mixed-type data by matching conditional distributions separately for numerical and categorical variables. Its total objective combines distributional reconstruction with a soft conditional-independence regularizer and is optimized through alternating updates.
- Architecture: The encoder maps mixed-type observations to Z, while the decoder models the conditional distribution of X given Z.
- Architecture: The numerical decoder uses auxiliary noise to capture variability in Xnum | Z, while the categorical decoder outputs softmax probability vectors.
- Numerical reconstruction: Energy-score reconstruction is strictly proper, so its objective is minimized when the learned numerical conditional distribution matches the true one.
- Categorical reconstruction: Categorical reconstruction minimizes a likelihood-based objective equivalent to conditional KL divergence up to an additive constant.
- Conditional independence: The conditional-independence regularizer compares categorical predictors using Z alone versus Z and Xnum to penalize residual predictive information.
- Assumptions: Conditional independence is treated as a soft structural bias because it may not hold exactly and low-dimensional representations limit factorization accuracy.
- Optimization: The total loss weights numerical reconstruction, categorical reconstruction, and conditional-independence regularization, with normalized positive weights interpreted as relative trade-offs.
- Optimization: Training alternates auxiliary-model updates that maximize predictive objectives with encoder–decoder updates that minimize the empirical total loss.
3 Theoretical analysis
The theory characterizes the population-optimal decoder and representation under expressivity and regularity assumptions. The resulting representation balances numerical variability, categorical uncertainty, and residual mixed-type dependence.
- The analysis studies optimal encoders and decoders under assumptions that make the relevant model classes sufficiently expressive and the energy-score objective finite.
- Strict propriety of the energy score implies that the numerical loss is minimized when the decoder matches the true conditional distribution of Xnum given Z.
- For any fixed encoder, the optimal decoder recovers both P(Xnum | Z = e(X)) and P(Xcat | Z = e(X)) almost everywhere.
- Theorem 1 states that the optimal decoder recovers the true conditional distributions given the learned representation.
- The reduced objective balances unexplained numerical variability, categorical conditional entropy, and residual dependence measured by conditional mutual information.
- The characterization extends numerical distributional reconstruction by incorporating categorical uncertainty and dependence disentanglement for mixed-type representations.
4 Experiments
The experiments evaluate mixed-type conditional distribution recovery on synthetic and real-world data using categorical TV distance and overall energy distance. CI-Reg variants consistently improve categorical recovery, while CI-Reg remains competitive or superior for overall recovery.
- Evaluation: The evaluation reports average categorical TV distance and overall energy distance on concatenated numerical and one-hot encoded categorical variables.Additional dependence, component-wise energy, and KS evaluations are deferred to Appendix C.
- Experimental setup: The synthetic study varies latent dimension across {3, 5, 10} and categorical cardinality across {2, 3, 5, 8}, using 15 numerical and 5 categorical variables.Each dataset has training sample size n = 2000 and encoder dimension d = 8.
- Results: CI-Reg and CI-Reg-0 achieve the best categorical TV distance in every synthetic setting and across all four real-world datasets.Lower categorical TV distance indicates more accurate recovery of categorical conditional distributions.
- Results: CI-Reg remains highly competitive for synthetic overall energy distance and achieves the lowest real-world energy distance on three of four datasets.CI-Reg-0 performs best on Insurance, while both variants outperform AE, VAE, and DPA by a large margin.
- Results: On real-world datasets, CI-Reg and CI-Reg-0 consistently outperform AE, VAE, and DPA in mixed-type conditional distribution recovery.The real-world evaluation uses Adult, Insurance, Shoppers, and PM2.5 datasets with repeated independent runs.
- Comparison: AE and VAE perform substantially worse because their reconstruction objectives target conditional means rather than full conditional distributions.DPA generally improves over AE and VAE but remains below the proposed mixed-type framework.
5 Discussion
The discussion presents CI-Reg as a mixed-type representation-learning framework that combines distributional reconstruction with conditional-independence regularization. It reports strong recovery and dependence preservation while identifying downstream utility and broader variable-group constraints as future directions.
- Contributions: CI-Reg combines distributional reconstruction with conditional-independence regularization to preserve conditional laws and encourage structural disentanglement.Theoretical analysis balances unexplained numerical variability, categorical uncertainty, and residual conditional dependence.
- Findings: Empirically, CI-Reg improves categorical recovery, achieves competitive overall recovery, and preserves mixed-type dependence structure across synthetic and real-world datasets.CI-Reg-0 has training complexity comparable to DPA, while CI-Reg introduces additional computational overhead.
- Scope and future work: Downstream predictive utility is not the paper’s focus, although an appendix simulation shows embeddings recovering latent cluster structure.Systematic evaluation for clustering, generation, and prediction remains future work.
- Future work: Extending conditional-independence regularization beyond numerical–categorical pairs and across multiple variable groups remains an open direction.Such extensions could incorporate domain knowledge and further improve interpretability and structural disentanglement.
A Proofs
The proofs characterize optimal decoders and reduce the learning problem to an encoder objective balancing numerical variability, categorical entropy, and residual dependence.
- For a fixed encoder, the optimal numerical decoder recovers the conditional distribution of Xnum given Z.
- The optimal categorical decoder recovers the conditional distribution of Xcat given Z.
- The conditional-independence term equals the conditional mutual information I(Xnum; Xcat | Z).
- After optimizing decoders and auxiliary models, the encoder is characterized by a reduced objective.
- The optimal encoder balances unexplained numerical variability, categorical conditional entropy, and residual dependence between components.
B Experimental details
The simulation evaluates conditional distribution recovery under controlled mixed-type settings, using matched latent variables and comparisons across representation-learning and generative baselines.
- The evaluation isolates learned-versus-true conditional distribution discrepancy by sharing latent variables between training and test samples.
- The simulation varies category count K ∈{2, 3, 5, 8} and true latent dimension dz ∈{3, 5, 10}, producing 12 settings.
- Each simulated dataset contains n = 2000 samples, 15 numerical variables, and 5 categorical variables.
- The comparison includes AE, VAE, CI-Reg, CI-Reg-0, DPA, and DDPM, with the first five sharing an 8-dimensional encoder.
- CI-Reg training uses raw weights λnum = 0.06, λcat = 0.04, and λCInd = 0.03, with 100 auditor updates per step.
B.2 Real data experiments
Real-world experiments cover four mixed-type datasets from demographic, healthcare, online-behavior, and environmental domains, with dataset-specific architectures and regularization tuning.
- Adult concerns income classification, Insurance concerns medical charges, Shoppers concerns purchase intention, and PM2.5 concerns concentration prediction.
- Experiments remove rows with missing values and use 10 independent runs with different random splits.
- Model selection tunes λCInd on validation data using combined association relative Frobenius error before retraining on training and validation data.
- The λnum, λcat, and λCInd normalization constraint defines relative trade-off weights but is not required during optimization.
- Latent dimensions are dataset-specific, with d = 8 for Adult and Shoppers and d = 16 for Insurance.
C.1 Additional evaluation metrics for the simulation in Section 4.1
Additional simulation metrics assess numerical distribution recovery, mixed-type dependence structure, and categorical distribution recovery across settings.
- CI-Reg and CI-Reg-0 consistently achieve strong numerical KS performance and remain competitive with DPA across latent dimensions and category configurations.
- Distributional approaches generally outperform AE and VAE on numerical KS statistics, highlighting the value of modeling conditional distributions rather than means.
- CI-Reg ranks among the best methods for relative Frobenius errors and is typically comparable to or better than DPA.
- CI-Reg and CI-Reg-0 remain highly competitive with DPA on numerical energy distance across nearly all settings.
- CI-Reg and CI-Reg-0 achieve dramatically smaller categorical energy distances than AE, VAE, and DPA in many settings, especially with more categories.
C.2 Sensitivity analysis of λCInd
The sensitivity analysis finds that performance is relatively robust to λCInd, with moderate regularization improving categorical recovery and mixed-type dependence preservation while numerical metrics remain stable.
- Across both simulation settings, performance varies smoothly with λCInd, and a broad range of moderate values yields similar results.The analysis treats conditional independence as a soft structural inductive bias rather than a hard modeling assumption.
- Moderate λCInd generally improves categorical recovery according to categorical TV distance and categorical energy distance.
- Moderate λCInd also tends to improve mixed-type dependence preservation, measured by combined association relative Frobenius error.
- Numerical energy distance and KS statistics remain relatively stable across a wide range of regularization strengths.
- Overall, the regularizer provides a useful structural inductive bias when applied moderately without making the method overly sensitive to the precise λCInd choice.
C.3 Additional structured-factor simulation study
The structured-factor simulations evaluate CI-Reg across latent-dimension specifications and regularization strengths, finding strong categorical recovery, competitive numerical reconstruction, improved dependence preservation, and retained latent structure.
- Simulation design: The simulation uses 15 latent factors partitioned into numerical-only, categorical-only, and shared groups.Factors 1–5 affect only numerical variables, 6–10 only categorical variables, and 11–15 both components.
- Simulation design: Encoder dimensions d = 10, 15, and 20 represent under-, correctly, and over-specified versions of the true latent dimension dz = 15.
- Performance comparison: CI-Reg and CI-Reg-0 substantially outperform AE, VAE, DPA, and DDPM on categorical TV distance and categorical energy distance.
- Performance comparison: CI-Reg remains competitive on numerical energy distance and KS statistics across all considered representation dimensions.
- Performance comparison: CI-Reg generally improves combined association relative Frobenius error over CI-Reg-0, with conclusions consistent across all three latent-dimension specifications.
- Latent structure and sensitivity: Across all three encoder dimensions, performance varies smoothly with λCInd, and moderate values from 0.01–0.1, sometimes 0.3, yield similar performance.
- Latent structure and sensitivity: Moderate conditional independence regularization improves categorical recovery and mixed-type dependence preservation while numerical reconstruction metrics remain relatively stable.
- Latent structure and sensitivity: Across encoder dimensions and regularization strengths, learned representations preserve identifiable latent clusters, including in the more challenging cluster-separation-3.0 setting.Moderate λCInd can produce visibly clearer separation in several settings, while regularization does not appear to harm latent-structure recovery.
C.4 Real data experiments
Real-world experiments assess numerical and categorical distribution recovery, dependence preservation, and reconstruction quality across multiple baselines. CI-Reg performs strongly overall, especially for categorical conditional distributions and mixed-type dependence structure.
- Evaluation metrics: The real-data evaluation reports KS statistics, combined association relative Frobenius error, and separate numerical and categorical energy distances.
- Results: CI-Reg and CI-Reg-0 achieve competitive KS statistics, remain close to DPA, and generally match or outperform AE while outperforming VAE in most settings.
- Results: CI-Reg consistently achieves strong combined association relative Frobenius error across all real-world datasets.
- Results: CI-Reg and CI-Reg-0 remain competitive with DPA for numerical energy distance and generally outperform AE and VAE across datasets.
- Results: CI-Reg and CI-Reg-0 achieve substantially smaller categorical energy distances than competing baselines, often by a large margin.The results support separating numerical and categorical reconstruction objectives for categorical conditional-distribution recovery.