Source-linked AI summary
Classical methods for modern applications: synthetic tabular data generation using lightweight discrete response continuous latent variable model
Bartosz Barzdajn, Mateusz Krzysztoń
TL;DR
The paper addresses generation and likelihood estimation for mixed continuous and discrete data. It uses quantile transformations and a continuous latent model, handling discrete observations as admissible latent regions in an MCEM procedure. The authors report that quantile transformations can provide robust, high-fidelity models with simple latent priors.
Problem
The paper targets data-generating distributions for mixed continuous and discrete features, motivated by cybersecurity applications requiring representative samples and likelihood estimates.
Source: §2 · pp. 1–2Method
The method maps features through empirical CDFs and selected quantile functions, then uses MCEM with importance-sampled latent regions for discrete observations.
Source: §3 · pp. 2–4, §4 · p. 6Results
Quantile transformations can provide robust, high-fidelity statistical models of data with simple latent priors.
Source: §7 · p. 12Takeaways & Limitations
For a Gaussian latent model, each EM iteration reduces to importance-weight evaluation and weighted maximum-likelihood estimation of the mean and covariance.
Source: §7 · p. 12
Abstract
from arXiv · showhide
In this paper, we present a lightweight algorithm for modelling tabular data containing both continuous and discrete variables. The method uses empirical marginal distributions and a selected quantile function to transform the data into a continuous latent space, where a distribution of choice, e.g. a multivariate Gaussian, can be efficiently estimated and applied. While this approach is closely related to classical well-established machine learning methods, we present a novel adaptation that fits well within the contemporary computational ecosystem. Unlike in the case of deep architectures, it results in a simple implementation with few dependencies, robust training/estimation, as well as efficient sampling. The transformation preserves complex marginal distributions and correlations, providing high-fidelity models, especially for data with unimodal distributions. One of the key characteristics of the proposed approach is its continuous representation of discrete variables. While highly desirable, this introduces a significant challenge, since a single observation corresponds to a region rather than a point in the latent space. We show that this ambiguity naturally leads to a Monte Carlo expectation-maximisation (MCEM) formulation, whose auxiliary objective can be expressed as a weighted latent-space likelihood. This, in turn, allows conventional maximum-likelihood estimators to be used within an efficient and robust optimisation procedure.
1 Note on the future revisions of this draft
The draft presents the complete scientific contribution while inviting feedback to guide future revisions.
1 Note on the future revisions of this draft
2 Motivation
The paper develops data-generating distributions for mixed continuous and discrete data, motivated by cybersecurity digital-twin needs for representative samples and practical implementation.
2 Motivation
- The target is a data-generating distribution for mixed-type data with formal treatment of continuous and discrete features.Source: §2 · p. 1
- The cybersecurity digital-twin component must generate representative synthetic network data and estimate likelihoods of observed network data.The synthetic data may support component testing, training-set balancing, and pre-training; likelihoods may support anomaly-detection decision-boundary evaluation.Source: §2 · pp. 1–2
- The motivating application calls for high-fidelity samples, few dependencies, efficient operation, and robust training without extensive monitoring.Source: §2 · p. 2
- These requirements motivate revisiting classical methods rather than pursuing state-of-the-art deep architectures.The authors draw inspiration from discrete-response latent-variable models and transformations related to Gaussian copula modelling.Source: §2 · p. 2
3 The core concept and its relation to other methods
The approach combines per-feature quantile transformations with a continuous latent model, treating discrete observations as regions of latent space rather than replacing them with continuous surrogates.
3 The core concept and its relation to other methods
- The method draws on a two-stage transformation that maps each feature through its empirical CDF and then a selected quantile function.The second stage can use a continuous distribution with a well-defined quantile function; Figure 1 illustrates the transformation.Source: §3 · pp. 2–3, Figure 1 · p. 3
- Because transformations are applied independently by feature, they reshape marginals while dependence can remain in the multivariate correlation structure.Source: §3 · p. 3, Figure 1 · p. 3
- The formal treatment keeps a continuous latent representation while mapping each discrete observation to a region of latent space.Unlike dequantisation, the method models the ambiguity implied by discrete observations rather than replacing them with continuous surrogates.Source: §3 · pp. 3–4
- Observation-specific latent regions make inference more involved, leading naturally to MCEM with an auxiliary objective expressible as a weighted latent-space likelihood.Source: §3 · p. 4
- The continuous latent representation supports efficient sampling and interpolation, and is presented as a basis for future regression models.The proposed framework also allows changes in device or deployment characteristics to be expressed through changes in latent-distribution parameters.Source: §3 · p. 4
4 The method
The method optimizes a shared latent model by accounting for each observation’s admissible latent region, then uses Monte Carlo sampling to approximate the required expectations.
4 The method
- The method starts from the ELBO framework to approximate likelihood maximization for latent states hidden behind a reversible transformation.Source: §4 · p. 4
- For discrete responses, the encoder is not one-to-one: each observation defines a latent pre-image region rather than a unique latent value.The transformation first applies the marginal empirical CDF feature-wise, then a selected distribution’s quantile function; the paper uses the standard normal target.Source: §4 · p. 5
- The auxiliary objective integrates the joint log-likelihood over each observation’s admissible latent region under an observation-specific variational posterior.The restricted integration domain distinguishes this objective from the standard variational framework.Source: §4 · pp. 5–6
- The E-step uses the current parameter estimate to define a posterior over latent states restricted to each observation’s pre-image; the M-step maximizes expected latent log-likelihood.The objective’s expectations can be approximated with Monte Carlo samples.Source: §4 · pp. 4, 6
- The sampling procedure draws latent candidates from observation-specific regions and uses importance weights to approximate the required integrals.Self-normalized importance sampling is used instead of MCMC, which the authors describe as difficult to implement and computationally expensive across many distinct truncated distributions.Source: §4 · pp. 6–7, Figure 2 · p. 7
- For a multivariate Gaussian latent model, the M-step reduces to weighted maximum-likelihood estimates of the mean vector and covariance matrix.The per-observation weights are normalized and determined by the current model-parameter estimate.Source: §4 · p. 8
5 The final algorithm
The algorithm iteratively estimates latent-model parameters by weighting samples from observation-specific pre-images. For a Gaussian latent prior, each M-step is a weighted maximum-likelihood update.
5 The final algorithm
- 10–30 iterations typically suffice for the EM algorithm to estimate the model parameters.Each E-step produces a weighted latent-space sample representation, reducing the M-step to weighted maximum-likelihood estimation.Source: §5 · p. 8
- Each latent sample z_ij is drawn from proposal ϕ over the pre-image Z_i associated with observation x_i.The proposal samples latent states from the region corresponding to each observation.Source: §5 · p. 8
- Self-normalised importance sampling lets the weights use the parent latent model because observation-specific posteriors differ only by a normalising constant.For the Gaussian case, the parent model is multivariate Gaussian.Source: §5 · p. 8
- Each iteration uses current prior parameters to form importance weights and estimate updated latent-model parameters.The authors use an unbiased covariance estimator in practice; implementation chiefly involves sampling and normalising estimates from observation-specific regions.Source: §5 · p. 9
6 Performance evaluation
The evaluation uses discretised Rosenbrock distributions to test mixed-type modelling across varying difficulty. It compares generated samples with reference data using energy distance.
6 Performance evaluation
- Figure 3 presents unnormalised Rosenbrock densities alongside discretised samples for selected parameter settings.The benchmark was chosen to vary from approximately elliptical distributions to structures with ambiguous marginals.Source: §6 · p. 9, Figure 3 · p. 10
- The benchmark varies across 35 four-dimensional Rosenbrock distributions, discretising two dimensions while leaving two continuous.The scenarios combine parameter settings intended to span simpler distributions and challenging structures, including ambiguous marginals.Source: §6 · pp. 9–10
- Figure 4 visually compares reference test data with DRLVM-generated samples for a = 2.5 and µ = 2.0.Reference data are gray and generated samples are black.Source: §6 · p. 10, Figure 4 · p. 11
- The benchmark compares a quantile-transformation baseline, DRLVM, and optimised CTGAN using energy distance between generated samples and test data.Models generate 2,500 observations for comparison after an 80:20 training/test split.Source: §6 · p. 11
- Figure 5 plots energy distance against Rosenbrock location and precision parameters, then summarises results across all 35 scenarios.Lines average over other parameter settings, shaded regions show 0.95 confidence intervals, and lower distances indicate better agreement.Source: Figure 5 · p. 12
7 Summary, discussion and future work
The authors conclude that quantile transformations and a simple latent prior can yield high-fidelity models, with an efficient weighted-estimation algorithm. They also discuss modular extensions and conditional modelling.
7 Summary, discussion and future work
- The method generates high-fidelity samples on synthetic tests, including difficult distributions, despite its simple latent model.The authors report that CTGAN reproduced overall Rosenbrock shape more accurately but had substantially higher energy distances.Source: §7 · p. 12
- EM iterations reduce to importance-weight evaluation and weighted maximum-likelihood estimation, which for a Gaussian latent model means updating a weighted mean and covariance.The authors describe the resulting method as computationally efficient and straightforward to implement.Source: §7 · p. 12
- The replaceable transformation and latent-model stages allow other quantile functions and models with practical weighted-likelihood estimators.The authors identify mixture models as a possible richer latent model, while noting that retaining simplicity remains an open question.Source: §7 · p. 13
- Conditional generation could use quantile regression for conditional marginals, but changing dependence structure would also need modelling.The authors expect estimation of conditional marginals and latent dependence parameters to be considerably more involved.Source: §7 · p. 13
A.1 Initial definition of the optimisation objective
The derivation uses a latent-variable likelihood objective and its Evidence Lower Bound to avoid direct optimisation of an integral. A variational distribution over latent states supplies the bound.
A.1 Initial definition of the optimisation objective
- The modelling strategy transforms data into a simpler latent space, samples there, and maps samples back to the data space.The derivation begins with the Evidence Lower Bound because the latent states are unobserved.Source: §A.1 · p. 14
- Jensen’s inequality yields a lower bound on log-likelihood that is easier to optimise than the likelihood’s integral.The likelihood is defined for a single data point after the derivation.Source: §A.1 · p. 15
- The variational distribution q(z) is arbitrary in principle but should approximate the true posterior, and its Shannon entropy regularises it against collapse.Maximising the bound with respect to model parameters approaches the maximum-likelihood solution, though the choice of q(z) introduces ambiguity.Source: §A.1 · p. 15
A.2 The transformation
The method maps each feature through its empirical marginal CDF into a continuous latent representation, where discrete observations correspond to admissible regions rather than single states. It turns those regions into an EM objective approximated by observation-specific importance-weighted samples, enabling weighted-likelihood optimisation.
A.2 The transformation
- Each feature is transformed through its empirical marginal CDF and then mapped to the standard normal target; the inverse applies the normal CDF and inverse empirical CDF.The transformations are performed independently by feature.Source: §A.2 · pp. 15–16
A.3 Auxiliary objective function for discrete variables with continuous latent representation
- Discrete observations map to latent pre-image regions, so their variational posteriors are restricted and normalised separately within each observation’s region.The pre-image sets define the integration limits for the auxiliary objective.Source: §A.3 · pp. 16–17
A.4 From a vicious cycle into a successful optimisation procedure.
- The restricted latent posterior supplies observation-specific weights in an EM procedure: estimate the posterior under current parameters, then maximise the expected weighted log-likelihood.The posterior is evaluated only over latent states consistent with each observation.Source: §A.4 · p. 17
A.5 Monte Carlo approximation and the self-normalising importance sampling (SNIS)
- Uniform samples from each observation’s quantile interval cover its admissible latent region; importance weights correct them to estimate expectations under the restricted latent model.The passage motivates importance sampling over per-observation Markov chains or rejection sampling, which it describes as computationally prohibitive.Source: §A.5 · pp. 17–18
A.6 SNIS and the final version of the auxiliary objective in our EM framework
- SNIS turns the integrals into weighted averages, and the final auxiliary objective sums these observation-specific contributions using samples from each pre-image.Weights are normalised separately for each observation; the resulting weighted likelihood can be optimised with maximum-likelihood estimates, including for Gaussian models.Source: §A.5 · p. 19, §A.6 · pp. 19–20
B Deriving the ELBO and EM surrogate objective
- For fixed variational density, the ELBO’s entropy term does not depend on model parameters, leaving the expected joint log-likelihood as the EM auxiliary function.The ELBO comprises expected joint log-likelihood minus the variational density’s expected log-density.Source: B Deriving the ELBO and EM surrogate… · p. 21
C Choosing the variational posterior
The variational density determines how closely the ELBO tracks the log-likelihood. For fixed model parameters, choosing the true latent posterior makes the KL divergence vanish and the bound tight; EM alternates posterior and parameter updates.
C Choosing the variational posterior
- The gap between the log-likelihood and ELBO is nonnegative and equals the KL divergence between the variational density and latent posterior.The figure describes bringing q closer to the posterior as reducing this divergence and tightening the lower bound.Source: C Choosing the variational posterior · p. 22, Figure 6 · p. 22
- For fixed model parameters, the ELBO is tight when the variational density equals the latent posterior, making their KL divergence zero.The log-likelihood minus the ELBO is exactly that KL divergence.Source: C Choosing the variational posterior · p. 22
- EM alternates between moving q closer to the posterior to tighten the bound and maximising the bound over model parameters.With q fixed, maximising the bound moves toward the maximum-likelihood estimate.Source: C Choosing the variational posterior · p. 22
D Likelihood of a sample from the latent distribution (WIP)
The section develops a Monte Carlo estimate of an observation’s marginal likelihood under a latent-variable model. Importance sampling uses a proposal concentrated in regions relevant to the observation, but estimate precision depends on how well that proposal covers those regions.
D Likelihood of a sample from the latent distribution (WIP)
- A proposal q(z) concentrated where an observation is relevant improves on direct sampling from the latent model, whose samples may contribute little to its likelihood.The passage identifies this issue when p(x | z) is concentrated in a small latent-space region.Source: D Likelihood of a sample from the… · p. 23
- The optimal importance-sampling proposal is the latent posterior; it yields a constant importance ratio and a zero-variance estimate.The passage notes that the exact posterior is usually unavailable, motivating a conditional proposal q(z | x).Source: D Likelihood of a sample from the… · p. 23
- The likelihood estimate’s precision depends primarily on whether q(z | x) covers latent regions that contribute substantially to p(x).The estimate uses log weights and LogSumExp over K samples.Source: D Likelihood of a sample from the… · p. 24