PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixed Engression: Distributional Regression for Multivariate Mixed-Type Responses

Seyong Hwang, Kean Ming Tan, Ji Zhu

arXiv:2610.32850v1stat.MEstat.ML

TL;DR

Estimating mixed continuous and binary response distributions requires modeling dependence across different supports while accommodating potentially heavy-tailed continuous outcomes. Mixed engression fits a continuous generator using bounded full-response and binary energy losses, then thresholds its soft binary outputs. Its theory gives convergence guarantees under stated smoothness and margin conditions, and simulations report gains under full-response and binary energy distances.

  • Problem

    Mixed-response distributional regression must capture dependence across continuous and binary coordinates while respecting their different supports and allowing continuous outcomes without finite first moments.

    Source: §1 · p. 2, §3.1 · p. 6

  • Method

    Mixed engression fits a continuous soft generator with an augmented bounded energy loss combining full-response and binary-vector comparisons, then thresholds the binary outputs for sampling.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    Under compositional smoothness and a uniform threshold-margin condition, sparse ReLU estimators have polynomial fixed-dimension convergence rates up to logarithmic factors; simulations show heavier-tail gains under full-response energy distance and binary-distribution gains from augmentation.

    Source: §1 · p. 3, §6 · p. 24

  • Takeaways & Limitations

    Evaluation discrepancy matters: bounded full-response distance limits the contribution of large pairwise distances, while Euclidean energy distance remains sensitive to their magnitude.

    Source: §6 · p. 24

    The binary structural function is restricted to admit a smooth compositional representation on the closed uniform-noise cube.

    Source: §4.2 · p. 14

Abstract

from arXiv · show

Estimating the conditional distribution of multivariate responses with continuous and binary components requires capturing their dependence, respecting their different supports, and accommodating potentially heavy-tailed continuous outcomes. We propose mixed engression, which fits a continuous generator using energy losses induced by bounded discrepancies and then thresholds its soft binary outputs to produce mixed-response samples. Because the discrepancies are bounded, the population objective remains well defined without response-moment assumptions. However, we show that the full-response energy distance for the complete response vector can tend to zero before thresholding yet remain bounded away from zero afterward as response dimensions grow. A separate energy loss on the entire binary vector captures discrepancies obscured by the full-response comparison and controls the effect of thresholding. The generator is therefore fitted using an augmented energy loss that combines the full-response and binary components. Under compositional smoothness of the structural functions and a uniform threshold-margin condition, we derive nonasymptotic bounds on the final sampler's full-response error for sparse ReLU estimators of models driven by Gaussian or Student-t structural noise, yielding polynomial convergence rates up to logarithmic factors. We also establish consistency when the covariate, response, and noise dimensions grow under suitable conditions. Simulations show improved accuracy relative to the comparison methods under the full-response energy distance, including in heavy-tailed settings. Moderate binary augmentation substantially improves binary-distribution accuracy while largely preserving full-response accuracy.

1 Introduction

Mixed-response distributional regression must capture dependence across continuous and binary coordinates, respect their different supports, and remain usable for heavy-tailed outcomes. Mixed engression combines bounded full-response and binary energy losses, thresholds soft binary outputs, and is supported by finite-sample theory and simulations.

1 Introduction

  • Conditional distributions capture uncertainty and heterogeneity beyond a single conditional functional, and their joint form captures dependence among response coordinates.They also support conditional sampling, prediction intervals, and distributional summaries, including conditional means when they exist.Source: §1 · p. 1
  • Mixed-response regression must model dependence across different supports, while continuous outputs do not enforce binary support and Euclidean energy loss requires finite first moments.These challenges motivate a loss that remains defined without response-moment assumptions and controls the final mixed-response samples.Source: §1 · p. 2
  • A bounded full-response loss combined with a binary-vector loss helps control thresholding error that the full-response comparison alone can miss.An example shows that the soft generator’s full-response energy distance can tend to zero while the thresholded sampler’s error remains bounded away from zero as response dimensions grow.Source: §1 · p. 2
  • Under compositional smoothness and a uniform threshold-margin condition, sparse ReLU estimators have nonasymptotic bounds for Gaussian- or Student-t-driven models and polynomial fixed-dimension rates up to logarithmic factors.The results also establish consistency as covariate, response, and noise dimensions grow under suitable conditions.Source: §1 · p. 3
  • Mixed engression improves full-response energy accuracy over standard engression, with larger gains in heavier-tailed settings; moderate binary augmentation improves binary-distribution accuracy while largely preserving full-response accuracy.The two loss components have complementary effects in the simulations.Source: §1 · p. 3

1.1 Notation

  • The notation defines P_Z as the distribution of a random vector Z and P_{g|X=x} as the generator output distribution with x fixed and randomness over auxiliary noise.The paper also uses ≲ for an eventual constant-factor upper bound and ≍ when both sequences bound each other up to constants.Source: §1.1 · p. 3

2 Preliminaries

The preliminaries introduce energy discrepancies and scores, standard engression, and sparse ReLU generator classes. These provide the comparison tools and estimator class used by mixed engression.

2.1 Energy Statistics and Engression

  • Energy distances arise from negative-type discrepancies; bounded discrepancies make their defining expectations finite without moment conditions, and strong negative type characterizes equality of distributions.The Cauchy discrepancy is bounded and of strong negative type, whereas the Euclidean discrepancy is unbounded and its energy distance requires finite first moments.Source: §2.1 · p. 4
  • Minimizing expected negative energy score is equivalent to minimizing energy distance, and standard engression fits a generator using the Euclidean energy-score objective.Under correct specification and an attained minimum, each population minimizer recovers the conditional response distribution almost everywhere in the covariates.Source: §2.1 · p. 5

2.2 Sparse ReLU Generators

  • The paper parameterizes candidate generators with sparse ReLU networks and minimizes the empirical energy loss over that class.The class’s approximation properties and covering-number bounds support the consistency analysis.Source: §2.2 · p. 5

3 Mixed Engression

Mixed engression fits a continuous soft generator for mixed responses, then thresholds its binary outputs to obtain mixed-type samples. Its augmented loss and population guarantees address the gap between fitting the soft distribution and evaluating the final thresholded sampler.

3.1 Mixed-Response Model

  • Shared structural noise represents dependence within and across the continuous and binary response blocks, while the underlying structural generators are not identifiable targets.Directly optimizing through the discontinuous binary threshold would also make the estimation problem discontinuous.Source: §3.1 · p. 6

3.2 Soft Generator and Thresholded Sampler

  • A soft generator uses uniform auxiliary noise and relaxed binary outputs during optimization; thresholding those outputs produces the final mixed-type sampling distribution.The target is the conditional distribution, not recovery of the original structural functions.Source: §3.2 · p. 7

3.3 Mixed-Response Discrepancies and Energy Distances

  • As response dimensions grow, full-response energy distance can vanish before thresholding while binary energy distance detects missing binary variation at a nonvanishing scale.The augmented discrepancy combines full-response and binary components; the full-response component assesses dependence across the response blocks.Source: §3.3 · pp. 8–9

3.4.1 Population Criterion

  • The population objective combines the soft full-response and binary errors, and its infimum can be zero without being attained when a binary coordinate is conditionally nondegenerate.A continuous soft generator can approach binary values 0 and 1 but cannot exactly produce a non-degenerate binary distribution while remaining continuous in auxiliary noise.Source: §3.4.2 · pp. 10–11, §3.4.1 · p. 10

3.4.2 Population Guarantee and Threshold-Transfer

  • If the augmented error tends to zero, both the soft full-response error and the final thresholded full-response error tend to zero, even without an exact population minimizer.The binary error controls the additional error introduced by thresholding.Source: §3.4.2 · p. 11

3.5 Finite-Sample Estimator

  • The finite-sample estimator minimizes an empirical Monte Carlo approximation to the augmented risk using soft generator outputs, then applies thresholding after fitting.The approximation replaces the data expectation with a sample average and auxiliary-noise expectations with Monte Carlo averages.Source: §3.5 · p. 12

4 Finite-Sample Theory for Mixed Engression

The theory bounds the full-response error of the thresholded mixed-response sampler under Gaussian and Student-t structural noise, then extends consistency to growing dimensions. Its guarantees rely on compositional smoothness and a uniform threshold-margin condition.

4.1 Compositional Smoothness for Mixed Responses

  • The smoothness framework allows unbounded function values while retaining compositional and Hölder controls, permitting unbounded conditional support; binary smoothness is imposed on the latent score before thresholding.Each component coordinate depends on a bounded number of inputs, and the unbounded class includes the Gaussian location example Y | X=x ∼ N(x, 1).Source: §4.1 · pp. 13–14

4.2 Finite-Sample Bounds under Gaussian Structural Noise

  • The Gaussian model uses shared structural noise for continuous and binary components, with a uniform threshold-margin condition controlling binary-score concentration near thresholds.The continuous component receives unbounded Gaussian noise directly, while the binary score uses its uniform transform; the latter is a structural smoothness restriction.Source: §4.2 · pp. 14–15
  • For Gaussian structural noise, the augmented-error bound combines sample and Monte Carlo stochastic error with sparse-ReLU approximation error, and the final sampler bound follows through threshold transfer.The polynomial exponents depend on compositional structure and smoothness, and the bound tracks ambient dimensions.Source: §4.2 · p. 16
  • Up to logarithmic factors, the Gaussian-model upper bound for final thresholded full-response error is n^-min{(1−λ)/4, r/2}, yielding convergence when λ < 1 and r > 0.The result holds with m ≥ 2, including when m is fixed.Source: §4.2 · p. 16

4.3 Mixed Responses with Student-t Structural Noise

  • For every fixed ν > 0, the final thresholded sampler is consistent under the Student-t structural model, even when m ≥ 2 is held fixed.The degrees of freedom affect the approximation exponent underlying the bound.Source: §4.3 · p. 17

4.4 High-Dimensional Consistency of the Thresholded Generator

  • High-dimensional consistency permits covariate, response, and noise dimensions to grow under sufficient growth conditions, with allowable response and noise growth depending on the noise-tail approximation exponent.The structural depth, intrinsic dimensions, smoothness, and radius parameters remain uniformly controlled along the sequence.Source: §4.4 · pp. 18–19

5 Numerical Study

The numerical study evaluates mixed engression against neural and non-neural alternatives, separating the effects of the bounded full-response loss and binary augmentation. Results favor engression under the full-response energy distance, while showing a tradeoff across metrics and noise distributions.

5.1 Experimental Design

  • The experiments vary sample size, response dimensions, and Gaussian or Student-t structural noise, evaluating full-response and binary-vector energy distances.The comparisons include standard and mixed engression, DRF, kNN, and kNN+logit; mixed engression varies the binary augmentation weight ω.Source: §5 · p. 19, §5.1 · p. 20

Sample-size behavior and overall performance

Effects of the full-response energy loss and binary augmentation

  • Mixed engression with ω = 0 has lower mean EDF than standard engression in all six configurations; positive weights add only about 1.6%–4.3% reduction over ω = 0 for the best weights under Gaussian and t5 noise.This comparison isolates replacing the Euclidean energy loss with the full-response energy loss, then adding binary augmentation.Source: Effects of the full-response energy… · p. 22, Table 2 · p. 22
  • Under t2 noise, increasing ω from 0 to 0.1 reduces EDB from 9.08 to 2.87, a 68.4% reduction; every positive weight lowers mean EDB across configurations.The binary energy distance assesses the conditional distribution of the entire binary vector.Source: §5.1 · p. 20, Effects of the full-response energy… · p. 22, Table 2 · p. 22
  • Under t2 noise, increasing ω from 0.1 to 0.2 raises mean EDF from 7.55 to 8.37 while leaving mean EDB essentially unchanged.Moderate augmentation can improve binary accuracy while largely preserving full-response accuracy, whereas a larger weight need not improve the overall fit.Source: Effects of the full-response energy… · p. 23

Sensitivity to the structural noise distribution

  • From Gaussian to t2 noise, mean EDF rises 71.2% for standard engression, versus 8.6% for mixed engression with ω = 0 and 11.7% with ω = 0.1.Mixed engression’s relative advantage under EDF becomes more pronounced in the heavier-tailed settings considered.Source: Table 2 · p. 22, Sensitivity to the structural noise… · p. 23
  • Under t3 and t2 noise, standard engression has the lowest mean ED among engression methods; the bounded and Euclidean distances can favor different fits because they treat large separations differently.Mixed engression does not uniformly outperform under ED, which remains sensitive to the magnitude of large continuous differences.Source: Sensitivity to the structural noise… · p. 23

6 Discussion

Mixed engression combines bounded full-response and binary energy losses in a continuous generator whose soft binary outputs are thresholded for sampling. The paper gives conditional guarantees under stated regularity conditions and reports simulation benefits that depend on the evaluation distance.

Method

  • Mixed engression fits a continuous soft generator with bounded full-response and binary energy losses, then thresholds its binary outputs to obtain mixed-response samples.The binary loss supplies control after thresholding that the full-response distance alone may not provide uniformly as dimensions grow.Source: §6 · pp. 23–24

Theory

  • Under compositional smoothness and a uniform threshold-margin condition, sparse ReLU estimators have polynomial convergence rates up to logarithmic factors for fixed dimensions and structural parameters.The guarantees cover shared Gaussian or Student-tν structural noise and unbounded continuous support, including cases without finite first response moments; polynomial convergence with growing dimensions holds under stated regularity and growth conditions.Source: §6 · p. 24

Numerical results

  • In the heavier-tailed simulations, mixed engression improves more under the full-response energy distance, while binary augmentation improves binary-distribution accuracy; standard engression can perform better under Euclidean energy distance.The bounded full-response discrepancy limits the contribution of large pairwise distances, unlike the Euclidean discrepancy.Source: §6 · p. 24

Related papers

Loading 2610.32850v1…