PaperlayerView exact revision on arXiv

Source-linked AI summary

CARD: Classification and Regression Diffusion Models

Xizewen Han, Huangjie Zheng, Mingyuan Zhou

arXiv:2206.07275v4stat.MLcs.LGstat.COstat.ME

TL;DR

The paper addresses the difficulty of accurately modeling uncertainty in continuous or categorical responses when mean-focused supervised models are inadequate, especially for multimodal conditional distributions. CARD combines a denoising diffusion-based conditional generative model with a pre-trained conditional mean estimator, and reports state-of-the-art benchmark regression performance plus instance-level classification confidence assessment.

  • Problem

    Mean-focused supervised models may fail to capture the full conditional distribution of y given x, particularly when that distribution is multimodal.

  • Method

    CARD injects covariate dependence into forward and reverse diffusion chains and combines conditional generation with a pre-trained estimator for continuous or categorical responses.

  • Results

    CARD achieves state-of-the-art results on benchmark regression tasks and uses stochastic outputs to assess classification confidence at the instance level.

  • Takeaways & Limitations

    CARD can represent multimodal conditional distributions and provide finer-grained confidence assessment for individual classification instances.

  • Takeaways & Limitations

    PICP can be close to 95% even when the learned and true distributions are mismatched because their percentile intervals are shifted.

Abstract

from arXiv · show

Learning the distribution of a continuous or categorical response variable $\boldsymbol y$ given its covariates $\boldsymbol x$ is a fundamental problem in statistics and machine learning. Deep neural network-based supervised learning algorithms have made great progress in predicting the mean of $\boldsymbol y$ given $\boldsymbol x$, but they are often criticized for their ability to accurately capture the uncertainty of their predictions. In this paper, we introduce classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of $\boldsymbol y$ given $\boldsymbol x$. We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets, the experimental results on which show that CARD in general outperforms state-of-the-art methods, including Bayesian neural network-based ones that are designed for uncertainty estimation, especially when the conditional distribution of $\boldsymbol y$ given $\boldsymbol x$ is multi-modal. In addition, we utilize the stochastic nature of the generative model outputs to obtain a finer granularity in model confidence assessment at the instance level for classification tasks.

1 Introduction

CARD addresses the limitation of mean-focused supervised models by modeling the full conditional distribution of continuous or categorical responses. It injects covariate dependence into diffusion processes and targets p(y | x, D), with reported benefits for multimodal uncertainty and classification confidence.

  • Motivation: Additive-noise models primarily estimate E[y | x] and may poorly capture uncertainty when p(y | x) deviates from their assumptions.This is especially problematic for multimodal conditional distributions, where the conditional mean may be far from any plausible response.
  • Motivation: Missing categorical covariates can produce multimodal conditional distributions, making deterministic mean functions restrictive for uncertainty estimation.The paper gives unknown gender as an example associated with bimodal biological and disease-related outcomes.
  • CARD approach: CARD injects covariate dependence into both forward and reverse diffusion chains to recover p(y | x, D) for continuous and categorical responses.The method combines diffusion-based conditional generation with a pre-trained relation between x and y.
  • CARD approach: The paper studies conditional generative modeling for regression analysis of continuous or categorical responses given corresponding covariates.Its focus is predictive distribution recovery rather than only conditional-mean estimation.
  • Contributions: CARD uses stochastic generative outputs to assess classification confidence at the individual-instance level.This provides finer granularity than previous confidence assessments described in the paper.

2 Methods and Algorithms for CARD

CARD uses a covariate-conditioned diffusion process informed by a pre-trained estimator, then adapts the representation for regression and classification. Reverse diffusion generates samples intended to represent conditional uncertainty and class-specific confidence.

  • Diffusion framework: CARD trains reverse transitions to approximate tractable denoising steps while using a forward process that conditions on x.The objective is derived through an ELBO, with the forward process denoted q(y1:T | y0, x).
  • Diffusion framework: The diffusion endpoint uses fφ(x), which can be a pre-trained estimator of E[y | x] or zero when the relation is unknown.The forward-process mean interpolates between the observed response and this conditional expectation.
  • Regression: For regression, reverse diffusion recovers local observation noise so generated samples can match the true conditional p(y | x).A neural network predicts the forward diffusion noise, following a standard DDPM-style procedure.
  • Classification: For classification, CARD replaces responses with one-hot labels and uses a pre-trained softmax classifier as fφ(x).The continuous diffusion framework reconstructs class prototypes rather than directly sampling from the probability simplex.
  • Classification: Stochastic reverse sampling produces different class-prototype reconstructions for the same x, enabling predicted probability intervals for every class.The construction converts sampled prototype errors into class probabilities through a temperature-weighted Brier-score formulation.

3 Related Work

Prior uncertainty methods often impose parametric response distributions, whereas CARD uses conditional generative modeling to represent continuous or categorical predictive distributions. Related generative approaches differ in whether they model p(y | x) or p(x | y).

  • Uncertainty methods: Bayesian neural networks and related methods quantify predictive uncertainty under supervised-learning formulations for p(y | x).The related-work discussion also describes Neural Processes as targeting out-of-distribution predictive uncertainty, particularly for few-shot learning.
  • Parametric models: Many prior uncertainty models assume Gaussian or Gaussian-mixture forms for p(y | x) and optimize Gaussian negative log-likelihood.These assumptions impose a parametric structure on the predictive distribution.
  • Generative models: GAN-based models have been proposed for conditional density estimation and predictive uncertainty quantification in regression.The paper contrasts these approaches with CARD’s diffusion-based conditional modeling.
  • Generative models: Generative classifiers model classification with generative models, while CARD directly models p(y | x) rather than p(x | y).The cited score-based generative classifiers predict labels using the conditional likelihood of x.
  • Diffusion models: Diffusion models have demonstrated strong performance for high-dimensional multimodal distributions, with extensions also developed for categorical data.Most earlier diffusion work discussed here uses Gaussian processes in continuous state spaces.

4 Experiments

CARD evaluates conditional distribution prediction through distribution-sensitive metrics, toy examples, UCI regression benchmarks, and CIFAR-10 classification. Its experiments show strong distributional reconstruction, state-of-the-art regression performance, improved classification accuracy, and instance-level confidence signals.

  • 4.1.1 PICP and QICE: PICP measures the proportion of true observations inside predicted percentile intervals, while QICE evaluates quantile-interval coverage with finer granularity.PICP uses selected prediction percentiles; QICE partitions generated samples into equal-sized quantile intervals and compares true-data proportions with the ideal 1/M.
  • 4.1.1 PICP and QICE: QICE avoids PICP’s quantile-mismatch failure mode by ensuring every true observation belongs to one generated quantile interval.Under the optimal scenario, QICE is 0, whereas PICP can appear ideal even when learned and true distributions are misaligned.
  • 4.1.2 Toy Examples: CARD reconstructs diverse data-generating mechanisms across eight toy regression tasks, including heteroscedastic and multi-modal conditional distributions.The generated samples blend remarkably well with true test instances across the toy examples.
  • 4.1.3 UCI Regression Tasks: CARD achieves state-of-the-art results on 9/10 UCI datasets for RMSE, 8/10 for NLL, and 5/10 for QICE.It outperforms existing methods on most datasets and remains competitive on the others, despite not explicitly optimizing MSE or NLL.
  • 4.2.2 Classification with Model Confidence on CIFAR-10 Dataset: CARD improves CIFAR-10 test accuracy from 90.39% for a deterministic classifier to 90.93% across 10 runs using majority-vote predictions.Its NLL is competitive among the best reported methods despite training without a cross-entropy objective.
  • 4.2.2 Classification with Model Confidence on CIFAR-10 Dataset: CARD’s stochastic class reconstructions provide instance-level confidence signals: correct predictions have true-class PIWs narrower by an order of magnitude than incorrect predictions.These metrics can indicate whether individual predictions are likely to be trustworthy.

5 Conclusion

CARD models conditional response distributions through conditional generative diffusion, achieving strong distributional prediction while representing multimodality and instance-level classification confidence. Its scope emphasizes aleatoric uncertainty rather than out-of-distribution uncertainty, and classification confidence can depend on the continuous label space.

  • Contributions: CARD approaches supervised learning through conditional generation and estimates p(y | x, D) for regression and classification.The method injects covariate dependence into diffusion chains and uses a conditional generative framework.
  • Contributions: CARD achieves state-of-the-art results on benchmark regression tasks without directly optimizing evaluation metrics.The paper reports this alongside a new distributional evaluation metric, QICE.
  • Distributional modeling: CARD represents conditional distributions with multiple density modes and does not assume a parametric distributional form for p(y | x, D).This contrasts with methods that assume Gaussian or related parametric forms.
  • Scope and uncertainty: CARD focuses on aleatoric uncertainty, whereas Neural Processes Family methods target out-of-distribution settings and can express epistemic uncertainty.CARD is designed for in-distribution supervised learning and models uncertainty intrinsic to the data.
  • Classification: For classification, CARD treats one-hot labels as continuous class prototypes, enabling Gaussian diffusion and instance-level confidence assessment.The approach uses prediction intervals in the raw continuous space before softmax conversion.
  • Classification: Discrete diffusion models may be preferable when the goal is the best NLL and ECE, while CARD is intended for instance-level prediction confidence.The paper notes that a direct comparison with discrete diffusion classifiers remains future work.

A.3 Classification on FashionMNIST Dataset

On FashionMNIST, CARD modestly improves mean accuracy over its deterministic classifier and supports prediction-confidence analysis using prediction intervals and paired t-tests. Narrowest-PIW classification provides a separate confidence-based prediction rule.

  • Accuracy: CARD improves FashionMNIST mean test accuracy from 91.12% for the pre-trained classifier to 91.79%.The comparison uses the same LeNet architecture for CARD and the baseline BNNs.
  • Confidence-based prediction: 89.36% test accuracy is obtained when each instance is assigned the class with the narrowest PIW.The PIW-based rule is evaluated separately from majority-vote classification.
  • Confidence assessment: Table 7 reports PIW values, multiplied by 100, together with paired two-sample t-test results for FashionMNIST.The table summarizes prediction-interval width and statistical rejection outcomes for the classification task.
  • Confidence assessment: The Q-Q plots assess normality using differences between the most and second-most predicted class probabilities across 16 instances.The passage reports close alignment with the 45-degree line, supporting the t-test normality assumption.
  • Confidence assessment: PAvPU measures the proportion of predictions that are correct when confident or incorrect when uncertain.The t-test result is used as a proxy for confidence, and higher PAvPU indicates better alignment between confidence and correctness.

A.6 Classification on Noisy MNIST dataset

On Noisy MNIST, CARD improves classification accuracy and achieves the best reported PAvPU among the compared models, while its uncertainty measures distinguish more and less reliable predictions. The results also suggest a possible human-machine collaboration use case.

  • Experimental setup: The Noisy MNIST experiment uses a two-hidden-layer MLP pre-trained classifier and trains the diffusion model for 1000 epochs.The pre-trained classifier reaches 85.50% test accuracy under the stated experimental setup.
  • Instance-level confidence: CARD’s correct predictions have much narrower true-class PIW than incorrect predictions across the Noisy MNIST test set.The result indicates that CARD tends to be confident in correct predictions and vague when making mistakes.
  • Human-machine collaboration: More than 6% of the data are identified with mean accuracy below 50% and could be passed to human agents under the reported rejection rule.The authors present this as a potential human-machine collaboration benefit while retaining automation for most instances.
  • Benchmark performance: 88.26% accuracy improves from 85.50% for the pre-trained classifier, while CARD achieves 89.12% PAvPU, both best among the compared models.The reported NLL after temperature scaling is also the best among the compared models.

A.7 Classification on Large-Scale Benchmark Datasets

Across CIFAR-100, ImageNet-100, and ImageNet, CARD improves accuracy over the corresponding base classifier across all reported experimental circumstances. Its uncertainty analysis also examines PIW contrasts, t-test rejection groups, and narrowest-PIW predictions at large scale.

  • Datasets: The large-scale evaluation covers CIFAR-100, ImageNet-100, and ImageNet with 100, 100, and 1000 classes, respectively.The reported test-set sizes are 10,000, 5,000, and 50,000 instances.
  • Base classifiers: The base classifiers achieve 71.37% on CIFAR-100 and 82.30% on ImageNet-100 before CARD is applied.The ImageNet base-classifier setup uses ResNet-50 training paradigms, but the cited passage is truncated before reporting all ImageNet values.
  • Benchmark performance: CARD improves accuracy from the corresponding base classifier fφ under all circumstances across CIFAR-100, ImageNet-100, and ImageNet.The comparison aggregates results from 10 experimental runs for each dataset.
  • Instance-level confidence: Large-scale uncertainty analysis finds that true-label PIW variability is strongly related to prediction correctness and that rejected instances are more accurate than non-rejected ones.The analysis reports PIW for selected classes and accuracy by t-test rejection status.

A.8 General Experiment Setup Details

CARD experiments use a shared diffusion training setup, with task-specific network inputs and architectures for regression and classification.

  • Training: CARD experiments use 1000 diffusion timesteps with a linear β_t schedule and Adam-based optimization.Regression uses AMSGrad, while classification uses Adam with cosine learning-rate decay.
  • Evaluation: Classification experiments include CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet, with accuracy and uncertainty-related evaluations reported.The supplied table captions identify classification results across these datasets and report PIW-based analyses.
  • Network architecture: The diffusion network uses timestep embeddings and combines them with covariates, noisy responses, and the pre-trained estimator output.Regression and classification integrate these inputs through task-specific fully connected architectures.
  • Pre-trained estimator: The pre-trained regression estimator is a two-hidden-layer feed-forward network with 100 and 50 hidden units and Leaky ReLU activations.Its form is adjusted according to the task.

A.9 UCI Baseline Model Experiment Setup Details and Dataset Information

The UCI regression comparisons re-run baseline models under standardized experimental procedures and evaluate both predictive accuracy and distributional calibration.

  • Baseline models: The UCI baselines are PBP, MC Dropout, Deep Ensembles, and GCDS, with GCDS implemented because its original code was unavailable.The experiments compare these models under controlled setup details.
  • Training setup: All models generally use learning rate 0.001 and Adam, while inputs and responses are standardized for training and unstandardized for evaluation.PBP on Boston uses learning rate 0.1 instead.
  • Model configuration: Network depth and width, batch sizes, and training epochs are selected across UCI tasks with settings reported in Tables 15–17.Architectures use ReLU for BNNs and Leaky ReLU for GCDS.
  • Datasets: The UCI dataset summary reports each task’s number of observations and features.These dataset dimensions are provided in Table 14.
  • Evaluation: The re-run baselines are evaluated with RMSE, NLL, QICE, and PICP, with PICP results reported for the corresponding experimental runs.Hyperparameters were tuned to produce results better than or comparable with the original reports.
  • Ablation: The prior ablation compares CARD’s N(f_φ(x), I) setting with an N(0, I) prior across different diffusion-step counts on UCI Boston.The comparison examines both predictive and distributional metrics.

A.12 Ablation Study on Diffusion Network Parameterization — CIFAR-10 Dataset

The CIFAR-10 ablation studies how the prior and diffusion-network parameterization affect predictive performance and training speed, while the surrounding regression experiments assess distributional recovery.

  • CIFAR-10 ablation: Four CIFAR-10 model variants combine two prior choices with two diffusion-network parameterizations and are evaluated over 10 runs.Accuracy and PAvPU are reported on the test set.
  • CIFAR-10 ablation: Variants using the informative N(f_φ(x), I) prior outperform variants using N(0, I), while including f_φ(x) directly in the diffusion network changes performance little under the same prior.The variant lacking f_φ(x) in both the prior mean and network input performs worst.
  • Training efficiency: After 20 epochs, CARD with the informative prior reaches nearly 90% one-sample accuracy, compared with about 75% for the N(0, I) variant.The comparison uses mean test accuracy over runs during the first 100 epochs.
  • Toy regression tasks: The toy regression suite spans uni-modal, heteroscedastic, and multi-modal conditional distributions, including inverse sinusoidal, 8-Gaussian, and full-circle tasks.Datasets are generated from eight designed examples and split 80%/20% into training and test sets.
  • Regression evaluation: QICE is low when RMSE is low and PICP is near 95%, while the most difficult log-log cubic task has QICE slightly above 1%.The passage presents this relationship as evidence that QICE tracks mean and distributional fit.

A.14 The Evolution of Samples through the Diffusion Process

The diffusion visualizations track how CARD’s forward and reverse samples evolve across timesteps, including the recovery of multi-modal conditional distributions and improving test metrics.

  • Training-time samples: On linear regression, the learned reverse-process samples match the forward-process samples across selected timesteps.The visualization uses t = 1, 200, 400, 600, 800, T.
  • Multi-modality: On the full-circle task, CARD captures the transition from uni-modal to multi-modal distributions as the timestep increases.The comparison focuses on whether q and p samples match at selected timesteps.
  • Coverage: A full-circle run obtains QICE 0.62, with sample coverage close to optimal across bins despite the central gap of the bi-modal distribution.The largest deviation occurs in the fifth bin and is offset by higher coverage in other bins.
  • Test-time samples: On UCI Boston, generated p samples mix well with q samples at intermediate and late timesteps, while the t = 1 samples differ visibly.The figure generates 1000 samples for each covariate value.
  • Metric evolution: On UCI Boston, RMSE, NLL, and QICE steadily improve as reverse diffusion proceeds, while PICP remains near its optimal 0.95 coverage.The metrics average across 20 dataset splits.
  • Confidence assessment: The paper’s calibration discussion contrasts subgroup-level ECE with finer instance-level confidence assessment.Finite-sample ECE approximates calibration by partitioning predicted probabilities into equal-width bins.
  • Model flexibility: CARD’s diffusion framework is presented as flexible because it does not impose a parametric form on the conditional response distribution.The paper contrasts this with explicit distributional assumptions used by BNN approaches.

B Broader Impact and Limitations

CARD is motivated by practical decision-making applications involving complicated, unknown real-world response distributions. The paper also identifies safety, modeling-scope, and computational-efficiency considerations for future work.

  • Broader impact: CARD is intended for industrial supervised-learning settings where real-world response distributions are complicated and unknown.Examples include relating drug dosage to patient biographical features and evaluating player performance from on-court measurements.
  • Safety concerns: CARD’s ability to model multi-modality could enable reverse engineering that reveals personalized information, such as a patient’s gender.The paper explicitly raises this as a potential malicious use of prediction results.
  • Limitations and future work: The study restricts CARD to Gaussian diffusion, reverse denoising, and a deterministic pre-trained neural network, while other modeling choices remain unexplored.The paper mentions discrete diffusion, cross-entropy objectives, Bayesian neural networks, and larger-scale efficiency as possible extensions.
  • Computational considerations: Experiments use a single Nvidia GeForce RTX 3090 GPU, and the paper reports training and sampling throughput for selected regression and classification settings.Reported settings include 100–200 training steps per second for regression and 44 steps per second for CIFAR-10 classification at batch size 256.
  • Computational considerations: CARD’s computational complexity is evaluated through model parameter size and throughput, with separate accounting for its diffusion and pre-trained prior-mean models.The comparison includes deterministic neural networks across different datasets, with UCI measurements taken on the subset having the largest data dimension.
Loading 2206.07275v4…