PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Jiahao Yu, Song Liu, José Miguel Hernández-Lobato, RuiKang OuYang

arXiv:2610.10021v1stat.MLcs.LG

TL;DR

Implicit generative models make MI difficult to evaluate, while singular output laws can make score-based gradient construction problematic. The paper introduces SMI, which applies a common spreading kernel and aggregates MI across noise levels. It reports strong synthetic dependence control and competitive performance on real-world tasks.

  • Problem

    In implicit generators, intractable densities complicate MI evaluation, and singular conditional distributions may yield infinite MI or undefined scores.

    Source: §1 · p. 1

  • Method

    SMI applies a common spreading kernel to the generated variable under both joint and product-of-marginals laws, integrates across noise levels, and estimates gradients using a classifier-learned density ratio.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    SMI shows strong synthetic optimization and competitive real-world performance, and under suitable conditions vanishes if and only if the variables are independent.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Gaussian spreading yields smooth, positive conditional and marginal densities even when the original distributions are singular.

    Source: §3 · p. 3

    Small SMI need not imply small MI, and improved dependence control need not translate into better task performance.

    Source: §6 · p. 8

Abstract

from arXiv · show

Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.

1 INTRODUCTION

The paper studies minimizing or maximizing dependence in implicit generators, where MI is difficult to evaluate and may provide uninformative gradients. It introduces SMI and a classifier-based gradient estimator, with theoretical and experimental support for dependence control.

1 INTRODUCTION

  • In implicit models, intractable densities complicate MI evaluation, and singular conditional laws can yield infinite MI or undefined scores.Even finite MI can be insensitive to support geometry: in the example, it remains log 2 for every a > 0 although the supporting lines approach each other.Source: §1 · p. 1
  • SMI measures dependence after applying the same noise channel to the generated variable under the joint and product-of-marginals laws, then integrates across noise levels.Gaussian spreading gives smooth, positive conditional and marginal densities and makes the example's information about Y sensitive to the distance between the supports.Source: §1 · p. 1
  • Under suitable kernel and weighting conditions, SMI vanishes if and only if the variables are independent.The classifier-based pathwise estimator targets the generator gradient through the difference between conditional and marginal scores, estimated from a learned log density ratio.Source: §1 · p. 2
  • SMI addresses singularity and poor overlap in MI optimization, and experiments show strong synthetic optimization and competitive real-world performance.The paper also characterizes independence and information dissipation under spreading and bounds gradient-sampling and score-difference estimation errors.Source: §1 · pp. 1–2

2 BACKGROUND

The background sets up MI optimization for conditional implicit generators and explains why gradient estimation, rather than direct density evaluation, is used. It motivates SMI because standard MI gradients rely on scores that may not exist for singular output laws.

2 BACKGROUND

  • The generator setup conditions output X on an input Y whose distribution does not depend on generator parameters θ.When densities exist, the conditional law and input distribution induce the joint and marginal densities used in the MI objective.Source: §2 · p. 2
  • MI is the KL divergence between the induced joint law and the product of its marginals, and the objective may be maximized or minimized.The density expression is a special case of the KL definition, which also applies without densities and permits MI to be +∞.Source: §2 · p. 2
  • Small SMI does not in general imply small MI.This caveat distinguishes the spread objective from the unspread MI objective.Source: §2 · p. 2
  • Implicit generators permit sampling without tractable density evaluation, making direct evaluation of MI generally intractable.The conditional law is formed by pushing fixed noise through the generator; even when densities exist, evaluating the conditional or marginal density is generally intractable.Source: §2 · p. 2
  • MI optimization can proceed by estimating its gradient without evaluating MI itself, but this introduces score-estimation error in implicit models.The gradient identity uses the generator Jacobian and the difference between conditional and marginal scores; implicit output laws may be singular, so the required densities and scores need not exist.Source: §2 · p. 2

3 SPREAD MUTUAL INFORMATION

SMI applies a common spreading channel to the generated variable and measures dependence across noise levels. The section develops its gradient construction, classifier-based score-difference estimation, diffusion-kernel formulation, and use alongside task objectives.

3 SPREAD MUTUAL INFORMATION

  • Gaussian spreading yields smooth, strictly positive densities even for singular or disjoint-support laws, while preserving identifiability through the injectivity of scaling and Gaussian convolution.The paper defines SMI by applying the same channel to X under the joint and product-of-marginals distributions, leaving Y unchanged; SMI is nonnegative, bounded above by MI, and detects independence exactly when spreading is injective.Source: §3 · p. 3

3.1 Gradients of SMI

  • At any positive Gaussian noise level, spreading makes the conditional and marginal scores well-defined, and relative gradient variance remains bounded near the deterministic limit where unspread MI diverges.For fixed σ > 0, the stated bound is 2 + a^2/σ^2, whereas the expression diverges as b → 0.Source: §3.1 · pp. 3–4

3.2 Estimating SMI gradients through density ratios

  • The score difference for the SMI gradient is estimated by differentiating a learned log density ratio from a classifier distinguishing joint samples from independently paired samples.The classifier is conditioned on the noise level; Gaussian spreading supplies positive densities for this construction.Source: §3.2 · p. 4

3.3 SMI with diffusion kernels

  • The diffusion-kernel formulation averages dependence across noise levels, with the noise schedule and nonnegative weights determining which scales contribute.Small effective noise retains fine-scale dependence, while larger noise measures dependence that persists under stronger perturbation; positive noise gives smooth, positive densities under the stated assumptions.Source: §3.3 · p. 4

3.4 Optimising SMI in practice

  • SMI controls dependence but not the output distribution, so the paper uses it as a regularizer alongside a task-specific objective.The sign of β selects minimization or maximization, and |β| controls the strength of dependence control.Source: §3.4 · pp. 4–5

4 THEORETICAL ANALYSIS

The theoretical analysis studies Gaussian-spread SMI’s regularity and bounds on gradient sampling and estimation errors. It also examines how classifier estimation error affects gradient estimates.

4.1 Regularity and gradient moments

  • Gaussian spreading gives conditional and marginal distributions smooth, positive densities with common support and finite SMI, even when X0 has no density.For α_t ≠ 0, zero SMI remains equivalent to independence; parameter differentiability still requires the assumptions of Prop. 3.3.Source: §4.1 · p. 5
  • At each fixed positive noise level, the exact-score SMI gradient estimator has finite second moment under a bounded generator Jacobian, and B samples yield O(B^-1) mean-squared sampling error.The second-moment bound scales linearly with signal-to-noise ratio α_t^2.Source: §4.1 · p. 5
  • An integrability criterion guides the choice of time distribution and weights for estimation across noise levels, particularly near zero noise.The weighted estimator bound requires a finite right-hand side and interchangeability of differentiation with the time expectation.Source: §4.1 · p. 5

4.2 Effect of classifier estimation error

  • Differentiated classifier log-odds approximate the score difference, and the error bound separates O(B^-1) sampling error from a squared-bias term controlled by δ_t^2.For randomly fitted classifiers and generators, the statements hold conditionally on their fitted state using fresh evaluation samples.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

The experiments compare SMI with MI-based methods and, where appropriate, task-specific baselines across synthetic and real-world dependence-minimisation and maximisation tasks.

Gradient accuracy.

  • SMI has lower mean relative errors than separately fitted DSM scores for both score differences and parameter gradients.The errors are pooled over noise levels and reported as means ± SD over ten seeds.Source: Table 1 · p. 6, Gradient accuracy · p. 6

MI optimisation

  • SMI reaches the 50-nat maximisation target in every run without numerical failure and keeps MI below 10^-3 throughout the final 1,000 minimisation updates in all runs.At d = 100 and 200, the best baseline meets the minimisation criterion in 50% and 48% of runs, respectively.Source: MI optimisation · p. 6, Figure 2 · p. 7

Support geometry.

  • For parallel conditional supports separated by 2a, SMI reaches 2a < 0.1 in all ten runs, versus four of ten without spreading.Without spreading, MI remains log 2 for every a > 0, so its gradient cannot guide contraction; the four successful unspread runs reach the target sooner.Source: Support geometry · p. 6, Figure 2 · p. 7

5.2 Missing Data Imputation

  • SMI has the lowest mean MMD2 rank among matched MI methods at every missingness rate and among all methods at 80%.HyperImpute and MissForest achieve better mean RMSE ranks, consistent with objectives more directly aligned with pointwise reconstruction; failed runs rank last.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Nuisance Suppression on Waterbirds

  • SMI has the lowest mean background leakage among MI methods under both probes, the lowest mean MLP leakage overall, and worst-group accuracy comparable to FLO.HSIC has higher mean worst-group accuracy and lower linear leakage but higher MLP leakage; conditional adversarial training has higher worst-group accuracy but higher leakage under both probes.Source: §5.3 · p. 7, Table 3 · p. 8

5.4 Edges-to-Photos Translation

  • SMI achieves the highest mean latent-code prediction R2 among MI methods in all settings and among all methods under JPEG compression and resizing.Its higher KID indicates a trade-off with photo-distribution matching.Source: Table 4 · p. 8, §5.4 · p. 8

6 CONCLUSION

The conclusion presents SMI as a way to mitigate singularity and poor overlap in MI optimisation, with theoretical analysis and experiments supporting its use. It also states the authors’ use and review of generative AI tools.

6 CONCLUSION

  • SMI mitigates singularity and poor overlap in MI optimisation; analysis establishes regularity and gradient-error bounds, while experiments show strong synthetic optimisation and competitive real-world performance.Under injective spreading, zero SMI is equivalent to independence, but small SMI need not imply small MI.Source: §6 · p. 8
  • The conclusion identifies principled noise selection, derivative-aware classifier training, and task-specific trade-offs as future work.Source: §6 · p. 8

AI use statement

  • The authors report using generative AI tools for proof and methodology work, experiments, implementation, interpretation, and information sourcing.They state that they reviewed all AI-assisted work and checked LLM-generated proofs and code.Source: AI use statement · p. 9

Controlling Dependence in Implicit Generative Models via Spread Mutual Information: Supplementary Materials

The supplementary analysis relates the spread-score difference to Gaussian noise and compares gradient-estimator precision with standard MI in a binary Gaussian model. In that model, greater effective noise reduces relative estimation error at fixed minibatch size.

A ADDITIONAL THEORETICAL ANALYSIS

  • Under the stated Gaussian-noise setup, the exact score difference is the gradient of the log ratio between spread conditional and marginal distributions.The setup defines Xt = αtX0 + σtϵ with independent standard Gaussian noise and positive αt, σt.Source: A ADDITIONAL THEORETICAL ANALYSIS · p. 12

A.1 A comparison with standard MI

  • The comparison considers Y uniform on {−1, +1} and X0 = aY + bZ, with a > 0 and fixed b > 0.The analysis uses a binary Gaussian model and the posterior mean of the binary input to a Gaussian channel.Source: §A.1 · pp. 12–13
  • In this model, spreading makes relative gradient variance nonincreasing in effective noise and no larger than for the standard-MI estimator; minibatch relative mean-square error is (1/mt − 1)/B.As effective noise increases, mt increases and relative estimation error decreases at fixed minibatch size; the estimated gradient is unbiased for the smoothed objective when exact scores are used.Source: §A.1 · p. 13

A.2 Information across noise levels

The supplementary analysis describes how Gaussian spreading changes information across noise levels and how the effective-noise formulation is used in the proofs. SMI decreases with effective noise, while its scale sensitivity can provide an optimisation signal.

A.2 Information across noise levels

  • With Gaussian spreading, SMI is nonincreasing as effective noise variance increases, approaches Iθ(X0; Y) as noise tends to zero, and approaches zero as noise tends to infinity.The zero-noise limit can be infinite.Source: §A.2 · p. 13
  • For X0 = aY with binary Y, standard MI is constant for a > 0, whereas SMI decreases smoothly as a approaches zero at fixed positive noise.The noise schedule determines the scale at which conditional outputs are distinguished.Source: §A.2 · p. 13

B PROOFS

  • The proofs establish Gaussian results under finite second moment, positive differentiable noise parameters, and independent standard Gaussian noise; parameter-gradient results additionally require regularity assumptions.The appendix says it proves the main theoretical results and gives additional results on classifier error and information dissipation.Source: B PROOFS · p. 13

B.1 Proof of the effective noise variance

  • Because αt is positive, Xt and its rescaled form Ut are deterministic functions of each other, so the proof applies data processing without changing their information.Source: §B.1 · p. 14

B.2 Proof of the data processing and identifiability

The proof bounds spread MI between zero and ordinary MI, then uses injectivity of spreading to show that zero spread MI is equivalent to independence.

B.2 Proof of the data processing and identifiability

  • Spread MI is nonnegative and no greater than MI, with the upper bound understood in the extended-real sense.Source: §B.2 · p. 14
  • If X and Y are independent, spreading only X preserves zero spread MI by the data-processing bound.Source: §B.2 · p. 14
  • Conversely, zero spread MI implies independence when the spreading operation is injective.Zero spread MI gives equality of the spread conditional and marginal distributions; injectivity then yields equality of the original distributions.Source: §B.2 · p. 14

B.3 Proof of the pathwise SMI gradient

The proofs establish the pathwise SMI gradient and classifier density-ratio identity, then derive regularity, gradient-error, and noise-level comparison results under their stated assumptions.

B.3 Proof of the pathwise SMI gradient

  • Under stated differentiability, positivity, integrability, and interchange assumptions, the proof applies the pathwise chain rule to SMI.Source: §B.3 · pp. 14–15

B.4 Classifier optimum and product-pair sampling

  • Balanced population cross-entropy has a unique pointwise optimum whose log odds equal the log density ratio; its input gradient supplies the score-difference field.Product-pair training reuses the same spread observation with an independent Y, and generator updates freeze the fitted classifier while using its input gradient.Source: §B.4 · p. 15

B.5 Smooth densities and finite SMI

  • At positive Gaussian noise, spread densities are smooth and positive and SMI is finite without requiring an input density; nonzero α preserves the independence equivalence.Source: §B.5 · pp. 15–16

B.6 Posterior identities and spatial regularity

  • Posterior conditional-mean and covariance identities represent score differences and bound spatial regularity; the stated spatial bounds require ∥X0∥≤R.Source: §B.6 · pp. 16–17

B.7 Proof of the gradient moment bounds

  • Conditional-expectation orthogonality and Jensen establish unbiased gradient estimates, with independent minibatch averaging yielding variance bounds for fixed and sampled noise levels.Source: §B.7 · pp. 17–18

B.8 Proof of the binary Gaussian comparison

  • In the binary Gaussian comparison, relative MSE is compared with standard MI, while expected gradient tends to zero as noise grows.Source: §B.8 · p. 18

B.9 Proofs for classifier estimation error

The classifier-error analysis shows that low classification risk need not ensure an accurate gradient. The information-dissipation results characterize how spread MI changes with noise and its limiting values.

B.9 Proofs for classifier estimation error

  • Under its stated assumptions, the classifier-error result extends to the weighted objective across sampled noise levels.Source: §B.9 · p. 19
  • Vanishing excess population cross-entropy need not control mean-square error in the input gradient of log odds, even for identical smooth positive joint and product laws.Source: §B.9 · p. 19

B.10 Information dissipation and limiting behavior

  • Spread MI is nonincreasing as the noise-to-signal ratio grows and strictly decreases when X0 and Y are dependent wherever that ratio increases.Source: §B.10 · p. 20
  • As noise vanishes, spread MI converges to unperturbed MI; as noise diverges, it tends to zero, and the information-gap identity applies when unperturbed MI is finite.Source: §B.10 · pp. 20–21

B.11 Independence and the scale dependence of weighted SMI

Weighted SMI is zero exactly for independent inputs under the stated weighting conditions, but small SMI need not mean small standard MI. The experiments examine gradient estimation, MI optimisation, and noise-level identities.

B.11 Independence and the scale dependence of weighted SMI

  • Small SMI does not uniformly imply small standard MI: for X0=aY, standard MI remains log 2 for a>0 even as positive-noise MI tends to zero when a→0.Source: §B.11 · p. 21
  • A zero weighted SMI is equivalent to X0 being independent of Y when positive weight falls on a nonzero-signal noise level.Source: §B.11 · p. 21

C EXPERIMENTAL DETAILS

  • Across tasks, matched MI methods share generator or representation architectures and update budgets, differing in estimator heads and gradient construction.Source: C EXPERIMENTAL DETAILS · p. 22

C.1.1 Support Geometry

  • The support-geometry experiment uses parallel conditional lines separated by 2a; standard MI is log 2 for every a>0, while independence occurs at a=0.Source: §C.1.1 · p. 22

C.1.2 Gradient Accuracy

  • The gradient-accuracy experiment compares a scale-conditioned log-ratio gradient with separately DSM-fitted conditional and marginal scores on shared held-out noisy samples.Source: §C.1.2 · p. 23

C.1.3 MI Optimisation

  • The MI-optimisation benchmarks span five data families, three dimensions, two parameterisations, and ten runs, with outcomes also reported by family and dimension.Source: §C.1.3 · p. 24

C.1.4 Information Across Noise Levels

  • All tested differential and integrated identities agree to absolute error below 10^-7; as noise contracts, weighted SMI approaches finite unperturbed MI or diverges for perfectly correlated Gaussian inputs.Source: Figure 10 · p. 26, §C.1.4 · p. 26

C.2 Missing Data Imputation

This section evaluates dependence objectives on missing-data imputation, nuisance suppression in Waterbirds classification, and latent-code retention in image translation using task-specific metrics. It also reports additional Edges2Shoes metrics and computational costs, alongside analytical complexity bounds for the evaluated methods.

C.2 Missing Data Imputation

  • For imputation, the objective minimizes dependence between completed data and the observed-entry mask while holding observed entries fixed.The study uses all rows from ten TabArena datasets in transductive imputation, with nested 30%, 60%, and 80% MCAR masks shared across methods.Source: §C.2 · p. 27
  • Imputation is evaluated by missing-cell pointwise error and biased empirical squared MMD between completed rows, with results reported across methods, datasets, and missingness rates.Figure 11 shows individual runs and mean ± SD only for groups with three successful fits; incomplete groups retain finite runs and mark failures.Source: §C.2 · pp. 28–29, Figure 11 · p. 29

C.3 Nuisance Suppression on Waterbirds

  • For Waterbirds, MI-based methods minimize dependence between the bird representation and background given bird class, while SMI replaces MI with its multiscale spread counterpart.SMI adds independent Gaussian noise at sampled scales while leaving labels unchanged.Source: §C.3 · pp. 29–31
  • Waterbirds evaluation measures worst-group test accuracy and probe-based background leakage, and Table 7 adds overall accuracy to the comparison.The table caption specifies test accuracy and background leakage as mean ± SD over five training runs.Source: §C.3 · p. 33, Table 7 · p. 34

C.4 Edges-to-Photos Translation

  • On Edges2Shoes, the objective encourages generated image features to retain information about the latent code, but unperturbed mutual information may be infinite for deterministic generation with continuous codes.The study therefore evaluates latent predictability and image-distribution matching separately; SMI spreads the generated feature vector.Source: §C.4 · pp. 33–34
  • Additional Edges2Shoes results report KID, edge F1, and latent-code prediction using ResNet-50 features at 20,000 generator updates.Reporting-set metrics are summarized as mean ± SD over three training runs.Source: §C.4 · p. 36, Table 8 · p. 36

C.5 Computational Costs

  • The computational-cost results report iteration time and peak allocated GPU memory across sweeps of dimension, minibatch size, and correlated coordinates, with costs specific to hardware and implementation.Measurements include sampling and generator backpropagation, and no measurements are excluded.Source: §C.5 · p. 37, Figure 12 · p. 37, §C.5.1 · p. 37

C.5.1 Analytical Costs

  • SMI requires O((Naux + 1)B(dh + Nℓh2)) estimator work per generator update, with peak memory from network/optimizer state and batch activations and score vectors.One reverse pass obtains the input gradients, which are stopped to avoid higher-order differentiation; sampling one noise scale per example avoids a discretized-schedule factor.Source: §C.5.1 · p. 38, Table 9 · p. 38

Related papers

Loading 2610.10021v1…