Deep Sufficient Representation Learning via Mutual Information

• MSRL learns nonlinear sufficient representations by optimizing a mutual-information objective with deep neural networks and an adversarial training procedure.It supports multidimensional continuous responses and can be adapted to categorical responses.Source: §1 · pp. 1–2
• Unlike information bottleneck, MSRL constrains the representation to follow a user-specified distribution rather than limiting mutual information between predictor and representation.The paper formulates this as a minimax problem using a variational f-divergence formulation.Source: §1 · p. 2
• The authors establish consistency and non-asymptotic error bounds under suitable conditions, deriving a generalized Dudley inequality for an order-two U-process indexed by deep networks.The inequality uses Rademacher chaos and may also be useful in other U-process problems.Source: §1 · p. 2
• The paper reports that MSRL outperforms some existing nonlinear sufficient dimension reduction methods and discusses categorical responses and intrinsic-dimension determination.The paper also describes the training algorithm and the organization of its numerical evaluation.Source: §1 · pp. 2–3

2. f-mutual information and sufficient representations.

• For strictly convex f and positive relevant densities, equality in the f-mutual-information data-processing inequality characterizes conditional independence given a representation, which defines sufficiency here.The method seeks a uniformized sufficient representation; its intrinsic dimension d0 is the smallest dimension among such representations.Source: §2 · pp. 3–4

2.1. Population objective function based on mutual informaiton.

• The population objective rewards mutual information between response and representation and penalizes deviation from a uniform reference distribution, with the target representation described as a minimizer.The paper focuses on the KL-divergence case.Source: §2.1 · p. 4

2.2. Deep neural networks.

• The method approximates the representation, mutual-information discriminator, and push-forward functions with feedforward neural networks using ReLU activations.The network classes are specified by input and output dimensions and architectural parameters such as depth, width, and size.Source: §2.2 · pp. 5–6

2.3. Empirical objective function.

• In the synthetic example, MSRL features make response signs more linearly separable than GSIR and GSAVE, while the learned representation follows the selected reference distribution.Figure 1 encodes negative and positive responses in red and blue, respectively; Figure 2 compares component density estimates with the reference distribution.Source: §2.3 · p. 6, Figure 1 · p. 7, Figure 2 · p. 7

3.1. Consistency.

• Under the stated assumptions and network-size conditions, the learned representation’s conditional density converges to that given the predictor, establishing asymptotic sufficiency.The consistency analysis focuses on continuous predictors and responses and assumes the intrinsic dimension is known.Source: §3 · p. 7, §3.1 · pp. 8–9
• The consistency result can converge arbitrarily slowly because approximating a very general continuous-function class is difficult.The paper attributes the slow rate to the breadth of the continuous-function class.Source: §3.1 · p. 9

3.2. Error bound.

• Under Hölder continuity assumptions, the paper establishes non-asymptotic excess-risk and density error bounds with prefactors that depend polynomially on relevant dimensions.The faster-rate analysis uses stronger smoothness assumptions on the relevant function classes.Source: §3.2 · pp. 9–10

4.1. Error decomposition.

• The error decomposition reduces the proof of the main bound to controlling separate terms, including approximation errors and stochastic errors.The stochastic terms include an empirical-process term and an order-two U-process term.Source: §4.1 · pp. 11–12
• Classic Dudley’s inequality does not apply to the order-two U-process term, so the authors derive a generalized inequality using Rademacher chaos.The new inequality is presented as potentially useful beyond this proof.Source: §4.1 · p. 12

4.2. A maximal inequality for U-process of order 2.

• The derived inequality bounds the expected supremum of a class of symmetric, real-valued functions indexed by pairs of observations.The inequality is stated for an order-two U-process.Source: §4.2 · p. 12
• The maximal inequality uses uniform covering numbers, a feature the authors link naturally to neural networks.The inequality’s integral upper bound can be tighter than the comparison bound discussed in the section.Source: §4.2 · pp. 12–13

• For a finite categorical response, the method uses a K-output discriminator whose outputs have separate parameters, and recasts the objective for the MSRL training algorithm.The categorical response is represented through one-hot vectors in the formulation.Source: §5 · p. 14
• The method extends to mixed categorical and continuous predictors and responses, targeting sufficiency conditional on the categorical predictor component and the learned representation of the continuous component.The formulation assumes finite categories for the categorical components.Source: §5 · p. 14
• The authors state that the theoretical analysis can be applied to these categorical cases without further difficulties.Source: §5 · p. 14

6. Implementation.

• For larger minibatches, the training procedure approximates the U-process term using permuted samples and uses distance covariance or correlation for early stopping.The stated examples of relatively large batch sizes are 256 and 512.Source: §6 · p. 14

6.1. Determination of the intrinsic dimension.

• In the real-data examples, the authors select intrinsic dimensions 5 and 10 by cross-validation in terms of APE.Source: §6.1 · p. 14
• The proposed dichotomy-based cross-validation method selects the intrinsic dimension by screening only ⌈log2 dU⌉ alternatives when starting from an upper bound dU.When predictive performance is the priority, the paper suggests selecting the dimension by cross-validation in terms of APE.Source: §6.1 · pp. 14–15

7.1. Simulation Study.

• MSRL has larger distance correlation than competitors in almost all p = 10 simulation cases, and lower APE when predictor distributions are sophisticated.SIR and SAVE perform best for the simple linear model but lose DC and APE on nonlinear models; GSIR and GSAVE have the best APE under simple predictor distributions, where MSRL is comparable.Source: §7.1 · p. 17, Table 1 · p. 17

7.2. The superconductivity dataset.

• On superconductivity data, MSRL outperforms other methods in empirical DC and prediction accuracy, while SIR exceeds GSIR and GSAVE.With d0 = 5, the critical temperature is nearly linear in three MSRL representation components; only the first directions of SIR and GSIR show near-linearity among the other SDR methods.Source: §7.2 · p. 18, Table 2 · p. 18, Figure 3 · p. 18

7.3. The Pole-Telecommunication dataset.

• On Pole-Telecommunication data, MSRL produces better results, while GSIR and GSAVE outperform linear SDR competitors and GSIR performs best among them.For classification at response threshold 50, each pair of MSRL components is informative, whereas only the first directions of GSIR and GSAVE are predictive.Source: §7.3 · p. 19, Table 3 · p. 19

7.4. Intrinsic dimension determination.

• In the toy intrinsic-dimension study with true d0 = 2, the proposed method works well and never estimates d0 as 1.An estimate of 1 would produce an insufficient representation.Source: §7.4 · p. 19, Table 4 · p. 19

8. Conclusion.

• Under reasonable conditions on the target representation and neural network classes, the authors establish MSRL consistency and derive non-asymptotic error bounds.Source: §8 · pp. 19–20
• The paper discusses categorical-response extensions and proposes dichotomy-based cross-validation to determine intrinsic dimension.Source: §8 · p. 20
• With relatively large samples, MSRL outperforms GSIR and GSAVE; with n = 300 or 500, it performs similarly, while remaining computationally feasible when kernel methods become prohibitive.The kernel-based methods involve an n × n Gram matrix.Source: §8 · p. 20
• This work considers only the Kullback-Leibler mutual information criterion; studying other divergence-based criteria is left for future work.Source: §8 · p. 20

APPENDIX A: PROOF OF LEMMA 2.1

• The proof of Lemma 2.1 uses the Markov chain Y → X → g(X) to reduce the result to Lemma A.3.Source: Appendix A · p. 21
• For distributions sharing the same conditional density, the joint f-divergence equals the f-divergence between their marginals.Source: Appendix A · p. 21
• Marginalization cannot increase f-divergence, as stated in Lemma A.2.Source: Appendix A · p. 21

APPENDIX B: PROOF OF MAIN RESULTS

• Theorem 2.1’s proof starts from R0 being a d0-dimensional USR and invokes data processing and nonnegativity of KL divergence.The appendix then states that the proof is complete.Source: Appendix B · p. 24
• Theorem 3.1 is proved using Lemma 4.1; Pinsker’s inequality then yields bounds used to establish Corollary 1.The proof considers the L1 difference between conditional response densities given X and R, as well as the L1 distance of pR from 1.Source: Appendix B · p. 25
• Theorem 3.2 and Corollary 2 follow by bounding the error decomposition, with (3.6) and (3.7) obtained as corollaries of (S.4).The proof states that the constant in the bound depends only on B2 and β.Source: Appendix B · p. 25, §3 · p. 26

1. Using part (b) of Lemma B.1,

• LMI(D,R*) ≤ 72(1 + e^BD)C1,β(dY + d0,B2)n^(-β/(2β + dY + d0)) is the bound obtained using part (b) of Lemma B.2.The constant C1,β(d,a) is defined in the appendix using c1 and c2 from (3.2).Source: §1 · p. 25

3. Using part (b) of Lemma B.3,

• |n(D,R) − LMI(D,R)| ≤ CC2,β((dY + d0) ∨ dX,BD)n^(-β/(2β + (dY + d0) ∨ dX)) under part (b) of Lemma B.3.The passage defines C2,β(d,a) = (1 + a)(⌊β⌋ + 1)^(9/2)d^(⌊β⌋+2).Source: §3 · pp. 26, 28
• A ReLU-network approximation satisfies E∥R0(X) − R̄n(X)∥ ≤ 18(d0dX)^(1/2)max_i ωR0,i(n^(-1/(2 + dX))), which tends to zero as n grows.The proof constructs the approximation under the network-parameter condition (NS1).Source: §3 · p. 33
• Under Assumption 3, applying Theorem 3.3 in Jiao et al. (2021) with N = 1 yields an R̄_n ∈ R satisfying (NS1), completing Lemma B.5.Source: §3 · p. 34

APPENDIX C: SUPPORTING LEMMAS

• Lemma C.1 states that an absolutely continuous probability measure with finite second moment admits a unique optimal transport map to the standard Gaussian, injective almost everywhere.The map pushes the measure forward to N(0,I_d).Source: Appendix C · p. 34
• Lemma 4.3’s first inequality follows from symmetrization, while its second is proved by chaining dyadic covers and controlling empirical Rademacher chaos across successive covers.The chaining argument uses finite covers at decreasing radii and applies Lemma C.4 to the successive-cover function classes.Source: Appendix C · pp. 37–39
• Lemma C.9 shows that D∘R and Q∘R are ReLU networks, with widths max{W_D,W_R+2d_Y,2(d_Y+d_0)} and max{W_Q,W_R,2d_0}, respectively.Their respective depths are L_D+L_R+1 and L_Q+L_R+1.Source: Appendix C · p. 39

APPENDIX D: ADDITIONAL NUMERICAL STUDIES AND EXPERIMENT DETAILS

• Table S.1 reports distance correlation and average prediction errors with standard errors for p = 30.The standard errors are based on six-fold validation and hyperparameters selected by validation.Source: Table S.1 · p. 39, Appendix D · p. 39

D.1. Experimental Details.

• Across experiments, training uses λ = 2, Adam in PyTorch, LeakyReLU activations, and batch size 512.Source: §D.1 · p. 40

D.2. Simulation for Model (2.10).

• For Model (2.10), training uses 4,000 samples, no early stopping, and at most 3,000 epochs; network widths differ by reference distribution.The representation network has two hidden layers for γU and three for γS; D and Q each have two hidden layers.Source: §D.2 · p. 40

D.3. Simulation Study in Subsection 7.1.

• The Subsection 7.1 study splits 6,000 observations into 4,000 training, 1,000 validation, and 1,000 test cases.MSRL early stopping uses validation distance correlation; ten initializations are compared, and the best validation result is retained.Source: §D.3 · p. 40
• In Subsection 7.1, SIR and SAVE select slice number by validation distance correlation, while GSIR and GSAVE use the tuning approach of Li (2018).Source: §D.3 · p. 40

• For each data split, 20% was reserved for testing; 3,000 points from the remaining 80% were randomly selected for validation.Source: §D.4 · p. 40
• GSIR and GSAVE used subsamples of 4,000 training, 2,000 validation, and 3,000 testing points because of computational limits.Source: §D.4 · p. 40
• MSRL used different network widths and training settings for superconductivity and pole-telecommunication, with 10 initializations for each dataset.For visualization, the study sampled 800 superconductivity test points and 400 pole-telecommunication test points from one fold.Source: §D.4 · p. 40

D.5. The extra plots for the superconductivity and pole-telecommunication datasets.

• For superconductivity, the plots relate critical temperature to each SIR or SAVE representation component at d0 = 5, with a fitted quadratic model.Source: §D.5 · p. 41, Figure S.1 · p. 41
• For pole-telecommunication, pairwise plots of SIR and SAVE representations at d0 = 5 distinguish responses Y ≥ 50 from Y < 50 by color.Source: §D.5 · p. 41, Figure S.2 · p. 41

D.6. Intrinsic Dimensionality Determination of Model (7.1).

• The intrinsic-dimensionality experiment uses 8,000 points, split into 4,000 training, 2,000 validation, and 2,000 testing points.R has one 128-wide hidden layer; D and Q each have one 64-wide hidden layer.Source: §D.6 · p. 41