PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Optimal Representations with the Decodable Information Bottleneck

Yann Dubois, Douwe Kiela, David J. Schwab, Ramakrishna Vedantam

arXiv:2009.12789v2cs.LGcs.ITmath.ITstat.ML

TL;DR

The paper asks how to characterize and learn representations that generalize for a specified predictive family, rather than merely compress inputs while retaining decoder-agnostic information. It introduces DIB to enforce family-relative sufficiency and minimality, proving optimality and learning guarantees. Empirically, the framework improves downstream classifier performance and robustness and predicts neural-network generalization.

  • Problem

    Existing Information Bottleneck representations are decoder-agnostic, while generalization depends on the predictive family V; standard IB also lacks optimality guarantees, is difficult to estimate, and is invariant to bijections.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    DIB learns representations that are sufficient and minimal relative to V, using information decodable by predictors in V rather than universal mutual information.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    DIB representations are optimal for expected test performance, can be learned with guarantees, improve downstream classifier performance and robustness, and correlate strongly with neural-network generalization.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Representation quality should be evaluated relative to the predictive family of interest, because DIB’s family-aware representations support generalization across downstream classifiers and neural networks.

    Source: §6 · p. 9

    IB’s bijection invariance can treat nonlinearly entangled and linearly separable representations as equally good even when classification uses logistic regression.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

We address the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottleneck, which compresses the inputs while retaining information about the targets, in a decoder-agnostic fashion. In machine learning, however, our goal is not compression but rather generalization, which is intimately linked to the predictive family or decoder of interest (e.g. linear classifier). We propose the Decodable Information Bottleneck (DIB) that considers information retention and compression from the perspective of the desired predictive family. As a result, DIB gives rise to representations that are optimal in terms of expected test performance and can be estimated with guarantees. Empirically, we show that the framework can be used to enforce a small generalization gap on downstream classifiers and to predict the generalization ability of neural networks.

1 Introduction

The paper asks how to learn representations that make downstream predictors generalize, arguing that optimality should be defined relative to the predictive family V rather than decoder-agnostic compression. It introduces DIB, which targets V-sufficiency and V-minimality, proves optimality with learning guarantees, and reports improved robustness and strong correlation with neural-network generalization.

Motivation

  • DIB reframes representation learning around enforcing generalization for empirical risk minimizers in a chosen predictive family V.This changes the problem from restricting V for a fixed representation to changing the representation so ERMs in V perform well.Source: §1 · pp. 1–2
  • Unlike J-ERM and standard IB, DIB accounts for the predictive family V when defining usable information and compression.J-ERM can favor non-generalizing classifiers, while IB may retain information that no predictor in V can decode and is difficult to estimate.Source: §1 · pp. 1–2

Contribution

  • DIB recovers representations that are V-sufficient and V-minimal, so predictors in V can predict labels but cannot distinguish examples sharing a label.Figure 1 contrasts J-ERM representations that permit non-generalizing ERMs with DIB representations that exclude them.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper proves that these representations are optimal: every downstream ERM in V reaches the best achievable test performance, and DIB can learn them with guarantees.The result is stated as a representation-level guarantee rather than only a guarantee for one selected classifier.Source: §1 · p. 2

Empirical evidence

  • Experiments show that DIB representations improve average and worst-case downstream-classifier performance and robustness.The introduction also reports that neural-network generalization is highly correlated with hidden-representation V-minimality across 562 models.Source: §1 · p. 2

2 Problem Statement and Background

The paper reframes representation learning as a game in which Bob learns representations for Alice’s chosen predictive family, seeking generalization rather than decoder-agnostic compression. It replaces mutual-information criteria with V-information, which measures information usable by predictors in V and is easier to estimate with guarantees.

2.1 Problem Set-Up: Representation Learning as a Two-Player Game

  • The setup treats Alice’s predictive family, task, and loss as fixed while Bob learns an encoder mapping inputs X to representations Z.Alice then evaluates empirical-risk-minimizing classifiers from V on Bob’s representation.Source: §2.1 · p. 2
  • The paper seeks representations that minimize Alice’s expected loss and an objective that can approximate the optimal encoder.The analysis assumes finite sample spaces, log loss, and mild conditions on V.Source: §2.1 · p. 3

2.2 Sufficiency, Minimality, and the Information Bottleneck (IB)

  • IB seeks representations informative about labels but minimally informative about inputs, yet lacks optimality guarantees, is difficult to estimate, and is invariant to bijections.That invariance means it does not favor simple decision boundaries for a constrained classifier family.Source: §2.2 · p. 3

2.3 V-information

  • The paper addresses IB’s decoder agnosticism by defining V-minimal V-sufficient representations through V-information, which is tailored to the predictive family of interest.V-information is introduced as the information decodable by predictors in V rather than by the universal family U.Source: §2.2 · p. 3, §2.3 · p. 3
  • Mutual information corresponds to the best possible probabilistic decoder, whereas V-information restricts decoding to the predictors that matter for the task.Thus label information that cannot be decoded by V is excluded from the criterion.Source: §2.3 · p. 3
  • V-information is non-negative, recovers mutual information when V = U, vanishes under independence, and inherits PAC estimation bounds based on V’s complexity.It corresponds to estimating predictive risk rather than Shannon information directly.Source: §2.3 · p. 3

3 Methods

The methods define representations that preserve all information decodable by V while removing V-decodable distinctions among examples sharing a label. They prove such representations are optimal for downstream ERMs and develop a DIB objective with finite-sample estimation guarantees.

3.1 V-Sufficiency and Best Achievable Performance

  • V-sufficient representations maximize V-information with labels and allow some predictor in V to attain the best achievable test risk.This differs from maximizing unconstrained mutual information when V cannot decode all label information.Source: §3.1 · p. 4

3.2 V-minimality and Generalization

  • V-minimal V-sufficient representations additionally minimize V-information about arbitrary label-preserving decompositions of the inputs.These decompositions test whether predictors can distinguish examples with the same underlying label.Source: §3.2 · p. 4
  • Under deterministic labels, every ERM in V trained on a V-minimal V-sufficient representation has expected test loss equal to the best achievable risk.The result assumes i.i.d. data and provides the paper’s generalization guarantee.Source: §3.2 · p. 4
  • V-minimality generalizes minimal sufficiency, recovers the unconstrained case, is monotone across predictive families, and always has at least one representation.The unconstrained case connects the framework to IB’s optimality when V = U.Source: §3.2 · p. 5

3.3 Practical Optimization and Estimation

  • DIB optimizes L_DIB = −I_V[Z → Y] + β I_V[Z → Dec(X, Y)], balancing label sufficiency against V-decodable label-preserving information.The second term creates a min-max optimization because its V-information includes an internal maximization over predictors.Source: §3.3 · p. 5
  • DIB can be estimated from finite samples using encoder samples and sampled decompositions, inheriting V-information estimation bounds.The estimate uses empirical expectations and K samples to approximate the decomposition average.Source: §3.3 · p. 5
  • A more complex V improves Alice’s best achievable prediction but makes DIB estimation harder because the error grows with V’s Rademacher complexity.This exposes a direct trade-off between predictive expressiveness and estimation difficulty.Source: §3.3 · p. 6
  • In neural networks, DIB uses one sufficiency head and K minimality heads with reversed encoder gradients, while β controls the minimality losses.The minimality heads classify arbitrary fixed labelings within classes using the same architecture V but different parameters.Source: Figure 2 · p. 5, §3.3 · p. 6

4 Experiments

The experiments show that matching the representation’s predictive family to Alice’s family improves achievable performance, while V-minimality supports generalization and predicts neural-network generalization.

4.1 V-sufficiency: Optimal Representations When the Data Distribution is Known

  • V-sufficiency is best when Bob’s family matches Alice’s family; larger or smaller families respectively make representations undecodable or unnecessarily constrained.The best performance occurs at V_Alice = V_Bob, while IB’s infinite-width sufficiency can be undesirable for constrained Alice classifiers.Source: §4.1 · p. 7

4.2 V-minimality: Optimal Representations for Generalization

  • Increasing β reduces V-information in the representation and monotonically shrinks the train-test gap, but beyond β = 10 it substantially reduces label information and achievable performance.Finite-sample DIB therefore exposes a trade-off between V-minimality for generalization and V-sufficiency for low loss.Source: §4.2 · p. 7, Figure 4 · p. 7
  • Removing decodable MNIST distractor information makes MNIST prediction harder and prevents Alice’s ERM from over-fitting spurious patterns.The encoder is trained on CIFAR10 while evaluation overlays MNIST digits as distractors.Source: §4.2 · pp. 7–8
  • DIB outperforms V+-DIB, VIB, and traditional regularizers in the reported representation comparisons, supporting V-minimality as a practical route to generalization.The comparison includes no regularization, stochastic representations, dropout, and weight decay; DIB performs slightly worse than dropout in standard classification settings.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Probing Generalization in Deep Learning

  • Across 562 CNNs, estimated V-minimality correlates with generalization and outperforms the compared measures based on entropy, path norm, gradient variance, and sharpness.Table 2 reports rank correlations with accuracy and log-loss generalization gaps.Source: §4.3 · pp. 8–9, Table 2 · p. 8
  • V-minimality is presented as a representation-level generalization measure that can be compared across architectures and datasets, with potential use in transfer learning.The measure uses a single internal representation rather than the full network architecture.Source: §4.3 · p. 9

5 Other Related Work

Related work spans generalized information, extended sufficiency and minimality, and kernel learning. The paper distinguishes its decoder-family formulation and optimality proof from these lines of research.

Generalized information, game theory and Bayes decision theory

  • Generalized-information work studies prediction rules and adversarial or robust formulations, whereas this paper adds a cooperative representation-learning agent optimizing the decision maker’s worst-case loss.Prior representation studies emphasized consistency rather than generalization.Source: §5 · p. 9

Extended sufficiency and minimality

  • Extended sufficiency and minimality work incorporates encoder families to characterize achievable representations, while this paper incorporates the decoder family to characterize optimal representations.The two perspectives are described as complementary.Source: §5 · p. 9

Kernel Learning

  • Kernel-learning methods implicitly learn representations through estimates or bounds of generalization error, but this work is not restricted to kernelizable predictors and provides an optimality proof.The stated distinction concerns both predictor-family scope and theoretical guarantee.Source: §5 · p. 9

6 Conclusion and Future Work

The paper develops predictive-family-aware representations that guarantee optimal downstream ERM performance and provides empirical evidence linking V-minimality to generalization. It concludes by identifying approximate V-minimality and deep-learning generalization as important directions for future work.

6 Conclusion and Future Work

  • V-minimal V-sufficient representations guarantee that downstream empirical risk minimizers achieve minimal expected test loss.The framework also supports learning these representations with PAC-style guarantees and improves image-classifier performance and robustness.Source: §6 · p. 9
  • Experiments suggest performance may be causally related to V-minimality, although the theory currently proves this only for perfect V-minimality.Deriving generalization bounds for approximate V-minimality remains an open question.Source: §6 · p. 9
  • The high correlation between neural-network generalization and V-minimality suggests that V-minimality may help explain generalization in deep learning.This conclusion is presented as a direction for further study rather than a completed theoretical result.Source: §6 · p. 9
  • The paper argues that information-theoretic analyses in machine learning should incorporate the predictive family V of interest.It specifically suggests this perspective could address issues involving mutual information in self-supervised learning and Information Bottleneck theory.Source: §6 · p. 9

Broader Impact

This material states the paper’s practical motivation, notation, and assumptions for finite multiclass classification and predictive families used in machine learning. The assumptions include complete class coverage, finite spaces, and structural properties satisfied by common classifiers.

Broader Impact

  • The proposed optimal representation is intended to make any training-fitting classifier generalize well, potentially reducing expensive hyperparameter search.The paper frames this as a possible contribution toward broader access to machine learning, without anticipating effects on particular groups.Source: Broader Impact · p. 10

A Notation

  • The notation uses X, Z, and Y for inputs, representations, and targets, while V denotes the predictive family and R(f, Z) the expected risk.Empirical risk and empirical risk minimizers are defined for an i.i.d. training set, and R*(V) denotes the best achievable risk.Source: A Notation · p. 18

B.1 Generic Assumptions

  • The analysis uses logarithmic loss, which is strictly proper and local, and may extend to other proper scoring rules through generalized predictive entropy.The paper emphasizes logarithmic loss because it is computationally attractive.Source: §B.1 · p. 18
  • The theorem setting assumes finite sample spaces, multiclass labels, at least one training example per class, and at least as many representation values as labels.The finite-space assumption is acknowledged as theoretically unsatisfactory, although it is compatible with computational machine learning.Source: §B.1 · pp. 18–19

B.2 Assumptions on Functional Families

  • Predictive families are assumed to be invariant to label permutations, provide non-empty label preimages, support arbitrary constant predictions, and allow monotonic biasing.These properties are stated to hold for practical families including neural networks, logistic regression, and decision trees.Source: §B.2 · p. 19

B.3 Assumptions for the Theorem

The appendix establishes the assumptions and propositions underlying the paper’s representation theory. It connects predictive sufficiency to V-information, characterizes minimal sufficient statistics, and shows that deterministic labels permit zero best achievable risk.

B.3 Assumptions for the Theorem

  • The theorem assumes deterministic labels Y = t(X), an assumption the paper says may not hold in the real world and should be investigated further.This assumption is used to simplify the proofs.Source: §B.3 · p. 19

C.1.1 Minimal Sufficient Statistics

  • For deterministic statistics, predictive sufficiency is equivalent to ordinary sufficiency, and minimal sufficiency is equivalent to minimality defined through retained input information.The equivalence of minimality does not extend to stochastic representations Z = T(X, ϵ).Source: §C.1.1 · p. 20

C.1.2 Replacing HV[Y | ∅] by H[Y]

  • Under the arbitrary-constant-prediction assumption, the conditional V-entropy at an empty representation equals the ordinary label entropy: HV[Y | ∅] = H[Y].The result uses properness of logarithmic loss and the existence of predictors that output any constant label distribution.Source: §C.1.2 · p. 21

C.2 V-Sufficiency

  • A representation is V-sufficient exactly when some predictor in V reaches the best achievable test risk using that representation.The appendix identifies HV[Y | Z] with the minimum risk attainable by V for Z.Source: §C.2 · p. 21

Z HV[Y | Z] Const. H[Y]

  • With deterministic labels, the best achievable risk over representations and predictors is zero for every predictive family V.The construction shifts classification into the encoder by mapping each input into representation buckets associated with its label.Source: Z HV[Y | Z] Const. H[Y] · p. 22

C.3 Theorem

The main theorem states that a V-minimal V-sufficient representation makes empirical risk minimization achieve the best possible test risk. This result holds under deterministic labeling and the paper’s stated assumptions on the predictive family and sample spaces.

C.3 Theorem

  • Any empirical risk minimizer using a V-minimal V-sufficient representation achieves the best achievable risk on any dataset.The theorem assumes deterministic labels, the functional-family conditions in Appendix B.2, and the generic assumptions in Appendix B.1.Source: §C.3 · p. 22

C.3.1 Lemmas for Theorem 1

The lemmas establish that V-sufficient representations separate deterministic labels and that V-minimality removes information about within-label decompositions. Together, these facts support the proof that zero-training-risk ERMs generalize under the theorem’s assumptions.

C.3.1 Lemmas for Theorem 1

  • V-sufficiency implies that representations associated with distinct deterministic labels have non-overlapping supports.If label-conditioned representation supports overlapped, no predictor could assign the correct label with certainty.Source: §C.3.1 · p. 23
  • For a V-sufficient representation, an ERM is equivalent to a predictor with zero empirical risk whenever some predictor achieves zero test risk.Nonnegative log loss makes zero the minimum attainable training risk in this setting.Source: §C.3.1 · p. 23
  • V-minimal V-sufficient representations are exactly those whose label-conditioned representations have zero V-information about every within-label decomposition of the inputs.The characterization follows because the overall nonnegative minimality objective reaches zero only when each decomposition term is zero.Source: §C.3.1 · p. 23

C.3.2 Proof Intuition

  • The proof uses a contrapositive: a zero-training-risk ERM with positive test risk would expose information about a train-versus-test selector in the representation.A constructed predictor g ∈V preserves prediction order while matching the selector’s marginal on correctly classified representations and improving prediction on misclassified test representations.Source: §C.3.2 · p. 24, §C.3.3 · p. 25

C.3.3 Formal Proof

  • Thus, any ERM with zero empirical risk must also have zero actual risk, because otherwise the representation would violate V-minimality.The binary argument extends to multiclass classification by reducing the predictor to two labels through monotonic biasing.Source: §C.3.3 · pp. 24, 26

C.4 V-minimal V-sufficient Properties

This section characterizes V-minimal V-sufficient representations, relates them across predictive families, and establishes PAC-style estimation guarantees for the empirical DIB objective.

C.4 V-minimal V-sufficient Properties

  • For the universal predictive family, minimality is equivalently independence from every within-label nuisance variable N ∈ Dec(X, y).This follows because universal information reduces to mutual information and vanishes exactly under independence.Source: §C.4 · pp. 27–28
  • Universal minimal sufficient representations recover ordinary minimal sufficient representations that are conditionally independent of the input given the label.The universal characterization is M_U = M ∩ Z, with minimality equivalent to Z ⊥ X|Y under deterministic labels.Source: §C.4 · pp. 26–27
  • V+-minimal V-sufficient representations are also V-minimal V-sufficient, and at least one universal minimal sufficient representation always exists.These are the monotonicity and existence properties of the representation sets under deterministic labeling.Source: §C.4 · p. 27

C.5 Estimation Bounds

  • The empirical DIB objective estimates population DIB using dataset expectations, encoder samples, and K sampled decompositions of each label-conditioned input space.The objective weights V-minimality by β and uses samples from TDec(X, y) to approximate the decomposition average.Source: §C.5 · pp. 28–29
  • With bounded log loss, empirical DIB inherits PAC-style estimation guarantees governed by the Rademacher complexity of log ◦ V.The result holds with probability at least 1 − δ under the stated boundedness and i.i.d. sampling assumptions.Source: §C.5 · p. 29

C.5.1 Lemmas for Estimation Bounds

  • The appendix notes that one displayed estimation bound is loose and not itself PAC-style, although tighter sub-Gaussian bounds could be derived.The looser form is retained because it is more succinct and sufficient for comparing DIB’s finite-sample estimation with LIB.Source: §C.5.1 · p. 31

D Reproducibility

The reproducibility appendix specifies the training setup, encoder and classifier procedures, dataset constructions, comparison regularizers, and correlation-study protocol used to evaluate DIB.

D Reproducibility

  • Experiments generally use 300 Adam epochs, learning rate 5e −5 with scheduled decay, batch size 256, no augmentation, and multiple random seeds.Models are checkpointed at the smallest training loss, with LeakyReLU activations and otherwise default PyTorch parameters.Source: D Reproducibility · p. 32

D.1 V-Minimality V-Sufficiency

  • Bob’s encoder normally parameterizes a diagonal Gaussian representation, samples one representation during training, and applies non-trainable batch normalization.The Gaussian choice improves comparability with VIB, while DIB itself does not require a Gaussian encoder.Source: §D.1 · p. 32
  • DIB trains jointly weighted V-sufficiency and V-minimality losses, with nuisance predictors built from label-conditioned decompositions and optimized through alternating encoder–head updates.The implementation uses base-|Y| nuisance labels, shared heads for GPU batching, and five optimization steps for the min-max interaction.Source: §D.1 · pp. 32–33

D.1.2 V-Minimality V-Sufficiency

  • For the main minimality experiments, the encoder is a roughly 21M-parameter MLP producing a 1024-dimensional representation, while Alice uses a one-hidden-layer MLP with 128 units.The large encoder is intended to avoid imposing a dimensional bottleneck unrelated to the training criterion.Source: §D.1.2 · p. 33
  • Table 1 compares unregularized, stochastic, dropout, weight-decay, VIB, and three DIB variants under average- and worst-ERM evaluation.The regularizers are applied only during Bob’s representation training, matching the downstream ERM formulation.Source: §D.1.2 · p. 34

D.3 Correlation

  • The correlation study trains 562 CIFAR10 models with approximately matched training performance, then compares candidate measures by rank correlation with train–test generalization gaps.Sweeps vary learning rate, batch size, dropout, width, depth, and representation dimensionality.Source: §D.3 · pp. 34–35

E Additional Experiments

  • Additional experiments examine predictive-family nesting, min-max optimization, nuisance sampling, GPU implementation, worst ERMs, regularization behavior, and generalization correlations.These studies extend the main evaluations across optimization choices, datasets, and representation-learning settings.Source: E Additional Experiments · p. 35

E.1 Sweeping over Predictive Families

The appendices test how predictive-family complexity, minimax optimization, Monte Carlo estimation, and nuisance-label decompositions affect DIB’s implementation and conclusions. Across settings, DIB’s expected behavior is largely recovered, while optimization and family-size choices introduce practical trade-offs.

E.1 Sweeping over Predictive Families

  • Width, pruning, and joint width-depth sweeps increase functional-family complexity, whereas depth alone has little effect; width is therefore used in the main experiments.Complexity is measured by fitting arbitrary CIFAR10 labelings from representations of dimensions 2, 16, and 1024.Source: §E.1 · pp. 35–36

E.2 Min-Max Optimization

  • −1.56±0.03 versus −1.65±0.00 test log likelihood favors five unrolled inner steps over joint optimization, but the computational cost is higher.Joint optimization corresponds to zero inner steps; five steps also make performance more robust to β.Source: Figure 7 · p. 37, §E.2 · p. 38

E.3 Diverging Representation from Min-Max

  • Batch normalization prevents the representation mean and standard deviation from diverging when the inner maximization is not run to convergence.Without normalization, the encoder can vary representations faster than the classifier adapts, exploiting the finite inner-loop budget.Source: §E.3 · p. 38, Figure 8 · p. 38

E.4 Monte Carlo Estimation of IV[Z →Dec(X, Y)]

  • Four, twelve, or twenty Monte Carlo samples have little effect on DIB, and the fewest predictors produce the best test log likelihood in the reported comparison.Using multiple predictors for the same nuisance labeling does not reproduce the benefit of sampling more distinct labelings.Source: §E.4 · pp. 39–40, Figure 9 · p. 39

E.5 y Decomposition of X Through Base Expansion

  • Base |Y| expansion yields −1.41 ± 0.05 versus −1.66 ± 0.09 test log likelihood for random decomposition at β = 10, without added computational cost.The construction supplies approximately uncorrelated nuisance variables and reduces redundant decompositions.Source: §E.5 · pp. 40–41

E.6 Sharing Predictors of {Dec(X, y)}y∈Y for Batch Training

  • Sharing predictors across {Dec(X, y)}y∈Y enables batch training and achieves better test performance in the reported sweep, although the authors attribute this to the chosen β values.The shared and unshared approaches have the same underlying computational complexity.Source: §E.6 · p. 41, Figure 11 · p. 41

E.7 Searching for an ERM That Does Not Generalize

  • The worst-case ERM search maximizes test loss while maintaining approximate empirical-risk minimization, using γ = 0.1 for both CIFAR100 and CIFAR10+MNIST.The search is implemented by reversing the test-example gradients and scaling them by γ.Source: §E.7 · pp. 41–42, Figure 12 · p. 42

E.8 Optimality of V-Sufficiency in Various Settings

  • Across 40 VAlice settings, maximizing IV[Z →Y] recovers the empirically optimal V-sufficient representation in all but three cases, while larger matched neural families show smaller observed differences.For large families, SGD favors simpler, nearly linearly decodable representations, but matching VBob and VAlice remains theoretically optimal.Source: §E.8 · p. 42, §E.9 · pp. 42, 44, Figure 13 · p. 43

E.11 V-Minimality as a Regularizer

This section tests DIB as a regularizer in jointly trained neural networks rather than the paper’s two-stage setting. DIB performs competitively, but its advantage is smaller because the evaluation and regularization setup differ from the theory’s focus.

E.11 V-Minimality as a Regularizer

  • DIB performs best alongside dropout on MNIST and second best after dropout on CIFAR10MNIST in the joint neural-network setting.The comparison uses test accuracy and applies each regularizer to both the representation and classifier.Source: §E.11 · p. 45, Table 3 · p. 45
  • DIB’s advantage is less pronounced because the experiments use accuracy although the theory concerns log likelihood, while standard training already generalizes relatively well.The stochastic-representation baseline is also strong under the standard training procedure.Source: §E.11 · p. 45
  • In the single-player setting, DIB regularizes only through the representation, whereas methods such as dropout regularize both the representation and downstream classifier.The latter distinction is outside the paper’s stated theory.Source: §E.11 · pp. 44, 46

E.12 Additional Correlation Experiments

The appendix evaluates V-minimality as a generalization probe across datasets, architectures, and training hyperparameters. The probe correlates better with observed generalization gaps than sharpness in these heterogeneous experiments.

E.12 Additional Correlation Experiments

  • V-minimality is bounded in [0, log |Y|] and is zero for V-minimal representations, motivating its use as a model- and dataset-agnostic generalization measure.The appendix hypothesizes that it decreases monotonically with a model’s generalization capacity.Source: §E.12 · p. 46
  • Across CIFAR-10 and SVHN, ResNet18 and MLP models, IV[Z →Dec(X, Y)] correlates significantly better with accuracy and log-likelihood generalization than sharpness.The experiment varies learning rate, weight decay, dropout, representation dimension, and VIB β across 560 trained models.Source: §E.12 · p. 46, Table 4 · p. 46

Related papers

Loading 2009.12789v2…