PaperlayerView exact revision on arXiv

Source-linked AI summary

What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Hassan Tavakoli, Bella Bose

arXiv:2610.36255v1cs.LGcs.ITmath.IT

TL;DR

The paper asks whether IB can separate representation learning from rate control, rather than requiring a new encoder for each budget. It develops sufficiency-based guarantees and evaluates frozen representations, finding lower estimated training compute and higher budget-averaged accuracy than independently trained VIBs.

  • Problem

    IB methods commonly train a new encoder for each operating point or condition the whole network on β, leaving open which parts of the solution must depend on the budget.

    Source: §1 · p. 1

  • Method

    The paper analyzes frozen budget-free representations using sufficiency deficits and tests a pipeline that fits budget-specific compressors and decoders.

    Source: §1 · p. 2

  • Results

    7.7× fewer estimated training MACs and higher budget-averaged accuracy than independently trained VIBs, with the largest gains at strong compression.

    Source: §8 · p. 18

  • Takeaways & Limitations

    For finite-label settings, the separation gap is controlled by the representation's sufficiency deficit, with extensions to adaptation and pooled domains.

    Source: §4.2 · p. 8, §8 · p. 18

    Pooled-versus-per-domain comparisons are not clean tests of pooling because pooled experiments use four times as many images and combine semantically inconsistent label spaces.

    Source: §7.1 · p. 13

Abstract

from arXiv · show

The information bottleneck (IB) trades compression of an input $X$ against prediction of a target $Y$ through a multiplier $\beta=1/\lambda$, yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as $A(\beta)=D(\beta)W$: the information-bearing directions $W$ do not depend on $\beta$, and $\beta$ acts only through scalar gains. We ask whether this separation of \emph{representation} from \emph{rate control} survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation \(F(X)\), the \emph{separation gap} is controlled by the \emph{sufficiency deficit} $\epsilon_F=I(X;Y\mid F(X))$: We show there is at least $\epsilon_F$ in the worst case for finite inputs, and it is bounded uniformly in $\lambda$ by a function of $\epsilon_F$ for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by $7.7\times$ and wall-clock time by about $2\times$ relative to end-to-end VIBs, and attains higher budget-averaged accuracy ($65.6\%$ versus $57.3\%$ per domain; $67.1\%$ versus $57.1\%$ pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression ($\lambda\le0.036$), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.

1 Introduction

The paper asks which parts of an information-bottleneck (IB) solution must depend on the operating budget. It investigates separating a budget-free representation from budget-dependent compression and decoding.

Representation–rate separation

  • The proposed architecture trains and freezes F without a rate penalty, then fits a compressor and decoder for the desired operating point.The decoder, like the compressor, depends on the budget; the paper does not claim a budget-free decoder.Source: §1 · p. 2

Questions and contributions

  • The paper studies when a budget-free representation suffices for the entire IB path and how its sufficiency deficit affects separation.It also examines adaptation, pooled domains, decoder error, and sufficient linear representations.Source: §1 · p. 2

Evidence

  • The numerical study uses four datasets, ten seeds, per-domain and pooled settings, and six baseline settings.The evidence section also reports confidence intervals and an explicit cost model.Source: §1 · p. 2

2 Problem formulation and related work

The paper formulates IB as a trade-off between prediction and compression, and places its variational neural surrogate alongside related approaches. Its operating budget is given rather than optimized.

Problem formulation

  • The IB objective combines conditional prediction uncertainty H(Y|T) with a rate penalty λI(X;T), while β=1/λ gives the equivalent multiplier form.For continuous Y, the paper uses an objective differing by the constant H(Y); this does not change gaps.Source: §2 · pp. 2–3

Variational surrogate

  • Neural IB minimizes a variational surrogate combining decoder cross-entropy with a KL-based rate term.The surrogate upper-bounds the IB objective, with equality when the decoder and prior match the corresponding distributions.Source: §2 · p. 3

Related work and operating budget

  • Unlike methods that train or condition the full network on β, this paper treats the budget as externally specified and not optimized.The experimental grid is prespecified and shared; no operating point is selected using test data.Source: §2 · p. 3

3 Gaussian IB: the factorization that motivates the paper

For jointly Gaussian data, the IB solution separates into budget-free information-bearing directions and budget-dependent gains. The resulting coordinates reduce the problem to scalar channels.

Gaussian IB factorization

  • The Gaussian IB optimum is a linear–Gaussian channel, and its matrix factorizes into a budget-free map W and budget-dependent diagonal gains D(β).The map W is computed from second moments; changing β changes rate allocation rather than information-bearing coordinates.Source: §3 · p. 4
  • In the coordinates S_G=WX, the Gaussian IB problem decouples into scalar channels with gains determined by β and canonical correlations.The budget-free map is whitened with respect to X, while the gains determine which coordinates are active.Source: §3 · pp. 3–4

Sufficient linear representations

  • A full-row-rank linear representation LX is sufficient exactly when its row space contains that of B_yx.Equivalently, B_yx=ΩL for some matrix Ω.Source: §3 · p. 4
  • For an arbitrary such linear representation, its canonical correlations with Y cannot exceed those of X with Y.The paper attributes this to the fact that canonical correlations cannot increase under a linear image.Source: §3 · p. 4

4 Separation theory: the sufficiency deficit

The paper separates a budget-free representation from rate control, with the sufficiency deficit measuring the information lost about Y. It establishes exactness under sufficiency and for Gaussian data, bounds the gap more generally, and extends certificates to adaptation, pooled domains, and changes of representation.

4 Separation theory: the sufficiency deficit

  • For jointly Gaussian data, a budget-free linear representation with diagonal, budget-dependent gains attains the unrestricted IB value at every λ.The representation uses canonical directions; the gains are independent of the invertible preprocessing map.Source: §4 · p. 5
  • The separation gap splits into representation sufficiency and compressor expressiveness, both nonnegative; only the first is characterized by ε_F.The sufficiency deficit is ε_F = I(X;Y|F(X)); it is zero exactly when F is sufficient.Source: §4 · p. 5

4.1 Exactness, stability and necessity

  • If ε_F=0, the representation loses nothing for arbitrary Y, and all budget dependence can be placed in a compressor on F(X).For finite X, the worst-case separation gap tends to ε_F as λ decreases to zero, making sufficiency necessary for exact separation.Source: §4.1 · pp. 6–7
  • For every λ≥0, the separation gap is between zero and Φ(ε_F); with finite Y, it is zero for λ≥1.The deficit is felt only below λ=1, with the worst case in the high-rate, small-λ regime.Source: §4.1 · pp. 6–7

4.2 Certificates for adaptation and pooled domains

  • For pooled domains, the deficit includes conditional alignment I(Y;S|F); this term vanishes when the label posterior given F does not depend on the domain.If label-index meanings differ across domains, the representation must implicitly encode the domain for that term to vanish.Source: §4.2 · pp. 7–8
  • Under covariate shift, each domain's deficit is at most ϱ_s times the pooled deficit, where ϱ_s≤1/w_s.The paper also gives an adaptation certificate for retraining only a compressor under a target distribution.Source: §4.2 · pp. 7–8

4.3 Change of representation: an exact identity for the deficit

  • For a change of representation, the new deficit is bounded by the original deficit plus predictor excess risk and a cross-representation term.The bound is an equality when the predictor on the new representation matches its posterior.Source: §4.3 · p. 8

5 Decoder error and decoder sharing across domains

The section decomposes decoder cross-entropy into representation, compression, and decoder-mismatch losses, then characterizes when pooled decoding is optimal. A noisy-XOR example shows that sufficiency alone does not guarantee a restricted compressor can attain the IB optimum.

5 Decoder error and decoder sharing across domains

  • A decoder’s excess cross-entropy splits into nonnegative representation loss, compression loss, and decoder mismatch.The terms are the sufficiency deficit, I_s(U;Y|T), and the expected KL divergence between the domain posterior and decoder.Source: §5 · p. 9
  • For pooled data, the pooled Bayes decoder is best among shared decoders, and no shared decoder’s average excess falls below I(Y;S|T).A shared decoder is Bayes-optimal on every domain exactly when the domain conditional label distributions agree wherever those domains overlap in T.Source: §5 · p. 9
  • When the representation and compressor are domain-independent, I(Y;S|T) is bounded above by I(Y;S|F(X)).This links the conditional alignment after compression to the corresponding alignment at the representation.Source: §5 · p. 9

5.1 What the rate controller must express: noisy XOR

  • In noisy XOR, the optimal unrestricted compressor flips the sufficient binary feature through independent Bernoulli noise, and has zero separation gap for every λ>0.The Gaussian-noise compressor has a strictly positive gap at each fixed λ, though the gap is not uniform in λ.Source: §5.1 · p. 10
  • The noisy-XOR example separates representation sufficiency from compressor expressiveness: a sufficient feature can still pair with a restricted Gaussian-noise class that misses the optimum.The unrestricted optimum’s budget dependence is only through the scalar noise probability q⋆(λ).Source: §5.1 · pp. 10–11

6 Learning algorithms and cost model

The experiments use a frozen probability interface followed by budget-specific compressor–decoder fits. Cross-entropy-only training controls the representation’s sufficiency deficit, while each fixed budget determines a separate rate-control fit.

6 Learning algorithms and cost model

  • Stage 1 trains a CNN and affine classifier head with cross-entropy only, then freezes both and retains the head.The implementation’s compressor input is the ten-dimensional softmax interface, not the CNN feature alone.Source: §6 · p. 11
  • For each fixed λ_j, Stage 2 fits a compressor and decoder on cached interface features by minimizing CE + λ_jR_agg.The latent is Gaussian with parameters output by the compressor; evaluation uses a prior fitted on training inputs and held fixed.Source: §6 · pp. 11–12
  • At evaluation, R_agg bounds I(X;T), but minibatch statistics make it only a surrogate during training.Its tightness can differ across models.Source: §6 · p. 12
  • The objective decomposes into a λ-independent sufficiency deficit and a budget-dependent compression objective, motivating separate representation learning and rate control.The deficit depends on the representation through its joint distribution with the target.Source: §6 · p. 12
  • Cross-entropy training bounds the representation’s sufficiency deficit, and the resulting separation objective equals the unrestricted optimum when that deficit is zero.For a general deficit, the paper gives a bound on the separation objective’s excess.Source: §6 · p. 12

7 Experiments

Across four datasets, the separated pipeline achieves higher budget-averaged accuracy than independently trained VIBs and uses fewer estimated training MACs. Its advantage is largest at strong compression, where VIBs approach collapse, while cross-entropy and calibration comparisons vary with operating point.

7.1 Design and protocol

  • The study compares frozen representations with one compressor–decoder pair per budget against independent VIBs, FiLM-conditioned VIBs, and OIB across four datasets and eight prespecified budgets.It uses 10 seeds; pooled-versus-per-domain comparisons are not clean pooling comparisons because pooled runs use four times as many training images and a semantically inconsistent common label space.Source: §7.1 · pp. 13–14

7.2 Results

  • +8.29 pp (E1−E3) and +9.77 pp (E2−E4) in macro budget-averaged accuracy; E2−E4 is +10.08 pp on the pooled test set.At λ = 0.5, E4 reaches 12.63% accuracy, versus 47.45% for E1 and 51.91% for E2.Source: Table 3 · p. 13, §7.2 · p. 14
  • At λ = 0.005, separated pipelines have higher cross-entropy and ECE; across budgets, the surrogate-Lagrangian comparison is mixed.For E1−E3, the budget-averaged L difference is +0.0735; E2−E4 is not significant in the macro average, while E2 is lower on the pooled test set.Source: §7.2 · p. 15, Figure 4 · p. 17
  • FiLM has lower per-domain macro cross-entropy than E1 and E3, but its budget-averaged accuracy is 4.7–6.0 pp below separation; OIB is 23–28 pp less accurate.OIB’s 128–149-nat rate is not commensurate with learned-compressor rates.Source: Figure 3 · p. 14, §7.2 · pp. 16–17
  • 7.7× fewer estimated training MACs and about 2.1× lower wall-clock time for separation than independent VIBs.The authors note that many small compressor–decoder fits dominate wall-clock time, so MAC counts overstate the realized speed-up.Source: §7.2 · pp. 16–17
  • The frozen-interface sufficiency proxy is at most 0.025 nats for per-domain representations, but its difference-of-upper-bounds estimate neither bounds nor lower-bounds the true deficit.For pooled F, the proxy is larger on three datasets; the authors connect this to unaligned label semantics and a conditional-alignment term.Source: §7.2 · p. 17, Table 6 · p. 18

8 Conclusion

The conclusion is that rate control can be separated from representation under sufficiency, with a deficit-controlled worst-case loss otherwise; experimentally, freezing a CE-trained representation lowers training cost and raises budget-averaged accuracy versus independently trained VIBs.

  • Exact separation holds for every λ when the representation is sufficient; otherwise, worst-case loss is bounded through the sufficiency deficit, with no loss for λ ≥ 1.Source: §8 · p. 18
  • For pooled domains, the certificate includes a conditional-alignment term, and the same separation also holds for the IB self-consistent iteration.Source: §8 · p. 18
  • 7.7× fewer estimated training MACs and 2.1× less wall-clock accompany +8.29 pp per-domain and +10.08 pp pooled-test budget-averaged accuracy versus independently trained VIBs.Source: §8 · p. 18
  • At small λ, the separated pipeline has higher cross-entropy and ECE; its surrogate Lagrangian is higher per-domain, indistinguishable from pooled VIB in the macro average, and lower on the pooled test set.The pooled representation is checkpoint-sensitive.Source: §8 · p. 18

Related papers

Loading 2610.36255v1…