PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Information Rates of Summaries in Simulation-Based Inference

Vansh Bansal, Pradip Sharma Poudel, James G. Scott

arXiv:2610.34703v1stat.MLcs.LGstat.CO

TL;DR

SBI summaries pass data to a posterior decoder, but their information budgets are usually implicit, complicating comparisons of summary quality. This paper models summaries as lossy channels, bounds the information needed for posterior fidelity, and evaluates summaries at matched rate. It establishes a strict stochastic advantage under a common-support condition and reports favorable rate–loss curves for stochastic bottlenecks across four simulator benchmarks.

  • Problem

    Summary choices in SBI are difficult to compare because their information budgets are rarely measured, so better reconstruction may reflect more information rather than better data discrimination.

    Source: §1 · p. 2

  • Method

    The paper models summaries as possibly randomized channels, defines rate by I(Y;S) and posterior distortion by expected KL divergence, and derives bounds using posterior quantization and probability-weighted converses.

    Source: §1 · pp. 2–3

  • Results

    Under common support, stochastic summaries strictly outperform deterministic summaries and time-sharing mixtures at every positive rate below exact preservation; stochastic bottlenecks also trace better rate–loss curves across four simulator benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Summary comparisons should match information rates, since architecture rankings change with the budget and stochastic summaries can achieve lower distortion at matched rate.

    Source: §1 · p. 3

    Experiments compare rates below roughly 12 nats, and the theorem does not establish empirical consequences at unconstrained-network rates or guarantee neural optimization reaches the better channel.

    Source: §7 · p. 12

Abstract

from arXiv · show

Simulation-based inference often encodes observations $Y$ into a fixed-dimensional summary $S$, with the posterior conditioning only on $S$. While exact preservation often requires infinite information, any such summary imposes a finite information budget on inference which is rarely an explicit choice or measurement. In this paper, we make the budget explicit by modeling the summary as a lossy channel from $Y$ to $S$, and ask two questions: how large must the budget be, and how best is it spent? First, we show for parametric models that this budget, measured by the mutual information $I(Y;S)$, grows unboundedly as the posterior concentrates, even when fixed-dimensional sufficient statistics exist. More generally, the required budget is set by the distribution of the random posterior, and we bound it from above by quantizing posteriors and from below by converses that weight each posterior state by its probability. Second, we show that a finite budget is best spent stochastically. Under a common-support condition on the simulator, covering the continuous observation spaces typical of simulators, a deterministic summary necessarily loses something: a stochastic summary achieves \emph{strictly} lower distortion than every deterministic summary and every time-shared mixture of deterministic summaries at every positive budget too small for exact preservation. Motivated by these results, we insert a binary bottleneck into neural posterior estimators, measure each trained summary's rate, and compare summaries at matched rate. Across four simulator benchmarks and several encoder families, stochastic bottlenecks achieve favorable rate--loss tradeoffs, and architecture rankings change at different information budgets.

1 Introduction

Summaries are a central SBI design choice, yet their information budgets are usually implicit and poorly measured. The paper formalizes that budget and studies both how much information posterior fidelity requires and how to spend a finite budget.

1 Introduction

  • A summary can confound comparisons between encoders: better posterior reconstruction may reflect a larger information budget rather than better discrimination of data.Dimension and parameter count are poor proxies for the information passed to the decoder.Source: §1 · p. 2
  • The paper models a summary as a possibly randomized channel and measures its rate by I(Y;S), with distortion given by expected KL divergence between full-data and summary-based posteriors.The intrinsic rate is the smallest budget achieving a chosen distortion tolerance.Source: §1 · p. 2
  • In regular d-dimensional models, bounded posterior error requires at least d/2 log n − O(1) nats, even when a fixed-dimensional sufficient statistic exists.As posteriors concentrate, the statistic must be transmitted with increasing precision.Source: §1 · p. 2
  • Quantizing posteriors gives achievable rate bounds, while converses weighted by posterior-state probability give lower bounds.The required budget depends on which posteriors occur and how often they occur.Source: §1 · pp. 2–3
  • Under common support, stochastic summaries achieve strictly lower posterior error than deterministic summaries and their time-shared mixtures at every positive rate below exact preservation.Across four simulator benchmarks, stochastic bottlenecks trace better rate–loss curves, and architecture rankings change with the budget.Source: §1 · p. 3

2 Posterior preservation as an information constraint

The section treats a summary as an information-constrained channel and defines posterior error through information lost about the parameter. Exact posterior preservation may require unbounded information, and regular parametric concentration forces the rate for bounded error to grow.

2 Posterior preservation as an information constraint

  • When posterior distributions vary continuously with the data, exact preservation requires infinite information, so no finite-rate summary suffices.Zero distortion corresponds to exact posterior sufficiency.Source: §2 · p. 4
  • Posterior distortion equals the parameter information lost through the summary: ΔKL(S) = I(θ;Y|S) = I(θ;Y) − I(θ;S).The summary posterior is the conditional expectation ΠS := E[ΠY|S].Source: §2 · p. 4
  • For fixed-dimensional sufficient statistics, increasing posterior concentration still requires unbounded transmission precision and therefore increasing information rate to keep distortion bounded.This follows by combining the posterior-loss identity with the Bayes-information expansion for regular parametric models.Source: §2 · p. 4
  • For regular d-dimensional i.i.d. models, every summary with distortion at most ε requires rate at least d/2 log n − O(1).The result assumes the prior-averaged Clarke–Barron conditions.Source: §2 · p. 4
  • The information criterion separates summaries with identical posterior distortion but different rates, unlike a fidelity-only criterion that does not charge for irrelevant information.The paper describes rate-based selection as complementary to expected-posterior-entropy selection within a preselected class.Source: §2 · p. 4

3 Posterior geometry and the information rate

The intrinsic rate is governed by the random posterior as a source: achievable rates depend on its approximation and converses account for how much predictive mass posterior states carry. Posterior quantization supplies upper bounds, while probability-sensitive bounds and the Gaussian example clarify why rate-constrained stochastic summaries can outperform deterministic ones.

3 Posterior geometry and the information rate

  • The posterior-source reduction makes the random posterior ΠY the source whose representation determines the intrinsic rate.Its rate-distortion problem minimizes mutual information between the random posterior and its reconstruction under expected KL distortion.Source: §3 · p. 5

3.1 Achievability by posterior quantization

  • With at most M summary values, optimal distortion equals that of quantizing the random posterior, and R⋆(ε) is upper-bounded by log Nav(ε).The quantization distortion averages over posterior occurrences under the simulator and prior.Source: §3.1 · p. 5
  • At fixed cardinality, a stochastic summary can be replaced by a deterministic one with no greater posterior distortion, but this does not preserve mutual information.The cardinality result therefore does not imply optimality at matched information rates.Source: §3.1 · p. 5

3.2 Converses that retain predictive mass

  • A converse based on the largest predictive mass inside a forward-KL ball retains the probability of posterior states rather than only their separation.For a uniform law on M sufficiently separated states, the bound has the familiar log M scale.Source: §3.2 · p. 6
  • Posterior approximation error and predictive concentration act as complementary controls on the information budget.The bounds clarify why cardinality and mutual information can yield different conclusions about encoder randomization.Source: §3.2 · p. 6
  • For the Gaussian location model, the optimal finite-rate summary adds independent Gaussian noise to the sample mean, and no deterministic summary at the same rate achieves that frontier.The noise variance is chosen to meet the rate budget.Source: §3.2 · p. 6

4 When are stochastic summaries strictly optimal?

Under a common-support condition, truly stochastic summaries strictly outperform deterministic summaries and their revealed time-sharing mixtures at interior information rates. The section explains the mechanism and clarifies the result’s scope and boundary cases.

4 When are stochastic summaries strictly optimal?

  • The comparison distinguishes deterministic quantizers from revealed time-sharing mixtures; the theorem’s advantage instead comes from summary randomness hidden from the decoder.The selector in a time-sharing mixture is revealed, unlike the noise added to the Gaussian sample mean.Source: §4 · pp. 6–7

4.1 The main separation result

  • Under common support and 0 < I(θ;Y) < ∞, stochastic summaries strictly beat deterministic summaries and their revealed time-sharing mixtures at interior rates.When R0 = ∞, the advantage persists at every finite information budget.Source: §4.1 · p. 7
  • Hidden random relabeling saves κ(T)t log(1/t) rate while adding o(t log(1/t)) distortion; revealed time-sharing spends that saving on a better quantizer.The argument controls both effects uniformly, without requiring an attained deterministic optimum.Source: §4.1 · pp. 7–8

4.2 Remarks about Theorem 4.3

  • For m-valued positive-probability observations with pairwise-equivalent posteriors, separation spans 0 < R < H(ΠY), with an optimum using at most m + 1 outputs.Beta–Binomial experiments with positive prior parameters satisfy these conditions.Source: §4.2 · p. 8
  • Common support is sufficient, not necessary: posterior overlap alone neither guarantees nor rules out separation, and when θ=f(Y) with finite entropy, revealed time-sharing suffices.In that target-determined case, revealed time-sharing between a constant quantizer and f(Y) attains the line.Source: §4.2 · p. 8
  • The result concerns mutual-information budgets for one dataset at a time, not deterministic block codes; deterministic summaries can still be optimal under fixed-cardinality constraints.The result also does not imply deterministic optimality for every output-entropy-constrained problem.Source: §4.2 · p. 9

5 Binary-bottleneck NPE: comparing summaries at measured rate

The section measures summary information rates in jointly trained binary-bottleneck NPEs and compares their evaluation loss at matched rates. It also compares encoder performance across information budgets.

5 Binary-bottleneck NPE: comparing summaries at measured rate

  • The experiments test whether stochastic bottlenecks improve evaluation loss at matched estimated rates and whether architecture comparisons change with rate.The section also asks whether stochastic channels outperform deterministic time-sharing where an exhaustive comparison is available.Source: §5 · p. 9
  • The binary bottleneck sits between encoder and posterior decoder; its stochastic channel samples conditionally independent bits, while its deterministic counterpart thresholds the same bit probabilities.The decoder receives only the reconstructed summary, with no skip connection from the original encoder output.Source: §5 · p. 9
  • The summary network, binary autoencoder, and posterior decoder are trained jointly with a sparse Bernoulli penalty that regularizes channel information rate.Both channels use forward codes and straight-through gradients; deterministic training anneals temperature before a hard-threshold phase.Source: §5 · pp. 9–10
  • Varying β produces operating points, and deployed rates are estimated after training using Monte Carlo and marginal-model approaches.The empirical-mixture estimate is downward-biased on average, while marginal models provide a variational upper bound; horizontal segments connect the estimates when exact enumeration is infeasible.Source: §5 · p. 10
  • Evaluation NLL differences combine information loss and decoder error because the conditional-entropy term is common within a simulator and parameterization.On analytic benchmarks, trained-decoder error is multiple orders of magnitude smaller than the measured NLL separation.Source: §5 · p. 10

6 Experiments

The experiments compare stochastic and deterministic summaries in a count benchmark and four simulators, then examine encoder rankings at matched rates. Stochastic summaries show favorable rate–NLL tradeoffs, while architecture comparisons vary with rate.

Count benchmark

  • The count benchmark found an approximately 0.036-nat maximum reduction in information loss beyond the deterministic time-sharing hull; decoder approximation gaps were approximately 4 × 10^-4 nats.The experiment enumerated 21,147 partitions of the nine possible count values and compared their deterministic hull with achievable stochastic channels.Source: §6 · p. 10

(H1) Stochastic versus deterministic summaries

  • Mean matched-rate NLL differences for stochastic BNPEs were −0.168 to −0.617 nats across the four simulator-encoder combinations, and six additional combinations showed the same advantage.In Figure 2, all evaluated stochastic BNPEs have lower NLL than the deterministic time-sharing hull at matched rate.Source: Figure 2 · p. 11, §6 · p. 11

Experimental setup

  • The simulator study used Bernoulli GLM, g-and-k, Lotka–Volterra, and Ricker, with a 128-bit bottleneck and a shared conditional-flow family.The count experiment isolates information loss from decoder approximation; simulator comparisons use evaluation NLL at estimated information rate.Source: §6 · pp. 10–11

(H2) Architecture comparisons depend on rate

  • Encoder comparisons changed with rate: Deep Sets changed sign on g-and-k, Set Transformer remained favorable over the displayed shared range, and TCN stayed above BiLSTM on Lotka–Volterra.On Bernoulli GLM, the Set Transformer gap narrowed at intermediate rates and Deep Sets versus Set Transformer rankings changed; on Ricker, the difference was smaller and varied with rate and seed.Source: Figure 3 · p. 11, §6 · p. 12

7 Limitations and Conclusion

The paper concludes that information rate complements posterior fidelity for comparing summaries, and that stochastic summaries can have a strict advantage at matched rates. The empirical scope and theorem assumptions leave several questions open.

7 Limitations and Conclusion

  • Experiments compare summaries below roughly 12 nats; empirical consequences of the theorem's separation at unconstrained-network rates remain open.The theorem addresses a summary channel on one observation or simulated dataset, not deterministic block codes over independent experiments, and does not guarantee neural optimization reaches the better channel.Source: §7 · p. 12
  • The common-support condition is sufficient but not necessary, and the separation for differing sampling supports remains uncharacterized.The theorem's scope does not establish that neural optimization will find a better channel.Source: §7 · p. 12
  • A fixed representation dimension does not fix information rate, and regular parametric concentration can require increasing rate even with a fixed-dimensional sufficient statistic.The paper connects rate requirements to the law of the random posterior and reports favorable stochastic rate–NLL tradeoffs and architecture rankings that vary with rate.Source: §7 · p. 12

AI use statement

The closing materials disclose AI assistance and guide readers through the appendices. The appendix foundations state the experiment's assumptions and define notation for quantizers, posteriors, relevance, and divergence.

AI use statement

  • Generative AI tools assisted with literature searches, code review, figure preparation, and manuscript editing; the authors retain responsibility for scientific claims and presentation.Source: AI use statement · p. 12

Appendix guide

  • Appendices A–D develop the separation proof and its scope, while the remaining appendices cover posterior geometry, parametric arguments, experimental details, and related work.Source: Appendix guide · p. 18

A Foundations for the separation theorem

A.1 Notation

  • G denotes countable-valued deterministic quantizers g(Y) with finite output entropy; Π_T denotes the random posterior for S = T, and relevance I(θ; T) measures retained parameter information.The appendix relates relevance to posterior distortion through I − ∆KL(T).Source: §A.1 · p. 18
  • The weighted Jensen–Shannon divergence combines KL terms from P and Q to their mixture, while singular mass measures probability P assigns to events impossible under Q.Singular mass lies in [0, 1] and vanishes exactly when P is absolutely continuous with respect to Q.Source: §A.1 · p. 19

A.2 Results for deterministic summaries

This section develops results on deterministic-summary rates and approximation, revealed time-sharing, mixture divergence, finite quantizers, and rate constraints from exact preservation. It also characterizes common support and shows that stochastic relabeling’s distortion cost is asymptotically smaller than its rate saving.

A.2 Results for deterministic summaries

  • A deterministic summary has finite rate exactly when its output law is countably atomic with finite entropy; any nonatomic component gives infinite rate, unlike stochastic summaries.In the finite-rate case, I(Y;g(Y)) = H(g(Y)).Source: §A.2 · pp. 19–20
  • Finite-valued deterministic quantizers can increase I(θ;U_n) to I while reducing ΔKL(U_n) to zero, so Ddet(R) → 0 as R → ∞.For every η > 0, some finite quantizer has distortion below η.Source: §A.2 · pp. 20–21

A.3 Properties of time-sharing mixtures

  • Revealed time-sharing of deterministic summaries realizes conv(Adet), and Dts is convex, nonincreasing, continuous, with 0 ≤ D⋆ ≤ Dts ≤ Ddet ≤ I.Dts(0) = I, Dts(R) → 0 as R → ∞, and its zero-distortion threshold satisfies R0 ≥ I > 0.Source: §A.3 · pp. 21–22
  • Truncating countable quantizers preserves the time-sharing frontier, while zero distortion requires the random posterior to be measurable from the summary and costs at least H(ΠY) when ΠY is countably valued.The lower bound may be infinite when H(ΠY) is infinite.Source: §A.3 · pp. 22–23

A.4 Asymptotics of mixtures

  • For P and Q, JSw(P,Q) is at most h2(w), and it decreases under measurable maps.The passage also states wKL(P∥Q) = JSw(P,Q) + KL(Mw∥Q).Source: §A.4 · pp. 24–25

B.2 The common support assumption

  • Common support means mutually absolutely continuous sampling laws—not equal topological supports—and, for dominated simulators, positive density π ⊗ P_Y-almost everywhere.The condition is preserved by sufficient statistics and finite i.i.d. replication; equivalence alone does not ensure finite mutual information.Source: §B.2 · pp. 26–27

B.3 Supporting lemmas for Theorem 4.3

  • If ψ(δ) < u, every quantizer with relevance I(θ;T) ≥ u has collision probability κ(T) > δ, using I(θ;T) ≤ ψ(1 − pmax).Source: §B.3 · pp. 27–28
  • Random relabeling preserves the quantizer marginal, saves at least t log(1/t) rate, and leaves positive information for every nonconstant quantizer.This rate-saving mechanism supports the paper’s result that stochastic summaries can strictly outperform deterministic summaries at positive budgets too small for exact preservation.Source: §B.3 · pp. 28–29
  • Its added posterior distortion is bounded by tI + h2(t), and under common support is o(t log(1/t)), below the rate-saving scale.The uniform bound is 0 ≤ Lt(T) ≤ tI + JSt(QθY, PθY) ≤ tI + h2(t); common support makes this bound o(t log(1/t)).Source: §B.3 · pp. 29–31

B.4 Proof of Theorem 4.3

The proof establishes that stochastic summaries strictly outperform deterministic summaries and their time-shared mixtures at every positive interior rate. In the Gaussian example, the two classes have the same initial slope yet retain a strict gap at every positive rate.

B.4 Proof of Theorem 4.3

  • At every positive interior rate R, D⋆(R) < Dts(R), without requiring either frontier to be attained.The proof relabels components of a nearly optimal deterministic mixture to save rate, then spends that rate on a finite quantizer with lower distortion.Source: §B.4 · pp. 31, 33
  • At rate zero the two frontiers equal I, and when R0 is finite both are zero for rates at or above R0.These endpoint conclusions complement the strict separation on the positive interior interval.Source: §B.4 · p. 33
  • For each fixed 0 < R < R0, rate-feasible channels sufficiently near D⋆(R) cannot be finite revealed mixtures of deterministic quantizers.The positive gap is pointwise in R; it is not asserted to be uniform across rates, experiments, or neural model classes.Source: §B.4 · p. 33

B.5 Equal initial slopes do not rule out a strict gap

  • In the Gaussian example, the deterministic and stochastic relevance-to-rate suprema both equal ρ², and their distortion frontiers have the same initial slope, −ρ².Deterministic threshold quantizers approach the shared ratio, but the deterministic supremum is not attained.Source: §B.5 · pp. 34–35
  • Despite matching initial slopes, D⋆(R) < Dts(R) for every R > 0 in the Gaussian example.Thus equality in first-order efficiency at zero does not eliminate a strict separation at positive rates.Source: §B.5 · p. 34

C Finite observation spaces: strict improvement and attainment

Finite observation spaces make the deterministic time-sharing frontier attainable and expose conditions under which stochastic summaries strictly improve distortion. They also permit exact zero-distortion thresholds, finite-output attainment, and an explicit count-benchmark witness.

C Finite observation spaces: strict improvement and attainment

  • The finite set of deterministic partitions makes the time-sharing envelope attainable and reveals how confusing two cells can save rate at a distortion cost set by their posteriors.The finite-observation results also extend separation beyond common-support cases when the directional support criterion holds.Source: C Finite observation spaces: strict… · pp. 35–37
  • At every interior rate below R0 = H(ΠY), the finite-experiment deterministic-mixture optimum is attained, has positive distortion, and uses the full rate.Its zero-distortion threshold is R0.Source: C Finite observation spaces: strict… · p. 36
  • If an optimal deterministic mixture contains two cells with α(Πa | Πb) < D/(R0 − R), a finite stochastic channel achieves strictly lower distortion at rate R.A rare confusion between cells trades saved rate for added distortion; the threshold is bounded by I/R0 ≤ 1.Source: C Finite observation spaces: strict… · pp. 37–38
  • With finite observations and pairwise-equivalent posteriors, strict separation holds throughout 0 < R < H(ΠY), including Beta–Binomial counts and ordered samples.For the Beta–Binomial count, the threshold is H(C).Source: C Finite observation spaces: strict… · pp. 38–39
  • For m positive-probability observation atoms, an optimal channel exists with at most m + 1 outputs, even for an infinite parameter space.The proof optimizes over probability laws on a simplex, then reduces an optimizer’s support.Source: C Finite observation spaces: strict… · p. 39
  • At rate 0.5, the feasible stochastic count channel has distortion 0.154529, below the time-sharing envelope 0.162253 and best deterministic count quantizer 0.187462.The witness is feasible but not claimed globally optimal; the quadrature values are numerical approximations, not certified enclosures.Source: C Finite observation spaces: strict… · pp. 40–41

D Examples delimiting the support and rate assumptions

The examples distinguish conditions that do and do not yield strict stochastic improvement, while separating mutual-information budgets from output-entropy constraints. They also show that infinite mutual information can arise under noisy, equivalent observation laws.

D Examples delimiting the support and rate assumptions

  • When θ is a discrete function of Y, distortion is bounded below by H(θ) − I(Y;S), and visible time-sharing of constant and identity summaries attains the bound.A positive-rate summary carrying only independent noise can lie above this frontier.Source: D Examples delimiting the support and… · p. 41
  • Overlapping posteriors and a target that is not a function of Y do not ensure strict separation: in the erasure example, every channel has I(θ;S) = αI(Y;S).The frontier is attained by visible time-sharing of constant and identity quantizers.Source: D Examples delimiting the support and… · p. 41
  • Unequal posterior supports can still permit strict separation: for the two-bit example, a directional support criterion yields improvement at every interior rate.The optimal deterministic mixture uses the identity quantizer with positive weight.Source: D Examples delimiting the support and… · p. 41
  • Equal topological supports are weaker than mutual absolute continuity, the measure-theoretic common-support condition used for the general separation result.The paper’s equal-topological-support example has mutually singular sampling laws.Source: D Examples delimiting the support and… · pp. 41–42
  • Under an output-entropy constraint, a randomized fair-bit channel improves on every feasible deterministic quantizer, but cardinality and entropy-penalized hard-assignment claims do not transfer.Relabeling preserves output entropy and therefore saves no cost under this constraint.Source: D Examples delimiting the support and… · p. 42
  • Infinite mutual information need not indicate a noiseless observation: an integer-valued parameter observed with additive Gaussian noise can have I(N;Y) = ∞ despite equivalent sampling laws.The paper notes that an exactly observed continuous parameter coordinate is only one possible cause of infinite mutual information.Source: D Examples delimiting the support and… · p. 42

E Proofs for Section 2

The proofs connect posterior distortion to information lost about the parameter and reduce rate optimization to compressing the random posterior itself. They derive converse bounds and characterize endpoint behavior, including when exact preservation requires infinite rate.

E Proofs for Section 2

  • Posterior distortion equals I(θ;Y | S), or equivalently the information about θ lost by replacing Y with S.The proof obtains the identity by expanding I(θ;Y,S) in two orders along the Markov chain.Source: E Proofs for Section 2 · p. 42
  • Data processing gives the converse bounds R⋆(ε) ≥ [I(θ;Y) − ε]+ and D⋆(R) ≥ [I(θ;Y) − R]+.These follow by combining the distortion identity with I(θ;S) ≤ I(Y;S).Source: E Proofs for Section 2 · p. 43
  • The random posterior Z = ΠY is itself a posterior, and the rate-distortion problem reduces to sending Z through a reconstruction channel with expected KL distortion.The reduction preserves feasibility in both directions and cannot increase rate.Source: E Proofs for Section 2 · pp. 43–44
  • A summary kernel can be conditionally averaged given the posterior ΠY without changing the joint law of (θ,S), while its rate can only decrease.The averaged kernel need not be deterministic, even if the original summary was a deterministic function of Y.Source: E Proofs for Section 2 · p. 44
  • R⋆ is nonincreasing and convex, equals zero for ε ≥ I(θ;Y), and is infinite at zero distortion when the law of ΠY is non-atomic.Exact preservation forces ΠY to be a measurable function of S, whose mutual information is infinite when ΠY is non-atomic.Source: E Proofs for Section 2 · pp. 44–45

F Proofs for Section 3

The Section 3 proofs turn posterior approximation into finite-codebook summaries and relate the resulting rate bounds to codebook size. The Hellinger extension supplies distortion guarantees under explicit geometric conditions, while warning that posterior and sampling-model covering rates do not automatically coincide.

F Proofs for Section 3

  • Replacing a stochastic M-symbol summary by a deterministic nearest-center quantizer gives no greater posterior distortion while retaining the alphabet-size constraint.The rate of the resulting deterministic summary is at most log M.Source: F Proofs for Section 3 · pp. 45–46
  • A uniform KL cover of the posterior family yields a deterministic summary with distortion at most ε and rate at most log N_unif^KL(ε).The nearest-center quantizer assigns each observation to a covering posterior center; its cell posterior can only reduce expected divergence.Source: F Proofs for Section 3 · p. 46
  • For unbounded Gaussian posterior means with a full-support predictive law, every finite codebook has infinite essential-uniform forward-KL radius; no finite uniform Hellinger cover exists below radius 1.The proof uses means far from the tight region occupied by finitely many codebook centers.Source: F Proofs for Section 3 · pp. 46–47

G Hellinger covering and the local modulus

  • A uniform Hellinger cover of radius r yields a deterministic summary with rate at most log M, squared-Hellinger distortion at most 4r², and KL distortion at most ω(2r).For an M-center codebook with expected squared nearest-center error e, the bound is ΔH(S) ≤ 4e; if KL ≤ C H² for relevant pairs, then ΔKL(S) ≤ 4Ce.Source: G Hellinger covering and the local… · pp. 47–48
  • Sampling-model Hellinger covering rates cannot be imported directly into posterior geometry; translation requires an explicit map and tail control.A local quadratic modulus also requires individual distances to stay within its validity range, which uniform covers provide but average bounds do not.Source: G Hellinger covering and the local… · p. 48

H.1 Proof of the posterior small-ball bound

The appendices develop posterior-based rate converses and parametric bounds, including conditions for matching rate exponents and valid numerical evaluation. This part verifies a posterior small-ball bound for non-Gaussian exponential-family models, yielding logarithmic required-rate growth in sample size with a dimension-dependent coefficient.

H.1 Proof of the posterior small-ball bound

  • The small-ball proof compares joint and product probabilities of a KL ball, while the Laplace form gives R⋆(ε) ≥ [sup_{s>0} {−sε − log A_Z(s)}]_+.The ball probability under the joint law is at least 1−ε/t, while under the product law it is bounded using Q_Z(t). The Laplace bound integrates the distortion profile and is the general Shannon lower bound of Riegler et al. (2018).Source: §H.1 · pp. 48–49, §H.2 · p. 50

H.3 Matching exponents under small-ball regularity

  • If Q_Z(t) ≤ Ct^α at every t > 0, the small-ball converse and matching upper bound give R⋆(ε) = α log(1/ε) + O(1) as ε ↓ 0.These are sufficient conditions for a matching exponent for a fixed law, not a general theorem; for experiment sequences, the dependence of C, K, and α on n must be tracked.Source: §H.3 · p. 50

H.4 A mass-sensitive cluster converse

  • A mass-sensitive cluster converse weights separated posterior regions by their probability and cluster-weight entropy, unlike a small-ball bound controlled by the largest cluster.It applies to unequal-probability, well-separated regions; for uniformly distributed separated posterior atoms, it recovers a finite Fano converse. The proof also establishes Δ_KL(S) ≥ 2Δ_H(S).Source: §H.4 · p. 51

H.5 Evaluating the converses

  • Numerical evaluations of Q_Z(t) or A_Z(s) support a converse only if they certify the supremum; restricted searches or local-optimizer values are merely candidates.Repeated restarts do not repair this limitation.Source: §H.5 · p. 52

I.1 A mean-square route to the lower bound

  • Under Clarke & Barron’s regularity conditions, the prior-averaged expansion yields a parametric information lower bound; alternatively, mean-square localization at scale n^-1/2 gives a d/2 log n lower bound.The mean-square route assumes finite prior entropy and an estimator with nD_n bounded, replacing the Clarke–Barron conditions; weak posterior convergence alone does not suffice.Source: I Parametric rates: hypotheses and… · p. 52, §I.1 · pp. 52–53

I.2 Achievability under a quadratic posterior chart

  • Under a quadratic posterior chart, a deterministic quantizer achieves the leading d/2 log n rate, but remains strictly above the optimal frontier at every fixed distortion.The chart assumption bounds posterior KL divergence by squared distance between chart coordinates; when L_n = O(n) and V_n = O(1), the quantizer has rate d/2 log n + O_{d,ε}(1). Its excess over R⋆_n(ε) is strictly positive and bounded in n.Source: §I.2 · pp. 54–55

I.3 A verified non-Gaussian class

  • For a one-dimensional exponential family with κ ≤ A'' ≤ K and a Gaussian prior, the posterior chart and estimator bounds imply R⋆(ε) = 1/2 log n + O_ε(1).The sample mean satisfies the quadratic chart condition, and the inverse-score estimator has Bayes mean-square error at most K/(κ²n); d independent coordinates yield d/2 log n + O_{d,ε}(1).Source: §I.3 · p. 55
  • The sample-mean chart verifies the required conditions using L_n = n/(2κ) and chart variance V_n ≤ K²τ² + K/n.The estimator bound yields lim sup_n nD_n ≤ K/κ².Source: §I.3 · pp. 55–56
  • The scalar model has R⋆_n(ε) = 1/2 log n + O_ε(1), with matching lower and upper bounds.The lower bound is 1/2 log n − O_ε(1); the upper bound follows from the verified chart and variance conditions.Source: §I.3 · p. 56
  • For d coordinate-wise models, the same argument gives R⋆_n(ε) = d/2 log n + O_d,ε(1).The posterior factors by coordinate, divergences add, and the chart variances, squared estimation errors, prior entropies, and Bayes informations add across coordinates.Source: §I.3 · p. 56
  • The curvature condition covers non-Gaussian equal mixtures of N(±b, σ²), but is not automatic for Bernoulli or Poisson data.For the mixture, κ = σ² and K = σ² + b²; Bernoulli curvature degenerates, while Poisson curvature is unbounded.Source: §I.3 · p. 56

I.4 Proof of Proposition 3.4

The Gaussian proof establishes the exact rate–distortion frontier: adding independent Gaussian noise to the sufficient posterior mean attains the converse bound, while exact preservation requires infinite rate. The accompanying calculations contrast this exact Gaussian frontier with grid-based achievable Beta–Bernoulli rates and describe the simulator bottleneck training choices.

I.4 Proof of Proposition 3.4

  • The converse uses conditional entropy power, and adding independent Gaussian noise to the sufficient posterior mean makes its inequalities equalities.The construction is S = m + N, with N independent Gaussian noise.Source: §I.4 · pp. 57–58
  • For 0 < ε < d/2 log(1 + a), every channel meeting distortion ε needs at least the rate solving d/2 log(1 + ae^(-2R/d)) = ε, attained by the noisy posterior-mean channel.At zero rate the constant summary has distortion d/2 log(1 + a).Source: §I.4 · p. 58
  • The random Gaussian posterior is non-atomic, so exact preservation has infinite rate; the sufficient mean nevertheless has O(n) covering cost under the stated assumption.Average quantization attains the same leading order even though no finite uniform cover exists.Source: §I.4 · p. 58

J Details of the parametric rate–distortion calculations

  • Figure 4 plots exact Gaussian rates and grid-based achievable Beta–Bernoulli rates against sample size or dimension; the Beta–Bernoulli curves are not optimized over all channels.At fixed total tolerance, the exact Gaussian rate grows as d/2 log d when dimension increases at fixed n.Source: Figure 4 · p. 59, J Details of the parametric… · p. 59

K.1 Training and checkpoint selection

  • The simulator experiments use a 128-bit binary code; stochastic BNPE samples codes at deployment, while deterministic BNPE uses hard-thresholded codes there.Both use straight-through gradients during training; the deterministic model anneals its Bernoulli temperature before thresholding.Source: §K.1 · p. 60

K.2 Evaluation loss and information-rate estimates

The evaluation estimates deployed code rate separately from posterior loss, then uses controlled count and Gaussian benchmarks to interpret the comparisons. The related-work discussion situates the paper’s posterior-fidelity frontier, quantization bounds, and matched-rate evaluation among information-bottleneck and compression approaches.

K.2 Evaluation loss and information-rate estimates

  • The empirical-mixture rate estimate is bounded by log M ≃ 12.477, and evaluation clips Bernoulli probabilities to [10^-5, 1 − 10^-5].Checkpoint NLL averages eight deployed code draws for stochastic deployment and uses one pass for hard deployment; rate uses 65,536 sampled codes.Source: §K.2 · p. 61

K.3 Count benchmark computation

  • The count benchmark enumerates all partitions of nine count values and their time-sharing convex hull, while iterative information-bottleneck updates provide attainable stochastic channels.It uses 512-point quadrature for the remaining integral over θ and five training seeds per neural-summary arm.Source: §K.3 · p. 61

K.4 Gaussian diagnostic with a fixed-covariance decoder

  • At zero rate, Dmean = 8 while posterior-KL distortion is 2 log 5 ≃ 3.2189; the fixed-covariance decoder’s error remains meaningful even with its mean optimally fitted.The unrestricted Gaussian squared-error reference is Dmean(R) = 8e^(-R/2), distinct from the posterior-KL frontier.Source: §K.4 · p. 63

K.5 Additional posterior diagnostics

  • Learned-ratio differences for I(θ; S) also include both estimators’ approximation errors, unlike the conjugate experiment’s direct posterior-loss and decoder-mismatch evaluation.Simulator results report NLL and code rate.Source: §K.5 · p. 63

L Related work and connections

  • The paper’s quantization bounds treat the random posterior as a source to represent, with converses retaining the predictive mass of distinguishable posterior states.The binary-bottleneck protocol compares architectures and stochastic or deterministic modes at matched estimated rates, distinguishing deployed rate from the variational training penalty.Source: L Related work and connections · pp. 64–65

Related papers

Loading 2610.34703v1…