PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Information-Theoretic Analysis of ReLU Nets

Johan Mylius-Kroken, Elisabeth Wetzer, Ali Ramezani-Kebrya, Robert Jenssen, Kristoffer Knutsen Wickstrøm

arXiv:2610.36864v1stat.MLcs.LG

TL;DR

High-dimensional MI estimates can be dominated by hyperparameter-dependent bias. This paper analyzes the intrinsic region labels of CPWL networks through an information decomposition, a routing estimator, and a functional-equivalence quotient. The routing estimator agrees with established baselines, and the quotient recovers informative estimates when the raw partition saturates.

  • Problem

    In high dimensions, MI estimators depend on user-chosen hyperparameters whose bias can dominate the signal in deep networks.

    Source: §1 · p. 1

  • Method

    The paper decomposes I(Y; T) into geometric terms, estimates routing information from the network’s discrete partition, and uses a functional-equivalence quotient to coarsen saturated partitions.

    Source: §1 · p. 2, §3.6 · p. 5

  • Results

    The routing estimate agrees with three established MI baselines on the same networks and data, and the functional-equivalence quotient recovers meaningful estimates as input dimensions grow.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The decomposition makes information-bottleneck observations about CPWL networks measurable as claims about specific terms grounded in network geometry.

    Source: §1 · p. 2

    As occupancy approaches 1, regions become nearly single-sample, the estimate approaches empirical label entropy, and estimates are unreliable.

    Source: §3.5 · p. 5

Abstract

from arXiv · show

Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label $\Pi$ alongside the continuous representation $T$ and label $Y$. We introduce a decomposition of the mutual information $I(Y;T)$ into interpretable geometric terms. We show that the dominant term, the *routing information* $I(Y; \Pi)$, admits an estimable plug-in estimator, that in the well-sampled regime, approximates a population lower bound on the joint information $I(Y; \Pi, T)$. When the partition saturates, a functional-equivalence quotient collapses regions implementing the same linear operator and restores informativeness; this extension has a single bounded, dimensionless parameter, which is reported through the quotient's compression ratio. The routing estimator agrees strongly with three established MI baselines applied to the same networks and data. The functional quotient recovers informative estimates in the regime bordering the well-sampled and under-sampled regime. We provide a framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.

1 Introduction

High-dimensional MI estimates can be dominated by hyperparameter-dependent bias. The paper uses CPWL networks’ intrinsic region partitions to develop a geometric information framework and an estimator whose calibration and saturation correction are empirically examined.

1 Introduction

  • CPWL networks partition inputs into convex linear regions, each identified by a binary activation pattern, giving every input a discrete region label alongside its continuous representation.Layer composition refines the partition, and the network is affine on each resulting region.Source: §1 · p. 1, Figure 1 · p. 2
  • The framework decomposes I(Y; T) into geometrically interpretable terms, including estimable routing information, and extends routing estimates with a quotient merging regions that implement nearly identical linear operators.The quotient is intended for the fine-resolution regime, where the number of regions approaches the number of samples.Source: §1 · p. 2
  • The plug-in routing estimate agrees with three established MI baselines on the same networks and data, while the functional-equivalence quotient recovers meaningful estimates as input dimensions grow.The calibration checks internal consistency, not whether routing information approximates I(Y; T).Source: §1 · p. 2
  • The framework aims to turn information-bottleneck observations about CPWL networks into measurable claims about specific terms grounded in network geometry.Examples include whether information changes with depth, folding affects class content, or MI saturates before generalisation.Source: §1 · p. 2

2 Related Work

Prior work studies CPWL region geometry, information estimation, and activation-pattern structure. This paper distinguishes its approach by analyzing the inherent discrete partition rather than substituting a surrogate for I(Y; T).

2 Related Work

  • Region-count research established combinatorial analyses for ReLU networks [29] [35], refined bounds and empirical counts [18] [41], and extended the framework to arbitrary CPWL activations.Practical region counts are far below theoretical maxima [41].Source: §2 · p. 2
  • Prior work found I(Y; T) ill-defined for deterministic networks on continuous inputs, with estimates dominated by binning and noise-injection choices [15].Alternatives include sliced MI [14] and variational lower bounds [46].Source: §2 · pp. 2–3
  • Unlike sliced MI, which adds choices of slice distribution and count, the paper uses finite-alphabet I(Y; Π) as a term in an exact decomposition, not as a surrogate for I(Y; T).I(Y; Π) lower-bounds I(Y; Π, T), while the gap to I(Y; T) involves within-region information and routing loss I(Y; Π | T).Source: §2 · p. 3
  • Activation-pattern studies include path-based space-folding measures and the ReLU Transition Graph [8], later extended with region-entropy and KL-based analysis [7].The graph framework was further developed by Gaines and Bi [10].Source: §2 · p. 3

3 Intrinsic information theoretic analysis of CPWL networks

CPWL networks provide a finite region label alongside their continuous representation, enabling an information decomposition into geometric terms. The framework estimates routing information from observed regions and coarsens fine partitions through functional equivalence.

3.3 Decomposition of I(Y ; T)

  • Theorem 3.1 separates joint information into routing information, within-region information, routing loss, and representation insufficiency, each with a distinct geometric interpretation.Routing information is the only directly estimable term; routing loss captures class-relevant region information lost when reading T alone.Source: §3.3 · p. 4
  • Routing loss is positive when distinct regions overlap in latent space and assign different class distributions to the overlapping values.A sufficient condition is overlapping regions with differing region-conditional label laws when within-region label uncertainty is independent of T.Source: §3.3 · p. 4

3.4 Structural inequalities

  • Neither I(Y ; Π) nor I(Y ; T) dominates unconditionally; their difference depends on whether within-region discrimination exceeds routing loss.I(Y ; T) > I(Y ; Π) exactly when within-region discrimination exceeds routing loss.Source: §3.4 · p. 4

3.5 The plug-in routing estimator

  • In the well-sampled regime, the plug-in routing estimate approximates I(Y ; Π), which lower-bounds I(Y ; Π, T) up to the stated truncation gap.The gap is bounded by H(Y) Pr(Π ∉ ΩD) + h2(Pr(Π ∉ ΩD)); the estimator is upward biased at finite N.Source: §3.5 · p. 5
  • As occupancy ρ approaches 1, most regions become singletons and the plug-in estimate approaches empirical label entropy, whether or not the network learned class structure.Such estimates are unreliable, and standard bias corrections do not repair them.Source: §3.5 · p. 5

3.6 The fine-resolution regime and the functional quotient

  • When the raw partition becomes too fine to estimate, the functional quotient groups regions implementing the same linear operator to extend estimation into that regime.The quotient estimator provides an additional estimable lower bound beneath I(Y ; Π).Source: §3.6 · pp. 5–6
  • The compression ratio ρfunc(ε) diagnoses pattern redundancy: low values mean many patterns collapse to few operators, while values near 1 indicate little change from raw routing estimates.The relative Frobenius criterion uses ε to compare the region operators.Source: §3.6 · pp. 5–6

4 Experimental setup

The experiments use fixed training and evaluation settings to test routing-estimator calibration and quotient behavior under saturation. Study 1 compares estimators across three datasets, while Study 2 varies input dimension and quotient tolerance.

4 Experimental setup

  • The framework is evaluated with fully connected ReLU MLPs trained for 150 epochs, using fixed settings and five seeds per configuration; estimates use held-out data.The setup compares calibration against established MI baselines and tests quotient behavior at growing input dimension.Source: §4 · p. 6

4.1 Study 1 – Calibration against established estimators

  • Study 1 evaluates calibration on Composite, WBC, and MNIST FC using 75 networks spanning the stated dataset-specific architectures.The baseline methods are uniform binning and k-means with Miller–Madow correction, plus mixed continuous–discrete KSG.Source: §4.1 · p. 6, §4 · p. 6

4.2 Study 2 – Saturation and the functional-equivalence quotient

  • Study 2 projects MNIST to seven PCA dimensions, trains networks at four widths, and evaluates the quotient over a grid of 15 tolerance values.This study covers 140 networks and also computes layerwise compression profiles for 40 Study-1 networks.Source: §4.2 · p. 6

5 Results

The routing estimator tracks established MI baselines across networks and layers, while wider networks’ raw estimates saturate as input dimension grows. A functional-equivalence quotient recovers informative estimates where regions share linear operators, with compression depending on tolerance and depth.

5.1 Calibration: pointwise agreement across networks

  • Across datasets, the routing estimator tracks binning and k-means; pooled Pearson correlations are 0.911, 0.922, and 0.892 for binning, k-means, and KSG, respectively.On Composite, KSG is within 0.02 bits of H(Y) for 23 of 30 networks while the routing estimate ranges from 1.777 to 2.629 bits.Source: Figure 2 · p. 7, §5.1 · p. 7
  • At ρ > 0.9, width-25 WBC networks saturate with N = 114 held-out points, and standard bias corrections leave estimates within 0.069 bits of H(Y).Miller–Madow instead over-corrects by several bits, producing negative values.Source: §5.1 · p. 7

5.2 Calibration: layerwise profile at convergence

  • Across all three datasets, routing MI rises monotonically with depth, tracks binning and k-means, and remains at or below H(Y); narrow MNIST networks retain class information within regions.The KSG-above-routing gap indicates that within-region information exceeds routing loss in those cases, and shrinks as deeper layers route more information into the partition.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 Saturation and the functional-equivalence correction

  • At d ≥10 and ε = 0.5, quotient estimates sit 0.266–0.425 bits below H(Y) at width 15, 0.027–0.074 bits below at width 25, and remain saturated at width 50.Raw estimates approach H(Y) for widths 25 and 50 as input dimension grows; large ε can over-compress.Source: §5.3 · pp. 8–9
  • The compression ratio drops at ε ∈ [0.2, 0.5], the range aligned with informative quotient compression; it starts at 1 for every network and decreases with depth on average for ε > 0.Figure 5 reports ρ_func(ε), with lower values indicating more aggressive merging.Source: §5.3 · p. 9, Figure 5 · p. 9

6 Discussion

The calibration tests show strong agreement with discrete MI baselines, while the functional-equivalence quotient can recover routing information when the raw partition saturates. These estimates have important boundaries: calibration does not establish that routing information proxies representation information, and high routing information alone does not establish learned class structure.

6.1 Calibration against established MI estimators

  • Calibration agreement is weak evidence at occupancy ρ = 0.900 and 0.961, where width-25 WBC estimates are pushed toward H(Y), and shared estimator bias remains possible.The routing estimate is not a proxy for I(Y ; T); the two can differ in either direction.Source: §6.1 · p. 9
  • r = 0.911 (binning) and r = 0.922 (k-means) across 75 networks show that routing estimates track discrete baselines across three datasets and a range from 0 bits to H(Y).The routing estimate comes from network geometry, without tuned bin widths, bandwidths, or cluster counts.Source: §6.1 · p. 9
  • A positive KSG-above-routing gap indicates within-region class information may remain unrouted; the gap narrows with depth on all datasets and reverses on Composite by layer 5.KSG overestimation in low-dimensional MNIST FC representations may contribute to the gap; separating contributions requires an independent estimator of I(Y ; T | Π).Source: §6.1 · p. 9
  • No estimate exceeds H(Y) across 75 networks, every hidden layer, and four estimators; the chain is empirically tight on MNIST FC and WBC and approached on Composite.The comparison spans an order of magnitude of label entropy.Source: §6.1 · p. 9

6.2 Functional equivalence as both correction and diagnostic

  • At ε = 0.3–0.5, the functional quotient lowers saturated raw estimates for widths 7–15 when regions implement redundant operators, but not at width 50.Within this window, estimates still vary by up to 0.304 bits; ε is bounded in [0, 2], dimensionless, and auditable through ρfunc.Source: §6.2 · p. 10
  • ρfunc = 1 for a single ReLU layer, while average ρfunc decreases with depth at fixed ε > 0; large drops signal when the quotient materially compresses routing paths.Composite is most compressible, whereas MNIST FC PCA-10 is more resistant at small ε and requires non-trivial ε before merging.Source: §6.2 · p. 10
  • High routing information need not indicate learned class structure: a wide Composite network retained 2.433 raw bits at 15.9% true-label test accuracy, and quotient estimates depend on ε.The quotient discounts the label-permutation effect only at moderate capacity; the authors recommend reading estimates with occupancy and, where possible, a label-permutation control.Source: §6.2 · p. 10

7 Conclusion and Outlook

The paper concludes that CPWL network geometry supports an information decomposition and routing estimator, with a functional quotient extending estimates into fine-resolution regimes. Its appendices specify how regions are represented and describe the construction and class statistics of the Composite dataset.

7 Conclusion and Outlook

  • The framework decomposes I(Y ; T) into routing, within-region, and routing-loss terms, estimates routing information from the partition, and uses a reported, audited ε for quotient estimates in fine-resolution regimes.The estimator agrees with binning and k-means, stays at or below H(Y) across 75 networks, and the quotient recovers informative estimates on PCA-reduced MNIST when raw estimates saturate.Source: §7 · p. 10

A Detailed Formulation of Linear Regions for CPWL Networks

  • Through layer l, the common refinement of layer partitions forms convex linear regions, and the network restriction within each region is affine.For ReLU, each layer's activation pattern records which pre-activations are positive.Source: A Detailed Formulation of Linear… · p. 17
  • A data-supported region is identified by concatenating activation patterns from a forward pass and hashing them; points sharing a hash belong to the same region.The stored representative pattern is also used as input to the functional quotient; the half-space form characterizes region geometry but is not used by the estimators.Source: A Detailed Formulation of Linear… · p. 17

B The 2D Composite Dataset

  • Composite combines two-moons, concentric-circles, and Gaussian-blob generators into one seven-class distribution in a shared two-dimensional domain.It was designed to mix structurally distinct classification geometries in one testbed.Source: B The 2D Composite Dataset · p. 18

B.1 Generators

  • The Composite generators contribute 4,000 moon samples, 4,000 circle samples, and 2,000 blob samples; moons and circles are nonlinearly structured, while the blobs are separated and comparatively easy.The circles enclose the moons without overlap, and the blobs lie outside the circles.Source: §B.1 · p. 18

B.2 Class statistics and label entropy

  • The four moon-and-circle classes each carry 1/5 of the mass and the three blob classes each carry 1/15, yielding H(Y) = 2.639 bits, below log2 7 ≈ 2.807.The held-out stratified split has the same entropy.Source: §B.2 · p. 18

B.3 Properties relevant to partition-based analysis

Composite combines heterogeneous class geometries and remains well-sampled in the reported experiments, while its train/test protocol evaluates partitions on held-out points. The dataset's composition and split provide a controlled setting for examining routing across varied regimes.

B.3 Properties relevant to partition-based analysis

  • Composite mixes corner-concentrated blobs, central moon-shaped curves, and concentric circles, requiring routing across regions with different sizes and shapes.The authors describe this heterogeneous geometry as a stronger stress test for routing estimation than a single-generator dataset.Source: §B.3 · p. 18
  • With two input dimensions and 2,000 held-out points, every Composite network remains well-sampled at deepest layer occupancy ρ < 0.411, while H(Y) = 2.64 bits.This permits tracking routing estimates across a multi-bit range without approaching saturation.Source: §B.3 · p. 19
  • Composite mixes blob classes separable by a few linear cuts with moon and circle classes requiring nonlinear bends, combining tasks expected to be solved at different rates.The source passage attributes earlier solutions to narrow networks and greater depth and training needs to the curved classes.Source: §B.3 · p. 19

B.4 Train/test handling

  • Composite uses a fixed, stratified 80/20 train/test split, and all information-theoretic quantities are computed on held-out data, with N = 2,000.The held-out protocol evaluates partitions on points not used to fit the networks, at the cost of smaller count tables and higher occupancy.Source: §B.4 · p. 19

C The WBC Dataset

WBC is the study's binary, low-entropy dataset with 30 real-valued features and 569 samples. Its small sample size lets the experiments examine how occupancy grows quickly with network width.

C The WBC Dataset

  • WBC contains 569 samples and 30 real-valued cell-nucleus features, with malignant and benign diagnoses as labels.The features are used without dimensionality reduction and scaled to [−1, 1]30.Source: C The WBC Dataset · p. 20
  • The labels comprise 212 malignant and 357 benign samples, with H(Y) = 0.953 bits overall and 0.949 bits on the 114 held-out points.The held-out split is the study's evaluation set for information-theoretic quantities.Source: C The WBC Dataset · p. 20
  • WBC is the study's smallest dataset and only binary-label dataset, so occupancy ρ grows quickly with width.The architectures and training protocol match those used for Composite.Source: C The WBC Dataset · p. 20

D The MNIST FC Dataset

The MNIST FC study adds a 10-class image task to the synthetic and clinical datasets, then develops the joint and per-region information identities used to analyze CPWL representations. Its proofs relate routing, representation, and joint information through chain rules and establish bounds on the joint information.

D The MNIST FC Dataset

  • MNIST FC adds a structurally distinct 10-class image task with label entropy H(Y) ≈ 3.32 bits, alongside the synthetic Composite and small clinical WBC datasets.The input is a genuine image rather than a low-dimensional synthetic vector.Source: D The MNIST FC Dataset · p. 20

D.1 Pipeline

  • Images are projected to 10 PCA dimensions, retaining 48.8% of training-set pixel variance, then min-max scaled componentwise to [−1, 1]; preprocessing is fit on training data.All reported information quantities use the 10,000-sample test set, where H(Y) = 3.319 bits.Source: §D.1 · p. 20

D.2 Architectures

  • Three-layer ReLU networks use widths [3],, and [7]; wider calibration networks are excluded because their partitions approach saturation.The [3] network is also omitted from the layerwise profile because its partition is very coarse and test accuracy is about 60%.Source: §D.2 · pp. 20–21

E.1 Joint factorization and pushforwards

  • Within each region, the activation law is the pushforward of the region-conditional input law through that region’s affine map.The joint law factors using the input-label distribution and point masses enforcing the region assignment and activation.Source: §E.1 · p. 22

E.2 Per-region form of the decomposition terms

  • Within a region, T depends on X through its projection onto row(Aω), so conditioning on T is equivalent to conditioning on that projection.The row-space and kernel components can be statistically dependent under the region-conditional input law, making the conditioning in the per-region form essential.Source: §E.2 · p. 22

E.3 Proof of Theorem 3.1

  • Applying the chain rule to I(Y; Π, T) in both orders yields I(Y; T) = I(Y; Π) + I(Y; T | Π) − I(Y; Π | T).The joint information also equals I(Y; X) − I(Y; X | Π, T).Source: §E.3 · p. 23

E.4 Proofs of Corollaries 3.3 and 3.4

  • The joint information satisfies I(Y; T) ≤ I(Y; Π, T) ≤ I(Y; X), with equality conditions tied to conditional information vanishing.I(Y; T) exceeds I(Y; Π) exactly when I(Y; T | Π) exceeds I(Y; Π | T).Source: §E.4 · p. 23

E.5 Proof of Proposition 3.2

Routing loss is positive when activation-space supports overlap and the regions disagree about class distributions at those activations. The accompanying arguments distinguish this condition from observed-support estimation effects and specify when functional-equivalence merging preserves label information.

E.5 Proof of Proposition 3.2

  • I(Y; Π | T) is positive exactly when distinct regions overlap on a positive-mass set of activations and their conditional label laws differ there.Overlap alone is insufficient: regions with matching conditional class content at shared activations contribute no routing loss.Source: §E.5 · pp. 23–24
  • Differing marginal region-conditional label laws suffice only when Y is independent of T given Π; otherwise disagreement must be assessed at the overlapping activation.Two regions can have identical marginal label laws yet different conditional laws at a shared activation.Source: §E.5 · p. 23

E.6 Truncation gap of the plug-in routing estimator

  • The plug-in estimate targets MI conditional on observing a region, which can lie above or below population I(Y; Π); it approximates I(Y; Π) when unseen-region probability vanishes.In that well-sampled regime, the chain’s lower-bound relationship to I(Y; Π, T) is approximate rather than general.Source: §E.6 · p. 24

E.7 Proof of Proposition 3.5

  • Merging functionally equivalent regions cannot increase MI, and equality requires label distributions to be constant within each equivalence class.For ground-truth labels, the quotient is a lower bound, but equality additionally depends on the data distribution having no label-relevant variation along the operator’s kernel.Source: §E.7 · pp. 24–25

E.8 Relation to prior IB literature

  • Unlike prior approaches that alter or replace I(Y; T), this analysis uses the joint (Π, T) without modifying the model, while tracking the routing-loss term.The passage contrasts the approach with noise injection [15] and prior discussions of deterministic continuous-network MI.Source: §E.8 · p. 25

F Clustering Regions by Functional Equivalence

The quotient groups regions using a bounded relative criterion, while the finite-sample analyses identify occupancy as a diagnostic for when routing estimates are informative. Tested bias corrections do not rescue saturated partitions, whereas quotient merging can help when redundant operators are present.

F Clustering Regions by Functional Equivalence

  • For ε > 0, regions join the first existing class whose representative is within ε; because this relation is not transitive, greedy clustering can depend on visitation order.At ε = 0.3, random visitation orders changed the Miller–Madow-corrected quotient MI by at most 0.148 bits.Source: F Clustering Regions by Functional… · p. 26
  • The relative equivalence criterion is bounded in [0, 2]; ε ∈ [0.2, 0.5] is an operating window, while ε ≥ 1 over-merges and estimates still vary within the window.The paper reports ε with quotient estimates and uses ε = 0.3 as a representative operating point when one value is given.Source: F Clustering Regions by Functional… · p. 26
  • At the first hidden layer, ρ_func = 1 for all tested datasets, architectures, and ε; for ε > 0, the mean ratio decreases with depth, though 232 of 560 curves rise somewhere.Those increases are at most 0.216, so the decrease is not monotone for every network.Source: F Clustering Regions by Functional… · pp. 26–27

G.1 Occupancy of the evaluated networks

G.2 Bias corrections

  • At ρ > 0.3, Miller–Madow subtracts 3.739 bits on average (up to 6.488), producing corrected values as low as −3.169 bits; no tested correction makes saturated partitions informative.At ρ > 0.9, Grassberger, Chao–Shen, and Chao–Wang–Jost corrected values remain within 0.010, 0.058, and 0.069 bits of H(Y), respectively; the corrections fail silently.Source: §G.2 · p. 29, Table 4 · p. 29
  • At small occupancy (ρ < 0.05), every correction is small relative to plug-in seed variability: 0.114 bits versus 0.148 bits on average.The plug-in estimate is described as reliable in this regime; at ρ > 0.3, its seed spread falls to 0.025 bits even as bias grows.Source: §G.2 · p. 29, Table 4 · p. 29

G.3 Held-out versus in-sample estimation

These checks examine estimator stability across sampling choices, baseline hyperparameters, tied representations, label permutation, and quotient-clustering order. They also document implementation and training metrics.

G.3 Held-out versus in-sample estimation

  • At the deepest layer, held-out and in-sample plug-in estimates differ by at most 0.020 bits on Composite and 0.031 bits on WBC.Across all layers, the differences are at most 0.023 and 0.075 bits, respectively; the authors caution that agreement is weaker evidence for saturated WBC architectures.Source: §G.3 · pp. 29–30, Table 5 · p. 30

G.4 Sensitivity of the baselines to their hyperparameters

  • On MNIST FC, Miller–Madow binning with 16 and 30 bins per coordinate correlates negatively with routing estimates (r = −0.570 and −0.888), while the same plug-in variants remain positively correlated.On Composite and WBC, correlations are high except for finer Miller–Madow binnings; among Miller–Madow binnings, K = 8 is not best-correlated on those datasets.Source: Table 6 · p. 30, §G.4 · p. 30

G.5 KSG on networks with collapsed representations

  • Removing the five tied networks leaves pooled KSG correlation nearly unchanged (0.892 versus 0.893), although four tied networks lower the WBC mean KSG curve from 0.837 to 0.619 bits.Without those four networks, KSG stays between 0.823 and 0.840 bits at every layer while the routing estimate rises from 0.675 to 0.905 bits.Source: §G.5 · p. 31
  • 22.8–78.1% of held-out points shared an identical last-layer representation in five WBC networks, where KSG was zero or nearly zero while routing and binning estimates were 0.2–0.9 bits.Ties violate KSG’s neighborhood assumption and bias its estimate downward; routing estimation is unaffected because tied points contribute their label counts to occupied regions.Source: Table 7 · p. 31, §G.5 · p. 31

G.6 Routing information without learned structure: label permutation

  • For the wide Composite network trained on permuted labels, routing information changes from 2.535 to 2.433 bits and remains near H(Y) = 2.639 bits, despite 15.9% test accuracy on true labels.The functional quotient changes from 2.362 to 2.329 bits; the authors report that it removes this effect only at moderate capacity.Source: §G.6 · p. 31, Table 8 · p. 32

G.7 Sensitivity of the functional quotient to the visiting order

  • At ε = 0.3, visiting-order variation in quotient MI reaches 0.148 bits; across ε values 0.1, 0.3, 0.5, and 1, it reaches 0.474 bits.At ε = 0.3, the largest quotient-size spread is 84 classes, or 9.4% of |ΩD|.Source: Table 9 · p. 32, §G.7 · p. 32

H Implementation, Assets and Compute

  • The released code, stored results, and reproduction instructions are MIT-licensed; the CPU pipeline takes about 2 hours of wall time, or about 42 CPU-hours.The implementation uses a locked environment and Dockerfile; the reported compute excludes preliminary experiments and earlier pipeline versions.Source: H Implementation, Assets and Compute · p. 33

I Training Metrics

  • Training figures report held-out test accuracy and cross-entropy loss for the paper’s Composite, WBC, and MNIST networks as five-seed means with ±1σ bands.The figures cover six Study-1 architectures each for Composite and WBC, and the MNIST networks from both studies.Source: I Training Metrics · p. 34, Figure 9 · p. 34, Figure 10 · p. 34

Related papers

Loading 2610.36864v1…