Measuring Learned Equivariance: An Information Theoretic Approach

• Equivariance is useful across several fields, but learned equivariance is not guaranteed and may relate to generalization error on inputs farther from training data.Source: §1 · p. 1
• Prior metrics require specified group actions on hidden features, which may be unknown and force testing candidate actions.Source: §1 · p. 1
• Maximal NMI between original and transformed representations implies a deterministic, invertible map without requiring knowledge of that map.Source: §1 · p. 1
• CKA provides a scalable proxy for high-dimensional activations and is exact under orthogonal feature transformations.Source: §1 · pp. 1–2
• Figure 1 shows CKA detecting equivariance at the correct rotation intervals where LEE fails, including for the SO(2) model.Source: Figure 1 · p. 2
• Across architectures, tasks, and symmetry groups, data augmentation generally increases learned equivariance in standard networks.Source: §1 · p. 2

• Prior output and latent-feature metrics assume known group actions, whereas this paper compares activations under given input transformations without fitting hidden actions.Source: §2 · p. 2
• For finite discrete representations, equivariance requires original and transformed representations to determine one another exactly; the paper uses NMI and CKA to assess this relation.Source: §2 · p. 2
• CKA compares example-similarity patterns without selecting feature directions or fitting an alignment; exact equivariance under orthogonal feature transformations yields maximal CKA.Source: §2 · pp. 2–3
• Because similarity scores may not track behavior and CKA can be biased, the paper tracks transformation consistency with shuffled-pair nulls and complementary checks.Source: §2 · p. 3
• The study examines augmentation-related changes across image-recognition, image-to-image, and point-cloud tasks, comparing whole representations rather than matching individual feature maps.Source: §2 · p. 3

3 BACKGROUND

• The background situates information theory, equivariance, and CKA, including the Hilbert–Schmidt Independence Criterion on which CKA is built.Source: §3 · p. 3

3.1 INFORMATION THEORY

• Mutual information measures shared information, is zero exactly for independent variables, and is normalized here to account for total available information.Source: §3.1 · p. 3

3.2 EQUIVARIANCE

• Equivariance requires transforming the output consistently with the input, while invariance is the special case of an unchanged output.Source: §3.2 · p. 3

3.3 HILBERT-SCHMIDT INDEPENDENCE CRITERION

• HSIC uses kernels to capture relationships between datasets and computes dependence from their centered kernel matrices.Source: §3.3 · pp. 3–4

3.4 CENTERED KERNEL ALIGNMENT

• CKA normalizes HSIC to [0, 1], removing dependence on data scale; 0 indicates no kernel-captured dependence and 1 a perfect statistical relationship.Source: §3.4 · p. 4

4.1 MIME: MUTUAL INFORMATION MEASURE OF EQUIVARIANCE

• MIME tests whether each layer’s original and transformed representations share all information, so each determines the other exactly.The paper names this mutual-information measure of equivariance MIME.Source: §4.1 · p. 4
• NMI below 1 rules out equivariance at the tested group element, while NMI equal to 1 is necessary but not sufficient.The implication follows because equivariance makes each representation a deterministic function of the other through an invertible group action.Source: §4.1 · p. 4

4.2 CKALE: CKA FOR LEARNED EQUIVARIANCE

• In the restricted discretized-image example, the Laplacian’s NMI stays at 1, while Sobel-X reads 1 at 180° despite not being C4-equivariant.Sobel-X is C2-equivariant and anti-commutes; NMI detects its equivariance without being told that the output action is a sign flip.Source: Table 1 · p. 5
• CKALE compares each layer’s activations on original and transformed inputs; any CKA below 1 across the tested rotations rules out exact equivariance at that layer.The method tests every operation for discrete groups and subsamples 16 operations for infinite groups such as SO(2); CKA equal to 1 is necessary but not sufficient.Source: Figure 2 · p. 4, §4.2 · p. 5
• Under orthogonal output transformations and orthogonally invariant kernels using the same feature map, CKA below 1 rules out equivariance at that group element.The stated examples of such kernels are linear and RBF kernels.Source: §4.2 · p. 5

5.1 MUTUAL INFORMATION AS AN EQUIVARIANCE MEASURE

• NMI falls below 1 only for Sobel-X under 90° and 270° rotations, consistent with its C2 equivariance.On the discrete comparison, MI detects lossless alphabet relabeling while CKA treats it as non-equivariant; both flag lossy merging.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 CKA FOR SYMMETRY PROBING

• CKA and MI agree for orthogonal readouts, but CKA can collapse outside that class even when MI does not.For the energy readout, MI/H stays at 1.000, while linear CKA is 0.7379 versus 0.0071 on the control; squared magnitudes introduce cross terms that prevent a matrix feature action.Source: §5.2 · p. 6

5.3 MUTUAL INFORMATION PRECISION PROBLEM

• At full 16-bit precision, MI becomes less trustworthy as its shuffled-null reading rises with the larger alphabet.The authors note that choosing a coarse-graining granularity is difficult in practice.Source: §5.3 · p. 6, Table 5 · p. 7

5.4 CKALE AS AN ASSUMPTION FREE EQUIVARIANCE MEASURE

• LEE reports high error where exactly equivariant models should have zero error, while CKA and LieGG identify the correct behavior.LieGG misses equivariance when a model has vector outputs because it assumes an invariant output; SEIS detects spatial, but not channel, re-encodings and requires an unpooled spatial grid.Source: §5.4 · pp. 7–8
• CKALE and LieGG recover the correct ordering on exactly-SO(2) point clouds, whereas LEE ranks the provably equivariant model last.The comparison uses three measures; lower values indicate greater equivariance.Source: Table 6 · p. 8
• SEIS detects spatial re-encodings but not channel re-encodings, and cannot be evaluated on point-cloud or pooled comparisons.Those settings lack an unpooled spatial grid for the measure to operate on.Source: §5.4 · p. 8, Table 8 · p. 9

5.5 ROBUSTNESS OF CKALE ACROSS MANY DOMAINS

• Augmentation improves final-layer equivariance on most tasks, but not colorization, where texture may matter more than position.Figure 3 reports CKA across models and datasets for C4 and SO(2) tasks; error bars show the 25th and 75th percentiles.Source: Figure 3 · p. 9, §5.5 · p. 9

6 CONCLUSION

• NMI gives a necessary equivariance condition without specifying the feature-space action and is reliable in small discrete settings.CKALE scales across tasks and architectures without discovering or assuming the group action, and is exact under orthogonal feature transformations.Source: §6 · p. 9
• MI cannot be computed exactly in continuous spaces, while CKA is limited to orthogonal transformations and infinite groups are subsampled.The authors identify avoiding subsampling and exploring kernels for nonorthogonal transformations as future work.Source: §6 · p. 9
• The authors report using generative AI tools for coding, mathematical claims and proofs, methodology, experiments, figures, and writing, while reviewing AI-assisted work.They state that they take responsibility for the final content.Source: AI USE STATEMENT · pp. 9–10
• The experiment code is provided in the supplementary materials, alongside descriptions of dataset creation and experiments in Section 5.5.Source: REPRODUCIBILITY STATEMENT · p. 10

A LEMMAS

• NMI(X, Y) = 1 exactly when both conditional entropies, H(X|Y) and H(Y|X), are zero.Thus each representation determines the other without uncertainty.Source: A LEMMAS · pp. 12–13
• Under the nondegeneracy assumption, CKA(KX, KY) = 1 exactly when the centered Gram matrices are proportional by a positive scalar.The centered matrices are defined as HKXH and HKYH.Source: A LEMMAS · p. 13

B.1 A RESAMPLING FLOOR FOR THE CONTINUOUS SWEEP

• Off-lattice rotations interpolate pixel grids, so rising error in a continuous rotation sweep may reflect damaged inputs rather than model non-equivariance.Raw-pixel CKA isolates the rotation operator’s departure from an isometry, with the model removed.Source: §B.1 · p. 13
• 0.04 for C16 to 0.51 for C1, the continuous-sweep peaks are two to three orders of magnitude above the resampling floor, indicating genuine signal.The measured floor is exactly zero at quarter turns and 0.00011–0.00013 elsewhere; model error is about 3× higher, with both quantities of order 10^-4 and falling to zero by a 1132 grid.Source: §B.1 · p. 13

B.2 CKA DOES NOT DETECT ALIASING

• 62%: naive subsampling raises equivariance error from 0.13429 to 0.21735, while prefiltering before subsampling brings it back to 0.15250.Table 10’s setup uses θ = 2.0 rad and 256 MNIST images.Source: §B.2 · p. 13, Table 10 · p. 14
• Linear CKA stays at ∼0.999 across the clean, aliased, and prefiltered feature maps, registering neither the damage nor its repair.The baseline error of 0.13429 reflects an off-lattice rotation and is the interpolation floor.Source: §B.2 · p. 14
• The shuffled control is ∼0.07, but CKA spans only 1.0000 to 0.9980 while the tracked quantity ranges from zero to 0.236.The passage describes this as a range limitation, not a validity failure: interpolation perturbs features while leaving pairwise similarities almost intact.Source: §B.2 · p. 14

C PROGRESSION OF MODELS’ LEARNED EQUIVARIANCE

• The progression views cover CNN, ViT, and perceptron classification models on CIFAR-10, plus CNN and perceptron models on MNIST-Font.The section presents each layer across all epochs.Source: C PROGRESSION OF MODELS’ LEARNED… · p. 14, Figure 4 · p. 15, Figure 5 · p. 15
• CIFAR-10 colorization views show CNN, UNet, ViT, and perceptron models across the full progression.Source: Figure 9 · p. 17, Figure 10 · p. 17, Figure 11 · p. 18
• STL-10 colorization views show CNN, UNet, ViT, and perceptron models across the full progression.Source: Figure 13 · p. 19, Figure 14 · p. 19, Figure 15 · p. 20
• Grid-like buoyancy fluid-dynamics views cover CNN, ViT, UNet, and perceptron models.Source: Figure 17 · p. 21, Figure 18 · p. 21, Figure 19 · p. 22
• Particle-like buoyancy fluid-dynamics views cover PointNet, SetTransformer, MLP, and perceptron models.Source: Figure 21 · p. 23, Figure 22 · p. 23, Figure 23 · p. 24
• Grid-like anisotropic stress-tensor prediction views cover CNN, ViT, UNet, and perceptron models.Source: Figure 25 · p. 25, Figure 26 · p. 25, Figure 27 · p. 26
• Particle-like anisotropic stress-tensor prediction views cover PointNet, SetTransformer, MLP, and perceptron models.Source: Figure 29 · p. 27, Figure 30 · p. 27, Figure 31 · p. 28