PaperlayerView exact revision on arXiv

Source-linked AI summary

From Plato to Aristotle: Toward a General Theory of Representation Hypotheses

Dai Shi, Andi Han, Xiaoyu Li, Luke Thompson, José Miguel Hernández-Lobato

arXiv:2610.20286v1cs.LGstat.ML

TL;DR

Studies of representation convergence reach conflicting conclusions using neighborhood, spanning-tree, and distance-based measures. This paper organizes those measures into a lattice of alignment criteria and uses it to analyze fixed and independently trained models. Its results attribute the disagreements partly to criterion strength, scale, and calibration, while leaving global convergence open.

  • Problem

    Studies of representation convergence use different measures and settings, leaving a general theory relating their potentially conflicting conclusions needed.

    Source: §1 · p. 2

  • Method

    The paper defines alignment criteria and orders them in a lattice, then analyzes their outcomes for pretrained representations and independently trained models.

    Source: §1 · p. 2

  • Results

    Calibration changed the LLaMA-1 CKA size trend from 0.68 to −0.85, while PRH-preprocessed calibrated CKA retained its sign at mid-depth and the last layer.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The lattice framework distinguishes what different alignment criteria establish, and the results leave global convergence open despite evidence of local agreement.

    Source: §1 · p. 2

    The theoretical learning results are proved in a linear autoencoder and finite Gibbs model; tests on pretrained models remain future work.

    Source: §6 · p. 18

Abstract

from arXiv · show

The Platonic Representation Hypothesis (PRH) holds that increasingly capable models converge toward a common representation. However, recent studies have shown different views on aligning representations via their neighborhoods, spanning trees, and distances. We show that these views can differ because the alignment criteria behind them sit at different points of one lattice, and we formalize this explanation as the Lattice Representation Hypothesis (LRH). Under the LRH, the convergence claimed by the PRH is agreement at the top of the lattice, whereas its evidence comes from neighbor overlap, a criterion near the bottom. The Aristotelian Representation Hypothesis (ARH) restricts convergence to local neighborhoods because calibration keeps the growth of neighbor overlap with model size but largely removes that of centered kernel alignment. We show that this removal can come from calibrating a maximum over layers, so global convergence remains open. Back into Plato's Cave (BPC) infers coarse shared structure from the collapse of neighbor overlap on larger evaluation sets, whereas we trace it at least partly to a fixed number of neighbors covering shorter distances. We show that the spanning-tree overlap used by Structure over Geometry (SOG) mostly repeats one-nearest-neighbor overlap. The LRH also extends to independently trained models, whose training procedure and objective decide the criteria met reliably across runs. We prove this claim for the training procedure in a linear autoencoder and for the objective in a finite contrastive Gibbs model. Experiments on dozens of pretrained vision and language models and thousands of runs of both theoretical models support these conclusions. We hope this work paves the way toward a general theory of representation hypotheses, with a framework to analyze them and compare their evidence. Project page: https://eethanshi.github.io/LRH

1 Introduction

The paper proposes that conflicting conclusions about representation convergence arise because studies test criteria of different strengths, rather than necessarily disagreeing about the same kind of alignment. It formalizes this account as the Lattice Representation Hypothesis (LRH) and applies it to pretrained and independently trained models.

1 Introduction

  • The LRH explains conflicting conclusions by locating studies’ alignment criteria at different points in one lattice, with training conditions also shaping reliable agreement across runs.For fixed representations, the criteria accepting a model pair describe what its representations share.Source: §1 · p. 2
  • PRH’s convergence claim corresponds to agreement on all distances at the lattice top, while its mutual-neighbor-overlap evidence tests a criterion near the bottom.The paper identifies mKNN as the lower-level evidence and the top criterion as agreement on all distances.Source: §1 · p. 2
  • Calibrating a maximum over layers can remove CKA’s size trend: on LLaMA-1, calibration changed the trend from 0.68 to −0.85.With PRH preprocessing, calibrated CKA’s language-model-size trend retains its sign at mid-depth and the last layer, leaving global convergence open.Source: §1 · p. 2
  • At fixed k, a larger evaluation set moves the neighbor boundary closer to each input, making BPC’s mKNN collapse at least partly a scale effect.At fixed k/m, mKNN stays close to stable for the reported sentence-encoder and last-layer vision-feature tests.Source: §1 · p. 3
  • On studied vision–language pairs, nearest-neighbor edges make up about 88% of shared MST edges, so SOG’s tree overlap mostly repeats one-nearest-neighbor overlap.The remaining shared edges exceed chance, but their size trend is not established.Source: §1 · p. 3
  • The paper extends the lattice account to independently trained models, proving training-procedure and objective effects in a linear autoencoder and finite contrastive Gibbs model.Preregistered tests support both theorems in those models, while tests on trained networks outside their premises contradict several predictions.Source: §1 · pp. 2–3

2 Alignment Criteria and Their Lattice

This section defines criteria for comparing representations, orders those criteria by strength, and formulates the LRH for fixed representations and repeated training.

2.1 Observables, Protocols, and Criteria

  • Observables capture different relations among samples, and their refinement order places neighbor sets and spanning trees on separate branches below the global order of distances.The criteria’s scale depends on evaluation-set size and neighbor count, while protocols specify preprocessing, layer aggregation, and calibration.Source: §2.1 · pp. 3–4

2.2 Reliable Agreement and the Hypothesis

  • The LRH says accepted criteria describe what fixed representations share, while training conditions determine which criteria are met reliably across runs.The learning condition includes data distribution, sample size, model class, objective, and training procedure.Source: §2.2 · pp. 4–5

3 What Determines the Outcome of a Criterion

The section explains how criterion outcomes depend on the relation being tested, its scale, and its evaluation protocol. It applies this account to neighbor overlap, spanning trees, CKA, and calibration.

3.1 Neighbor Overlap Is Determined by Boundary Crossings at the Scale k/m

  • At fixed k, increasing evaluation-set size moves the neighbor boundary inward, so mKNN tests finer distances and its collapse can be partly a scale effect.At fixed k/m, the paper predicts stable mKNN for large evaluation sets; the crossing law relates replaced neighbors to boundary crossings at each anchor.Source: §3.1 · pp. 5–6

3.2 Spanning Trees Contain the Nearest-neighbor Forest

  • Every MST contains its nearest-neighbor forest, making shared tree edges partly dependent on one-nearest-neighbor overlap, while bridges carry additional structure.The paper bounds MST overlap below by at least half of one-nearest-neighbor overlap; the trend supports SOG’s global-structure conclusion only to the extent it comes from bridges.Source: §3.2 · p. 6

3.3 Unlike Neighbor Overlap, CKA Also Depends on a Monotone Link

  • Unlike neighbor overlap, CKA depends on a monotone link between distance matrices as well as residual disagreement, so shared distance order need not imply CKA agreement.A monotone distortion can preserve neighbor sets and spanning trees while changing CKA; this helps explain why PRH’s top-level claim is not established by mKNN evidence alone.Source: §3.3 · p. 7

3.4 Calibration of a Layer Maximum Can Reverse a Size Trend

  • Calibration of a layer maximum can reverse a size trend when the largest random-pairing baseline grows faster than the aggregated score.The paper links this mechanism to ARH’s loss of the CKA trend, while the calibrated mKNN trend is consistent with the condition under which calibration does not reverse score ordering.Source: §3.4 · p. 8

4 What Determines the Criteria Met Reliably across Runs

Across runs, which alignment criteria are met reliably depends on whether the training procedure selects among equivalent solutions or different objectives converge through different distance links. The linear autoencoder and finite Gibbs model show that criteria can agree at one level of the lattice while failing at another.

4.1 The Training Procedure Selects the Gauge of a Linear Head

  • In the linear autoencoder, gradient flow leaves small-tolerance CKA agreement obstructed as samples grow, while weight decay or tied weights remove that obstruction.Gradient-flow runs reliably meet a projector criterion as samples grow; weight decay selects the gauge under any optimizer reaching a global minimum.Source: §4.1 · p. 9
  • Neighbor sets can survive gauge differences when their relative distance gaps exceed the condition-number distortion between the two solutions.When both minimizers use the same regularized loss, the condition number is 1; at k = 10, typical gaps on a moderate-dimensional manifold exceed one by a few percent.Source: §4.1 · pp. 9–10

4.2 The Objective Decides Whether Contrastive Learners Agree beyond the Order of Distances

  • In the finite Gibbs model, InfoNCE and spectral learners learn the same relation through different increasing links, reliably agreeing on distance order while scale-normalized geometry remains obstructed.Agreement at every positive tolerance is reliable under an isotonic discrepancy; using the same objective also removes the scale-normalized obstruction.Source: §4.2 · p. 10
  • Changing InfoNCE temperature alone rescales its limiting squared distances, so scale-normalized geometry criteria accept the limits at the original and any lower temperature.This holds when the stated limiting matrix is positive semidefinite, as it is for every tA ≤ t.Source: §4.2 · p. 11

4 What Determines the Criteria Met Reliably across Runs

  • The results support the lattice account for two settings: training procedure determines gauge-related agreement in the linear autoencoder, while objective determines agreement across monotone-linked limits in the finite Gibbs model.The authors note that trained networks are outside both theorems’ premises.Source: §4 · p. 9, §4.2 · p. 11

5 Experiments

Experiments test whether scale, observable, and protocol explain differing representation-alignment conclusions, then assess predictions about agreement across independent training runs. Results support scale-dependent neighbor comparisons, nearest-neighbor dominance in spanning-tree overlap, calibration effects, and distinct agreement patterns in controlled training.

5 Experiments

  • The authors test explanations of conflicting alignment findings in pretrained models and predictions about training procedure and objective under controlled training.The study proceeds from growing evaluation sets through spanning trees and CKA calibration, then tests linear autoencoders, CIFAR-5m networks, and the finite Gibbs model.Source: §5 · p. 11

5.1 Experimental Setup

  • The experiments compare 17 vision transformers with BLOOMZ, OpenLLaMA, and LLaMA-1 on 1,024 WIT image–caption pairs, alongside controlled training tests.Controlled-training agreement is classified using one-sided 95% Clopper–Pearson bounds against preregistered reliability thresholds.Source: §5.1 · p. 11, §5 · p. 11

5.2 Neighbor Overlap at a Fixed k Stays Flat or Falls with the Evaluation Size According to the Model Pair

  • At fixed k = 10, median vision–language mKNN falls from 0.091 to 0.0085 as evaluation size grows, while the size-trend slope varies by model-pair class.At shared k/m, medians for k = 100 lie within 4% of k = 10; sentence-encoder pairs also pass the preregistered stability test at shared k/m.Source: §5.2 · pp. 12–13, Figure 2 · p. 12

5.3 Shared Spanning-tree Edges Are Mostly Nearest-neighbor Edges, and Pretrained Vision and Language Representations Differ by More than a Monotone Link

  • On 136 vision–language pairs, the nearest-neighbor forest accounts for median 0.875 and 0.889 of shared MST edges at mid depth and the last layer, respectively.MST overlap tracks one-nearest-neighbor overlap at both depths; beyond an affine fit, a monotone fit explains median shares of only 0.0004 and 0.0012 of squared-distance discrepancy.Source: Figure 2 · p. 12, §5.3 · pp. 13–14

5.4 The Threshold of a Calibrated Layer Maximum Can Be Forecast from Single-model Features

  • Across the tested model families, calibration removes or reverses the CKA size trend when the largest baseline grows at least as fast as the raw score, but preserves it when raw CKA grows faster.For example, LLaMA-1 changes from 0.68 to −0.85, while Pythia changes from 0.93 to 0.89; with PRH preprocessing, calibrated CKA retains its sign at mid depth and the last layer in all four families.Source: §5.4 · p. 14, Figure 4 · p. 15

5.5 The Training Procedure Decides Whether Two Runs of a Linear Autoencoder Agree on Distances

  • In the linear autoencoder, gradient descent without regularization yielded no agreeing pairs on neighbor sets, mean-normalized distances, or CKA, while weight decay, tied weights, or small initialization enabled reliable agreement at N ≥ 2^12.In CIFAR-5m networks, weight decay produced agreement on neighbor sets and mean-normalized distances in 96 of 96 pairs in specified settings, though some preregistered predictions failed.Source: §5.5 · pp. 15–16

5.6 Agreement of InfoNCE and Spectral Learners on the Order of Distances Becomes Reliable in the Finite Gibbs Model

  • In the finite Gibbs model, InfoNCE and spectral learners agreed on the global distance order in 94 of 96 pairs at 234 examples; scale-normalized geometry remained outside tolerance while isotonic discrepancy met it.The predicted sample size for 90% agreement was 233.75; all nine preregistered confirmatory predictions held, but two secondary Monte Carlo predictions failed.Source: §5.6 · pp. 16–18, Figure 6 · p. 17

6 Potential Applications

The applications section uses the lattice perspective to frame representation comparisons across several machine-learning areas. It identifies possible tests and cautions that conclusions from a single score reflect one criterion, scale, and protocol.

6 Potential Applications

  • For multimodal pretraining, the authors propose testing mKNN across values of k/m and checking whether connector-training comparisons are invariant to invertible feature maps.They note that CKA and Gromov–Wasserstein scores change under such maps.Source: §6 · p. 18
  • For post-training and reinforcement learning, the section connects increasing score links to unchanged best-of-N responses but potentially changed group-relative advantages, and relates run agreement to a probability pQ.In the cited three-feature task, replay under a sufficient condition raises pQ to one.Source: §6 · p. 18
  • The authors outline possible lattice-based comparisons for agents, generative models, interpretability, molecular representations, and operator learning.Examples include baseline subtraction for agent embeddings, scale-aware CKNNA comparisons, and gauge effects in autoencoders and operator models.Source: §6 · pp. 18–19
  • Across applications, a conclusion based on one representation score concerns one lattice criterion at one scale and under one protocol.The authors present the lattice view as a framework for comparing such conclusions.Source: §6 · p. 19

A Related Work

Prior studies reach conflicting conclusions about representational convergence using different alignment criteria and experimental settings. This section relates those criteria to their observables, tolerances, and reliability across learning conditions.

A Related Work

B Scores, Observables, and the Four Studies

  • When calibration aggregates a maximum over layer pairs, it recomputes that maximum under each permutation, and the maximum’s upward bias grows with the number of layers.The protocol therefore fixes one representation map per model and treats layer aggregation as part of the protocol.Source: B Scores, Observables, and the Four… · p. 27

C Supplements to Section 2

  • The supplements bound cross-condition agreement for exact criteria and inner-product expected scores using within-condition agreement probabilities, and define transitions for interventions.The transition definition uses a weaker and a stronger criterion, a probability level, and a separation parameter.Source: C Supplements to Section 2 · p. 28

C.1 Refinement and Criteria with Positive Tolerances

  • Positive-tolerance criterion strength depends on both the observable and its tolerance, not observable refinement alone.Linear CKA can accept independent wide representations at small tolerance, while mKNN remains at its chance-level expectation.Source: §C.1 · p. 28

C.2 The Lattice of Criteria

  • Observables correspond to partitions ordered by refinement, while criterion joins and meets combine normalized discrepancies using maximum and minimum, respectively.A distance-error bound implies neighbor agreement only where the neighbor gap exceeds twice the error; criteria with different neighbor counts or protocols are generally unordered.Source: §C.2 · p. 29

C.3 Shared Corpus and the Within-condition Ceiling

  • Across learning conditions, exact-criterion agreement is bounded by the geometric mean of within-condition seed-pair probabilities, but positive-tolerance criteria lack that bound.With a shared corpus, agreement is averaged over that corpus; within-condition expected scores do not decrease.Source: §C.3 · pp. 29–30

C.4 Transitions

  • More data leaves small-tolerance CKA obstruction in place, whereas weight decay removes it; using one objective removes asymptotic scale-normalized-geometry obstruction across different objectives.Interventions change learned representations under a fixed criterion; changing the criterion instead changes which representational differences count as disagreement.Source: §C.4 · p. 30

C.5 Proof of Proposition 1

The supplements establish which representation criteria refine others and where their implications fail. They also relate neighbor overlap to distance-scale effects, distinguish ordinal agreement from geometric agreement, and show how calibration can reverse comparisons of layer maxima.

C.5 Proof of Proposition 1

  • Positive affine distance changes preserve rankings, while the MST refines the ordered edge structure and contains each input’s nearest-neighbor edge.Counterexamples show that rankings do not determine the MST, and an MST does not determine k-nearest-neighbor sets.Source: §C.5 · pp. 30–31
  • The same rankings can yield different MSTs, while the same MST can accompany different k-nearest-neighbor sets.The constructions establish that neither relation determines the other in general.Source: §C.5 · p. 31

D.1 Rows, Margins, and the Margin Form

  • Neighbor overlap depends on crossings at the neighbor boundary, with expected crossings shaped by distance density near that boundary relative to coupling dispersion.A row-wise bound based on the largest discrepancy becomes less certifying as the neighbor gap shrinks with evaluation-set size.Source: §D.1 · pp. 31–32

D.2 Scale Classes

  • Under the stated scale-class conjecture, fixed k yields overlap tending to one for conformal embeddings, an intermediate limit for nonconformal embeddings, or zero when relative dispersion diverges.At fixed k/m, the neighbor radius stays fixed and the first-order law predicts stable overlap; the derivation is heuristic pending a local Poisson limit.Source: §D.2 · p. 33

D.3 Ordinal Agreement without Geometric Agreement

  • A strictly increasing kernel transformation can preserve all observables no stronger than global distance order while CKA ranges from one toward a participation-ratio-determined value.When effective dimension is small relative to sample count, representations can agree on neighbor sets, rankings, and spanning trees while CKA is close to zero.Source: §D.3 · p. 33

D.4 Aggregation before Calibration

  • Calibration of layer maxima can reverse their ordering when baseline and threshold growth outweigh the difference between the uncalibrated maxima.The formal result also gives a condition under which the larger uncalibrated maximum retains the larger calibrated score, or the scores both become zero.Source: §D.4 · p. 34

D.5 Proofs

The proofs establish bounds and invariances for neighbor overlap, relate spanning-tree overlap to nearest-neighbor structure, and characterize how monotone links and residual noise affect CKA. They also give results on reliable agreement across learning conditions and distinguish agreement from recovery in the finite Gibbs model.

D.5 Proofs

  • A balancing threshold equates expected incoming and outgoing neighbor crossings, and the expected replacement count is invariant to strictly increasing row-wise transformations.The conditional expected replacement count is bounded through the balanced crossing count.Source: §D.5 · pp. 34–35
  • Every minimum spanning tree contains each model’s nearest-neighbor forest, yielding a lower bound on shared tree edges from one-nearest-neighbor overlap.The proof relates the shared edges to the number of inputs with matching nearest neighbors and mutual-neighbor pairs.Source: §D.5 · pp. 35–36
  • Under the stated bounded-moment assumptions, CKA for a noisy monotone link approaches CKA for the monotone link as the evaluation size grows.The residual entries are independent with mean zero and variance bounded away from zero and infinity.Source: §D.5 · p. 36
  • An entrywise exponential transform preserves distance order and observables below the all-distance ordering, while CKA varies with the transform parameter.As β approaches zero, CKA approaches 1; as β grows, the transformed Gram matrix approaches the identity.Source: §D.5 · p. 36

E Learning Conditions: Full Statements and Proofs

E.1 The Finite Gibbs Model

  • In the finite Gibbs model, recovery bounds transfer to agreement, but two inaccurate models can still agree on the same incorrect relation.Training examples pair anchors with positive partners and independently sampled negatives.Source: §E.1 · pp. 38–39

E.2 Sample Size: Co-occurrence and Fixed-target Transitions

This section examines how co-occurrence, model capacity, optimization, and objective shape agreement among representation criteria. It also considers how temperature and monotone links affect discrepancies in the finite Gibbs model and documents the experimental protocol.

E.2 Sample Size: Co-occurrence and Fixed-target Transitions

  • Information about a pair’s similarity decays exponentially with its target distance, so reliable agreement on distant-pair distances requires more examples.The required sample size is at least of order t^2/(ε^2 P_s^+(x, x′)) for the least co-occurring pair under a maximum-distance-error tolerance ε.Source: §E.2 · p. 39
  • Agreement on exact observables becomes reliable in lattice order as sample size grows, but a target on a criterion-changing hyperplane leaves asymptotic agreement at 1/2.Coarser observables have at least as large a margin as finer ones; when the target lies on a single separating hyperplane, exact agreement is asymptotically obstructed for δ + η ≤ 1/2.Source: §E.2 · pp. 39–40

E.3 Model Class: Rank

  • A representation of dimension n_h incurs distance error of at least 2χ_nh(S) against an accurate model, regardless of sample size, budget, or optimizer.Two models constrained in the same way can nevertheless agree on the same biased relation if their population optimum is unique and empirical risk minimizers are consistent.Source: §E.3 · pp. 40–41

E.4 The Gauge of a Linear Head

  • With gradient flow, independent initializations yield probe Gram matrices with CKA below 1 almost surely; weight decay or tied weights instead make the probe Gram matrix unique.For independent samples, the relevant projectors and, under weight decay or tied weights, probe Gram matrices converge in probability to common limits as sample size grows.Source: §E.4 · p. 41
  • Two linear-head minimizers share each probe’s k-neighbor set when its kth-to-(k+1)th distance gap exceeds the condition-number-scaled kth distance; weight decay or tied weights gives condition number 1.For gradient-flow limits, the condition number is bounded in terms of the initial values of the conserved quantity C.Source: §E.4 · p. 43

E.5 Objectives with Different Monotone Links

  • With different objectives, InfoNCE recovers the target distance matrix while spectral contrastive learning converges to its concave entrywise link; exact criteria agree, but normalized geometry and CKA can differ.For the stated positive-definite target with pairwise-distinct off-diagonal distances, sufficiently tight normalized-distance discrepancy criteria have agreement probability tending to zero, while CKA converges to a value below 1.Source: §E.5 · p. 43
  • Fitting a nondecreasing entrywise link makes the two objectives’ distance matrices agree asymptotically, while learners with the same objective agree under every continuous discrepancy vanishing at equality.Because fitted monotone links include the links induced by objective changes, the geometric discrepancy removed by fitting one upper-bounds the share explained by such a change.Source: §E.5 · pp. 43–44
  • With at least three probes having distinct distances, different monotone links yield positive normalized-distance discrepancy, and their limiting CKA is below one.The discrepancy converges in probability to a positive value.Source: §E.5 · p. 44
  • When tA ≤ t, an InfoNCE learner converges to (tA/t)K⋆ + (1 − tA/t)11⊤, with squared distances scaled by tA/t and accepted by the described positively homogeneous criteria.The convergence holds when the matrix is positive semidefinite; it is positive semidefinite for every tA ≤ t.Source: §E.5 · pp. 44–45

F Experimental Details and Full Results

  • Appendix F records the experimental protocol, compute, preregistration history, results, and failed preregistered predictions, with frozen result files generating the tables.Ten frozen files are withheld for now.Source: F Experimental Details and Full Results · p. 46

F.1 Protocol

This appendix records preregistered protocols and outcomes across pretrained models and controlled-training experiments, alongside deviations and design errors.

F.1 Protocol

  • Preregistered plans froze predictions, settings, and decision rules; controlled-training agreement used one-sided 95% Clopper–Pearson bounds on disjoint run pairs.The protocol also distinguished exploratory and post hoc analyses from tests.Source: §F.1 · p. 46
  • Calibration changed the layer-max CKA trend from 0.68 to −0.85 for LLaMA-1 and from 0.88 to 0.18 for OpenLLaMA, while mKNN shifted −0.03 and 0.00, respectively.Fixed-depth CKA kept its sign in both families.Source: §F.1 · p. 47
  • Pythia’s preregistered calibration prediction was not confirmed: CKA changed 0.93 to 0.89 (drop 0.034; pperm = 1612/5040), so secondary tests were not run.mKNN and debiased CKA each moved by 0.00.Source: §F.1 · p. 47
  • All 15 sentence-encoder pairs met the evaluation-size prediction, with elasticities within ±0.05 and a median of 0.006.The test concerned k = 10 and k = 100 on COCO evaluation sets.Source: §F.1 · p. 47
  • The linear-autoencoder prediction was falsified in 12 of 12 cells, while neighbor sets, Gs, and CKA were reliable at N = 216 under weight decay, tied weights, and small initialization.Projector-criterion agreement was reliable under every tested training procedure.Source: §F.1 · p. 47
  • The crossing-law predictor was not tested on real data because every candidate estimator failed the preliminary check.The predictor was reported as unvalidated.Source: §F.1 · p. 47, §F.2 · p. 50

F.2 Preregistration, Gates, and Deviations

  • Audit qualifications include LLaMA-1 freeze ordering based on file timestamps and post hoc acceptance of Pythia’s agreement check after its decision category was seen.The automated LLaMA-1 order check inspected an unused directory.Source: §F.2 · p. 50
  • The authors report seven Stage 2 design errors that leave reported numbers and decisions unchanged; with a fixed budget, smaller samples also meant more epochs.The contrastive falsification threshold ruled out the theorem’s sharp link but not a general monotone link.Source: §F.2 · pp. 50–51

F.3 Evaluation Size

Neighbor overlap generally falls as evaluation sets grow at fixed k, but stays close to stable at fixed k/m for sentence encoders. In fixed-depth pretrained-model tests, spanning-tree overlap mostly repeats one-nearest-neighbor overlap, while size trends depend on model family, layer, and preprocessing.

F.3 Evaluation Size

  • At k = 10, the median neighbor-overlap slope is −0.01 for seed pairs, −0.21 for vision pairs, and −0.57 for vision–language pairs, versus −1.00 for chance.The preregistered rising class appeared in 0 of 87 pair-and-k combinations; the authors note that the slope rule fails to detect it at these overlap levels, leaving the conformal case untested.Source: §F.3 · p. 51
  • The crossing-law predictor recovers all 29 slope classes at k = 10 descriptively, but its vision–language slopes are too shallow and its preregistered test was not run.The preliminary check failed because every candidate estimator lost to a null on synthetic couplings generated by that null; the predictor is therefore unvalidated.Source: §F.3 · p. 52, Figure F.1 · p. 52
  • Sentence-encoder overlap falls from 0.105 at m = 256 to 0.013 at m = 16,384, yet is close to stable at shared k/m under the preregistered test.All 15 pairs qualified for the stability test; its elasticity-sum median was 0.006, and the result held across the main and 11 further arms.Source: §F.3 · p. 53
  • Last-layer vision features raise sentence-encoder overlap by a median factor of 4.3 at k = 100 and m = 10,000, while overlap still falls with evaluation size.These exploratory follow-ups use COCO, five ViT-B models, and language models of at most 3B parameters; vision pairs were not scored at the last layer.Source: §F.3 · p. 54

F.4 Pretrained Models at Fixed Depth

  • On the primary evaluation set (m=2047), the preregistered rule classified shared MST overlap as mostly one-nearest-neighbor overlap at both depths; bridge overlap exceeded its permutation null in 93% and 100% of pairs.Forest-share medians were 0.875 and 0.889, versus 0.850 and 0.852 expected if shared edges were a random subset of the language-model tree.Source: §F.4 · pp. 54–55
  • The in-sample global-monotone fit adds a median share below 0.10 at both depths, and the bound certifies zero anchors at k = 10 in all 544 configurations.The authors attribute neighbor overlap on these pretrained pairs to typical crossings rather than worst-case margin.Source: §F.4 · p. 55
  • At fixed depth, size trends depend jointly on model family, layer, and preprocessing: BLOOMZ CKA trends reverse under plain ℓ2 normalization but rise with the PRH clip at mid depth.For BLOOMZ, the clip yields CKA and mKNN trends of 0.70 and 0.84; plain ℓ2 yields −1.00 and −0.92.Source: §F.4 · p. 55
  • A 128-component projection does not drive the fixed-depth CKA trend, and the preregistered rule classifies the result as “genuine global loss” despite persistent trends with the PRH clip.With the clip, the trends are 0.70 at mid depth and 0.92 at the last layer; projection changes the latter by −0.07.Source: §F.4 · p. 55

F.5 Layer Aggregation

The layer-maximum experiments test whether calibration changes size trends in CKA and mKNN, comparing calibrated maxima with random-pairing baselines. Across families, calibration can remove or reverse CKA trends when the threshold grows with the score, but the outcome varies by family and protocol.

F.4 Pretrained Models at Fixed Depth

  • On WIT, MST overlap is classified as mostly one-nearest-neighbor overlap at both depths, while bridge overlap exceeds its permutation null in 93% and 100% of pairs.The bridge trend is not established: after Holm correction, the last-layer estimate does not meet the required threshold.Source: Figure F.3 · p. 56

F.5 Layer Aggregation

  • The layer-maximum protocol recomputes the maximum over layer pairs under each random input permutation, so calibration uses a threshold tied to the largest random-pairing baseline.The experiments evaluate 80,960 layer pairs across the preregistered vision–language comparisons.Source: §F.5 · pp. 55–56
  • For BLOOMZ, the layer-max CKA trend of 0.89 falls to −0.02 under scal, as the mean threshold rises by 0.048, matching the observed maximum’s gain.The threshold and largest predicted baseline rise together; the threshold of the layer pair attaining the maximum remains between 0.107 and 0.121.Source: §F.5 · pp. 56–57, Figure F.4 · p. 57
  • For LLaMA-1, calibration changes the layer-max CKA trend from 0.68 to −0.85 under scal, while the threshold gains 0.033 against 0.014 for the observed maximum.The trend remains reversed when subtracting the selected-pair threshold or restricting the maximum to later layers.Source: §F.5 · p. 57
  • For OpenLLaMA, the CKA trend of 0.88 falls to −0.59 after permutation-mean subtraction as the threshold and largest baseline gain 0.022, exceeding the observed maximum’s 0.016 gain.The mKNN trend stays at 1.00 under calibration, and calibration before the maximum lowers the CKA trend to 0.21.Source: §F.5 · p. 58
  • For Pythia, calibration changes the layer-max CKA trend from 0.93 to 0.89, so the preregistered prediction of a drop of at least 0.10 was not confirmed.The threshold order matches the predicted order in 307 of 307 material comparisons, and predicted post-calibration orders are correct in 352 of 357 comparisons.Source: §F.5 · p. 59
  • The LLaMA-1 and OpenLLaMA size-trend outcomes are directional results from three sizes, with intervals covering vision-model choice but not language-model choice.The joint family tests are exploratory, and the OpenLLaMA plan declared its outcomes descriptive.Source: §F.5 · p. 59
  • Under plain ℓ2 normalization, calibration moves layer-max CKA trends from −0.50 to −0.56 for LLaMA-1 and from −0.26 to −0.62 for OpenLLaMA.These secondary-preprocessing analyses were declared not material before scoring because predicted baselines differed too little between sizes.Source: §F.5 · p. 59

F.6 Comparison with the Published Results

The published studies’ conclusions differ in part because their measures and settings differ; controlled experiments further show that a linear head can add geometric discrepancy without reducing neighbor agreement. In linear autoencoders, cross-run agreement depends on the training procedure, with unregularized gradient descent remaining unreliable as sample size grows.

F.6 Comparison with the Published Results

  • Across published-model comparisons, calibration and preprocessing effects vary by models and settings, while the source of some differences from ARH remains open.The BLOOMZ mid-depth trend reverses under plain ℓ2 normalization, making preprocessing part of the criterion.Source: §F.6 · p. 60
  • In CIFAR-5m networks, a linear head adds geometric discrepancy without lowering neighbor-set agreement, and weight decay removes that geometric excess.Under momentum without weight decay, a gauge shared across runs accounts for most of the excess at 250,000 images.Source: §F.6 · pp. 60–61

F.7 Linear Autoencoder

  • At every sample size, unregularized gradient-descent autoencoder runs remain unreliable on neighbor sets, Gs, and CKA, while weight decay, tied weights, and small initialization yield reliable agreement as N grows.At N = 2^16, at least 47 of 48 pairs agree under those three procedures on all four criteria; the projector criterion is reliable for N ≥ 2^12 under every procedure.Source: §F.7 · p. 61, Figure F.5 · p. 62
  • Minibatch SGD and Adam change the imbalance outside the theorem’s premises, and their run agreement on neighbor sets, Gs, and CKA remains unreliable at every N.At N = ∞, only 164, 12, and 322 of 500 pairs meet tolerance under SGD, and 13, 0, and 172 under Adam, respectively.Source: §F.7 · p. 61
  • The preregistered prediction that gradient-descent runs would have mean discrepancy at N = 2^16 inside the N = ∞ interval fails for Gs and CKA, although paired differences have 95% intervals containing zero.The conclusion that more data leaves agreement unreliable rests on the prediction’s separate first part, which held in all 12 cells.Source: §F.7 · pp. 61–62

F.8 CIFAR-5m Networks with a Linear Head

In CIFAR-5m networks, weight decay makes linear-head representations agree across all nine criteria at 250,000 images, while without weight decay neighbor-set agreement can coexist with geometric disagreement. The size and initialization effects depend on the tolerance and on whether the experiments satisfy the theorem’s premises.

F.8 CIFAR-5m Networks with a Linear Head

  • At 250,000 images, weight decay yields agreement of u in 96 of 96 pairs under all nine criteria, while without it u agrees on 10-nearest-neighbor sets in 96 of 96 but not reliably on scale-normalized geometry.The neighbor tolerance is loose at 0.84; Gs agreement is unreliable in 65 of 96 pairs, and CKA is inconclusive in 73 of 96.Source: §F.8 · pp. 62–63
  • Under small initialization, agreement is inconclusive on Gs and CKA, and the prediction that its agreement fractions stay within 0.1 of weight decay fails on Gs at 250,000 images.The fractions differ by 0.146 on Gs; at 25,000 images no pairs meet tolerance under any of the three procedures, with tolerance set by 100,000-image pilots.Source: §F.8 · p. 63
  • Momentum changes the head imbalance during training, while the projector criterion compares column spaces and yields reliable agreement at 250,000 images in every arm.The relative imbalance is 0.31 without weight decay, 0.040 with weight decay, and 0.28 under small initialization.Source: §F.8 · p. 63
  • Without weight decay, u’s Gs discrepancy exceeds y’s in all 96 pairs at both sizes, and weight decay removes this excess while small initialization retains most of it.At 250,000 images, the median excess is 0.020 without weight decay, −0.0001 with weight decay, and 0.016 under small initialization.Source: §F.8 · p. 63
  • A gauge shared across runs accounts for about three quarters of the Gs excess without weight decay at 250,000 images, while the remaining mismatch plus output discrepancy stays within tolerance in all 96 pairs.The authors note that the preregistered decision on u depends on both the shared gauge and run-to-run mismatch.Source: §F.8 · p. 63
  • In the linear-autoencoder runs, the corollary’s certificate covers at least 94% of anchors at k = 10 under weight decay, tied weights, and small initialization, but none under gradient descent at step 0.002.The latter procedure has median condAB = 8.9, and 99% of its 10-neighbor sets differ across run pairs.Source: §F.8 · p. 64
  • In CIFAR-5m networks outside the theorem’s premises, the certificate covers at most 5% of k = 10 anchors, despite no violations among the covered anchors.A head-gauge exchange moves 2–6% of neighbor slots, while outputs from two runs without weight decay share a mean of 0.19 of their 10 nearest neighbors.Source: §F.8 · p. 64
  • The neighbor-set decisions do not evidence the gauge corollary, because changing the gauge alters their mean cross-run discrepancy by at most 0.0002 in every cell.The decisions instead reflect the learned functions; the same qualification applies to failed prediction P1.Source: §F.8 · p. 65

F.8.1 Second Experiment: Head Flow and Same-size Tolerances

The second network experiment separates head-flow, momentum, and weight-decay effects on coordinate discrepancies, while the finite Gibbs model and CIFAR-5m networks test whether contrastive objectives yield shared representations. The finite model exhibits the predicted agreement on order criteria, but trained CIFAR-5m networks disagree across objectives even on neighbor sets.

F.8.1 Second Experiment: Head Flow and Same-size Tolerances

  • Momentum and finite steps both changed C, so the experiment did not identify momentum alone as the main cause; bfloat16 precision did not explain the earlier imbalance growth.The median imbalance was 0.3121 with bfloat16 versus 0.3119 with float64.Source: §F.8.1 · p. 66
  • Under head flow, the imbalance contribution was 0.19 and 0.18 times the momentum contribution on Gs at the two sizes; weight decay removed the Gs excess.On 1−CKA, the corresponding head-flow ratios were 0.39 and 0.48, while weight decay did not consistently remove the excess.Source: §F.8.1 · p. 66, Figure F.7 · p. 66

F.9 Contrastive Objectives in the Finite Gibbs Model

  • At 234 examples, exact agreement across objectives on probe order held in 94 of 96 pairs, rising to 96 of 96 at 237 and 240, as predicted by the finite Gibbs model.At 240, all pairs also met the specified tolerance within each objective, while no cross-objective pair met the tolerance on scale-normalized geometry.Source: §F.9 · pp. 67–68
  • The finite-model order-criterion discrepancy across objectives stayed below the sum of the within-objective discrepancies, but the geometric criteria Gs and 1−CKA exceeded that sum.At 240 examples, 1−CKA discrepancy across objectives on all inputs reached 0.14135, near its 0.1414 limit.Source: §F.9 · pp. 67–68

F.10 Contrastive Networks

  • Across CIFAR-5m runs, every InfoNCE–spectral pair missed tolerance on neighbor sets, rankings, global order, and spanning trees at both sizes.At 500,000 images, median discrepancies on these criteria were 1.34 to 2.51 times tolerance; pairs within a common objective met every criterion.Source: §F.10 · p. 69
  • The contrastive-network setup fell outside Theorem 3’s premises, including because the spectral arm used unit-sphere embeddings and the embedding dimension was far below the input-domain size.The InfoNCE empirical risk also had no positive definite minimizer at any temperature.Source: §F.10 · p. 69
  • The contrastive-network link prediction was falsified: the cross-objective median warp share was 0.147 at 500,000 images, and the link explained −0.04 of the median pair’s discrepancy.The conclusion that these networks differ by more than a monotone link rests on order criteria and isotonic discrepancy; Theorem 3 remains a finite-model result.Source: §F.10 · p. 69

G Limitations and Open Problems

  • The appendix identifies unresolved explanations for the contrastive networks’ differing representations, including rank or spectrum changes and a possible common temperature path.The experiments also report that more data shrank near-pair log-distance error less than far-pair error in all three arms, contrary to the prediction.Source: §F.10 · p. 70

G.1 Limitations

The paper bounds its conclusions by the tested models, protocols, and statistical evidence, and identifies open measurement and learning questions. It proposes tests and conjectures for scaling, objectives, pretraining runs, and multimodal connectors, while noting that these proposals are not proved here.

G.1 Limitations

  • The mKNN predictor based on the crossing law was not validated because every candidate coupling estimator failed the preliminary baseline check, so the preregistered out-of-sample test was not run.On vision–language pairs, the predictor also estimated the fall with evaluation-set size too slowly and performed worse in level than a no-change prediction.Source: §G.1 · p. 71
  • The reported size trends each use one draw of evaluation inputs, and the calibration-related CKA trend loss still awaits a confirmatory test on a qualifying model family.On Pythia, the confirmatory family, calibration kept the trend.Source: §G.1 · p. 71
  • The lattice criteria require paired inputs and therefore do not test whether correspondence can be identified from each representation space alone..Source: §G.1 · p. 71

G.2 Open Problems

  • Predicting mKNN at larger evaluation sizes requires a law for how coupling dispersion changes across scales, alongside the local Poisson limit needed for Conjecture D.1.Calibration also needs baselines that preserve shared structure for reasons unrelated to convergence, such as shared dependence on caption length.Source: §G.2 · p. 72
  • How momentum, finite steps, and adaptive optimizers change the imbalance C remains open; a law for that change could predict reliably met criteria and network imbalance disagreement.Gradient flow conserves C in Theorem 2, unlike these practical training procedures.Source: §G.2 · p. 72
  • A model where changing contrastive objectives changes the learned relation itself is still lacking, despite the trained-network results motivating one.Such a model could also test when agreement is reliable on neighbor sets but not spanning trees, or the reverse.Source: §G.2 · p. 72

H.1 Pretraining, Scaling, and Multimodal Encoders

  • The proposed SigLIP limit is a conjecture, and the appendix’s potential-application tests are left to future work rather than proved in this paper.The conjecture predicts convergence toward an affine transform of K⋆ under its finite Gibbs model assumptions.Source: H Extended Potential Applications · p. 72, §H.1 · p. 73
  • A preregistered scaling test could examine whether mKNN trends reflect near-neighbor or coarse-structure agreement, and whether calibrated CKA’s random-pairing baseline grows faster than its raw maximum.The proposed CLIP-model comparison would hold depth and width fixed while training data grow.Source: §H.1 · p. 72

H.2 Post-training, Fine-tuning, and Reinforcement Learning

The discussion applies the lattice view to post-training, agents, generative models, interpretability, scientific models, and operator learning. It uses this framework to distinguish kinds of representation agreement and to propose tests for when they hold.

H.2 Post-training, Fine-tuning, and Reinforcement Learning

  • LoRA updates are gauge-invariant, but factor-based comparisons and adapter compositions can depend on the chosen gauge.The authors propose balancing and orthogonally aligning adapters; this recovers a shared update when two adapters learn one update, while improvement for different tasks remains a hypothesis.Source: §H.2 · pp. 73–74
  • The discussion proposes separating order changes from geometry changes when comparing RL with SFT, and link effects when comparing reward models.Rank correlation and residuals around an isotonic link are proposed for fine-tuned versus base models; an isotonic map between classifier and Bradley–Terry scores would measure the link's effect on GRPO policy gaps.Source: §H.2 · p. 74
  • Across RL runs, representation agreement can depend on which policy each run reaches; replay is a candidate intervention for making agreement reliable.In the three-feature task, exact adaptation yields agreement probability pQ = q^2 + (1 − q)^2, which is at most 0.8 for q in [0.113, 0.887]; replay can bring runs with a positive initial probability of the better branch to that branch when its reward advantage exceeds 7/16.Source: §H.2 · p. 74

H.3 Agents and Harness Engineering

  • For agent systems, baseline-corrected cosine similarity or mKNN could replace raw output similarity, while harness comparisons can range from equal success rates to equal solved-instance sets.Internal-state communication could also be tested with calibrated scores and mKNN; for least-squares linear channels, canonical correlations are expected to predict gain better than CKA when probe sentences outnumber dimensions.Source: §H.3 · p. 75

H.4 Generative Models

  • Whether CKNNA trends in generative models hold across evaluation scales remains open, and patch-similarity losses can change even when neighbor order is preserved.The discussion proposes testing transformed target similarities or order-based losses; it also notes that which representation criteria independent diffusion-model runs meet reliably remains unknown.Source: §H.4 · pp. 75–76

H.5 Mechanistic Interpretability

  • The discussion distinguishes recurring subspaces from recurring individual latents and expects initialization and regularization to matter more for agreement on directions than on subspace projectors.It also connects scale-dependent top-token overlap to neighbor-set criteria and predicts that suitable random projections can preserve neighborhood overlap under superposition even where CKA is low.Source: §H.5 · pp. 76–77

H.6 Proteins, Molecules, and Materials

  • Conflicting interatomic-potential alignment results may reflect different criteria, data, feature levels, or projections; the discussion proposes comparing criteria on shared samples and features.It also notes that uncalibrated convergence-with-performance trends may change after baseline correction, that protein comparisons can be affected by homology, and that weight decay removed excess feature-distance differences in the paper's CIFAR-5m networks.Source: §H.6 · p. 77

H.7 Dynamical Systems and Operator Learning

  • For operator learning, equivalent predictions can coexist with gauge-dependent factors, while dynamics-based criteria may accept networks rejected by geometry-based criteria.The discussion proposes resolution-transfer tests at a chosen neighbor scale and criteria indexed by chaos order; fixed Wick features remove the gauge in some operators, whereas it remains in variants multiplying two learned factors.Source: §H.7 · pp. 77–78

Related papers

Loading 2610.20286v1…