Conditional Predictive Sufficient Statistics for Visual Representation Learning

• Predicting future content has progressed from pixels and discrete tokens to latent embeddings, but these objectives share the goal of retaining what future observations confirm.Source: §1 · p. 1
• A shared-factor model makes predictive information about the next patch informative about the latent scene factor, defining a conditional predictive sufficient statistic.The next patch is conditionally independent of the past given the shared factor.Source: §1 · p. 1
• Cosine next-embedding prediction is directional maximum likelihood under a von Mises–Fisher model of the next embedding.Source: §1 · pp. 1–2
• A constant embedding remains a global population-risk minimizer, while a shallow target predicts that the sufficient statistic is read from intermediate activations.Stop-gradient blocks the symmetric gradient but does not remove the constant solution.Source: §1 · p. 2
• Small causal-Transformer experiments on MNIST and CIFAR-10 test whether future shift, stop-gradient, target choice, and layer index move probe accuracy as predicted.The experiments are diagnostics, not a claim about ImageNet accuracy.Source: §1 · p. 2

• CPSS treats the next observation as the relevance variable for predictive information, without labels or hand-specified augmentations.The shared-factor assumption plays the bottleneck role of an explicit I(x; z) penalty.Source: §2 · p. 2
• Earlier visual pretexts predict context or transformations, while contrastive and Siamese methods estimate mutual information or use stop-gradient and teacher asymmetries.The paper specializes the collapse concern to a causal next-embedding loss whose predictor is the whole backbone.Source: §2 · p. 2
• NEPA predicts the next patch embedding from the model’s own embedding layer with cosine loss and target stop-gradient; CPSS derives this objective’s implications for intermediate representations.Unlike masked joint-embedding prediction, it removes the auxiliary target encoder and pixel head.Source: §2 · p. 2

3.1 Shared-factor patches

• Under conditional independence of patches given shared scene factor s, predictive information equals information about s plus what the next patch fails to reveal.Private noise does not increase predictive information; a representation preserving the predictive sigma-algebra is sufficient for predicting the future and, under the model, for s.Source: §3.1 · p. 3

3.2 Cosine loss as directional likelihood

• Fitting the cosine loss is maximum likelihood for a von Mises–Fisher conditional law of the next embedding direction, and raises a variational surrogate of predictive information.It is not the InfoNCE bound.Source: §3.2 · p. 3

3.3 The constant embedding is still optimal

• A constant nonzero embedding reaches the pointwise minimum cosine loss, so low training loss does not establish that the representation preserves s.Stop-gradient changes the gradient, not the loss value; its asymmetry may move a non-constant initialization away from collapse, but the paper does not prove this for a full causal Transformer.Source: §3.3 · p. 4

3.4 Where the statistic sits

• Because the predictor output is matched to a shallow patch embedding, intermediate state r_t must retain information about s that the final output need not preserve.The predicted linear-probe pattern is an intermediate peak; a pixel head does not impose the same shallow-range constraint.Source: §3.4 · p. 4

3.5 What the future shift contributes

• Removing the temporal shift permits copying the current patch without reading others, whereas next-pixel regression may spend capacity on predictable private residuals.For labels that are coarse functions of s, embedding prediction should provide an easier statistic; when labels are already simple functions of pixels, the gap can shrink.Source: §3.5 · p. 4

• The estimator is NEPA, retaining a patch map, causal stack, future shift, and stop-gradient as the objects needed to test the derivation.Source: §4 · p. 4
• The model uses shared linear patch embeddings and a pre-norm causal Transformer; teacher forcing supplies true embeddings without a pixel decoder, target encoder, or negative samples.Source: §4 · p. 5
• NOSTOP removes stop-gradient, NOSHIFT predicts the current embedding, and PIXEL regresses the raw next patch to test distinct hypotheses.Source: §4 · p. 5
• Linear probes use frozen last-token states from each block, while effective rank measures the patch-embedding spectrum; training loss is not treated as representation quality.The caption describes Table 1 as comparing each control’s predicted probe change relative to CPSS.Source: §4 · p. 5, Table 1 · p. 5

5.1 Protocol

• Runs use small causal Transformers on unaugmented MNIST and CIFAR-10 with short schedules, and do not support comparisons with fully trained contrastive or masked models.Source: §5.1 · p. 5

5.2 Layer-wise probes

• On MNIST, CPSS peaks at block 2 (87.3%) versus 77.2% at output; on CIFAR-10, it peaks at block 3 (32.2%) versus 28.2% (Figure 1; Table 2).Source: §5.2 · p. 5, Figure 1 · p. 6, Table 2 · p. 6
• Next-pixel regression avoids the same output penalty: its MNIST output is 87.1% against a peak of 89.6%, and its CIFAR-10 output at 33.4% is its best layer.Source: §5.2 · p. 6

5.4 Degeneracy is not the loss

• Without stop-gradient, effective rank falls from 17.1 to 4.0 on MNIST and 21.0 to 2.1 on CIFAR-10; perfect cosine does not ensure a worse CIFAR-10 probe.Source: §5.2 · p. 6, §5.4 · p. 7, Figure 2 · p. 7

5.3 MNIST versus CIFAR-10

• 91.6% pixel, 89.6% next-pixel, and 87.3% CPSS probes on MNIST; CIFAR-10 predictive best probes span 31.2%-33.4%, around the 32.1% pixel score.Source: §5.3 · pp. 6–7, Table 2 · p. 6

5.5 What the diagnostics do not show

• Linear probes, brief unaugmented training, and one seed bound absolute scores and exact block-index claims; conditional independence given the shared factor idealizes CIFAR-10 object boundaries.Source: §5.5 · p. 7

6 DISCUSSION

• With a patch-embedding target, downstream heads should read an intermediate block; the output is trained toward the shallow range and can discard information about s.An objective targeting a semantic space can instead place the statistic at the output.Source: §6 · p. 7
• Predictive information is the population quantity, cosine loss is a directional-likelihood surrogate, and stop-gradient constrains optimization dynamics rather than raising population mutual information.The constant embedding remains a global risk minimizer, so non-collapse depends on the optimization path and a non-degenerate embedding.Source: §6 · pp. 7–8
• The argument neither proves that gradient descent on a causal Transformer avoids the constant solution from random initialization, quantifies class information in embedding norms, nor estimates I(x≤t; s | xt+1).When the next patch is a blank background, the remainder is large and (3) fails.Source: §6 · p. 8
• For masked non-causal prediction, the information identity remains unchanged, while the statistic’s location changes only if the target is no longer a shallow embedding.Source: §6 · p. 8

7 APPLICATIONS

• The derivation is a placement and diagnostic rule for NEPA, not a new training objective.The four runs test which readings of the rule survive a short budget.Source: §7 · p. 8

7.1 Read the intermediate block

• 87.3% versus 77.2% on MNIST and 32.2% versus 28.2% on CIFAR-10: CPSS probes peak at intermediate blocks, unlike pixel regression, whose best probes are at or near the output.The CPSS peaks are at block 2 on MNIST and block 3 on CIFAR-10; pixel regression reaches 89.6% at its MNIST peak and 33.4% at the CIFAR-10 output.Source: §7.1 · p. 8

7.2 Do not rank runs by the pretext loss

• Noshift and nostop both reach cosine −1.000, but nostop’s effective rank falls to 4.0 on MNIST and 2.1 on CIFAR-10 while CPSS remains at 17.1 and 21.0.On CIFAR-10, nostop’s probe remains near 33%, so the probe alone misses the rank collapse.Source: §7.2 · p. 8

7.3 Choose the target from the label

• MNIST favors pixel-level prediction for ink-based labels, while CIFAR-10’s best predictive probes of 31.2%–33.4% do not exceed the 32.1% pixel baseline at this budget.The proposed embedding-target advantage is conditional on labels being coarser than local pixels and having enough training for f to discard private residuals.Source: §7.3 · p. 8

7.4 A check before a longer run

• Across these two datasets, CPSS probes rise after patch embedding and fall at the output; removing the shift hurts last-token probes, while removing stop-gradient collapses rank on both datasets but probe accuracy only on MNIST.The authors identify a longer CIFAR-10 run as useful for testing whether removing stop-gradient also collapses the probe or for exceeding the 32.1% pixel baseline.Source: §7.4 · p. 9

8 CONCLUSION

• Conditional predictive sufficiency targets the shared factor underlying future observations, but cosine prediction alone does not guarantee that statistic: constant embeddings minimize the loss and the shallow target leaves the statistic in intermediate blocks.MNIST shows probe changes with future shift and stop-gradient; CIFAR-10 shows the output gap and embedding-spectrum collapse rather than accuracy above pixels.Source: §8 · p. 9

A Proof details for Proposition 1

• Conditioning on the shared factor makes the next patch independent of the past, so I(x≤t; xt+1 | s) is zero and the joint information reduces to I(x≤t; s).This follows from the assumption xt+1 ⊥ x≤t | s and the chain-rule expansion of the joint entropy.Source: A Proof details for Proposition 1 · p. 13
• Expanding the same joint mutual information by conditioning first on xt+1 yields the next-patch information plus the residual I(x≤t; s | xt+1).That residual is small when the next patch reveals s or the past adds little once the next patch is known, but can dominate for blank next-patches.Source: A Proof details for Proposition 1 · p. 14
• The two chain-rule expressions describe the same mutual information, yielding Proposition 1’s decomposition of predictive information into information about s and the remainder.The proof introduces the joint variable (s, xt+1) and applies chain rules in both conditioning orders.Source: A Proof details for Proposition 1 · pp. 13–14

B VON MISES–FISHER DERIVATION

• Summing the per-position von Mises–Fisher terms for t = 1, . . . , T −1 and absorbing κ into the learning-rate scale produces LCPSS.For fixed κ, the normalizing constant does not depend on the predicted direction.Source: B VON MISES–FISHER DERIVATION · p. 14
• Because the radial coordinate is not modeled, standardized linear-probe results assess feature directions and would miss class information carried only by ∥zt∥.Standardization also discards global scale.Source: B VON MISES–FISHER DERIVATION · p. 14

C TRAINING AND PROBE DETAILS

• The diagnostics use public training and test splits without a held-out validation split, and hyperparameters were chosen for eight short runs rather than tuned against test probes.The raw-pixel baseline uses the same linear-probe optimizer, holding the readout fixed.Source: C TRAINING AND PROBE DETAILS · p. 14
• Effective rank is the exponential entropy of normalized singular values from centered training embeddings, equaling 1 for rank one and increasing as the spectrum flattens.The embeddings form a 4096 × d matrix.Source: C TRAINING AND PROBE DETAILS · p. 14

D DATA PROCESSING AND THE CONSTANT SOLUTION

• A representation reaches the predictive-information upper bound when it is sufficient for the next patch, but a constant representation also minimizes the cosine loss.Stop-gradient does not change the bound or loss value; it changes which parameters receive a gradient.Source: D DATA PROCESSING AND THE CONSTANT… · p. 14

E THE GAP ON TWO PATCHES

• When the next patch determines the shared class, predicting it is as informative as predicting the class; a blank next patch can make their mutual information near zero.Raster-order pairs near CIFAR-10 object borders can have this gap, whereas digits rarely have blank neighboring patches.Source: E THE GAP ON TWO PATCHES · p. 15

F LAYERWISE NUMBERS

• CPSS rank rises through middle blocks and falls toward the output, while nostop rank jumps at the output without restoring the class information.The nostop output probe on CIFAR-10 is only 32.7%; effective ranks are measured on a 4096-sample training slice.Source: F LAYERWISE NUMBERS · p. 15, Table 4 · p. 15, Table 3 · p. 15

G COMPUTING THE ESTIMATOR

• The estimator uses a causal attention stack and cosine comparisons over T − 1 pairs; probes are separate frozen-feature linear heads.The runs use one forward pass, no target encoder or negative queue, and fit on one NVIDIA H200.Source: G COMPUTING THE ESTIMATOR · pp. 15–16, Table 5 · p. 16