A Theory of Contrastive Learning with Natural Images

1. Contrastive Losses and their Optima

• For Gaussian representations and an infinite batch, InfoNCE equals the Gaussian Uniformity Plus Alignment loss up to an additive constant.The Gaussian uniformity term encourages white covariance.Source: §1 · p. 3
• Under the centered average squared-norm constraint of 1, Gaussian uniformity is minimized by a whitened representation with equal variance in every direction.InfoNCE makes augmented views close while pushing apart representations of different images.Source: §1 · pp. 2–3

1.1. Optimal Weights

• For fixed penultimate-layer features, the optimal linear weights are rescaled generalized eigenvectors of (B, Σ), with their scaling found by waterfilling.The solution allocates power across eigenvector channels subject to the representation norm constraint.Source: §1.1 · p. 3
• The optimal projection matrix is typically low-rank because waterfilling assigns nonzero power to only a subset of channels.The theorem assumes fixed features, though the paper notes its analysis can extend to more realistic settings.Source: §1.1 · pp. 3–4

2. Natural Image Statistics

• Translation invariance, or stationarity, means image statistics such as neighboring-pixel covariance do not depend on location (Hyv¨arinen et al., 2009).Source: §2 · p. 4
• Natural-image expected power spectra decrease with frequency, with CIFAR10 showing greater power at low than high spatial frequencies.Source: §2 · p. 4
• As N→∞, for almost all k, stationary-signal DFT coefficients are complex Gaussian, asymptotically pairwise independent across frequencies, and satisfy Var(|xF[k]|²)=g(k)².Here g(k) is the expected power at frequency k.Source: §2 · p. 4
• For CIFAR10, most DFT-coefficient histograms are highly Gaussian and squared-coefficient covariance is nearly diagonal; CIFAR10 and fractal noise show similar covariance patterns.These empirical observations approximately match the stationary-signal predictions and motivate using them to predict optimal representations.Source: §2 · p. 4

3. Optimal Representations for Contrastive Learning with Natural Images

• For circular crops, brightness and contrast jitter, and ideal low-pass blur, a unit-normalized representation measuring squared DFT coefficients at K non-DC frequencies below the cutoff, scaled inversely to expected variance, is globally optimal.The result applies to stationary signals satisfying the stated regularity conditions.Source: §3 · pp. 4–5
• The representation achieves zero alignment loss across these augmentations and a whitened covariance, so it minimizes both alignment and Gaussian uniformity losses.The optimum is not unique: any frequency set is allowed, and an orthogonal transformation of the representation remains optimal.Source: §3 · p. 5
• A CNN with sinusoidal convolution filters, squaring, global average pooling, and a linear whitening projection computes the optimum; an alternative uses ReLU.For these augmentations, this architecture suffices for any stationary signal, without deeper layers.Source: §3 · pp. 5–6

3.1. Optimal CNN Weights with More Augmentations

• With cyclic crops plus noise, linear jitter, and arbitrary blur, optimal CNN weights still use sinusoidal filters to measure frequency power, with selected frequencies and positive weights computed by waterfilling.The simple CNN cannot achieve complete invariance to these augmentations; perfect invariance to added noise would require denoising.Source: §3.1 · p. 6
• For the harder augmentations, the optimal CNN is sensitive to a subset of spatial frequencies and has equal representation variance in all directions.Across the illustrated datasets, sensitivity concentrates at mid-range frequencies; it forms a ring for datasets with power falling off from the origin and a diamond for CIFAR10, while the sensitivity-weighted power indicates partial whitening.Source: §3.1 · pp. 6–7, Figure 6 · p. 6

4. Experiments

• With equal-size random crops, first-layer filters converged to sinusoids, and ReLU networks showed theory-consistent frequency sensitivity and partial whitening.The crops were 84% of the original image size.Source: §4 · p. 7
• Across varied augmentations, filters were sinusoids or sums of a few sinusoids; horizontal flips produced frequency-localized, horizontally symmetric filters.The account attributes this pattern to uniformity favoring individual DFT coefficients when alignment loss is low with simple functions of those coefficients.Source: §4 · p. 7
• SOTA augmentations—horizontal flips, randomized crop size and aspect ratio, and nonlinear color jitter—produced more localized filters, presumably to better minimize alignment loss.The passage characterizes these augmentations as not very different from the other tested augmentations.Source: §4 · p. 7
• With color images, CNNs learned first-layer sinusoids sensitive to only one decorrelated channel, approximately gray-level or one of two color-opponent channels.The theory holds for color images when color channels are decorrelated.Source: §4 · p. 7
• On CIFAR10, contrastive learning improved KNN recognition accuracy over the original weights, while linear partial whitening matched the ReLU CNN on most augmentations; SOTA augmentations were an exception.The authors attribute most of the improvement to partial whitening; localized filters under SOTA augmentations capture more than partial whitening.Source: §4 · p. 7
• Training on non-CIFAR10 synthetic images improved CIFAR10 recognition accuracy when their expected power spectrum was closer to CIFAR10’s.This is reported as consistent with the theory’s prediction.Source: §4 · p. 7

• The analysis uses very simple CNNs and augmentations whose performance is far from state of the art, though theory may extend to richer augmentations and deeper networks.Source: §5 · p. 7
• The theoretical analysis uses LGUPA rather than InfoNCE, and the losses are identical only when y is Gaussian.Source: §5 · p. 7
• The analysis can extend straightforwardly to other contrastive losses based on embedding covariance (HaoChen et al., 2021; Zbontar et al., 2021).Source: §5 · p. 9
• Experiments find the same representations with InfoNCE and LGUPA, consistent with reports that contrastive learning leads to Gaussian distributions (Betser et al., 2026; Balestriero & LeCun, 2025).Source: §5 · p. 9
• The contrastive-loss minimum is not unique, and gradient-descent dynamics empirically influence which minimum a CNN finds, potentially relating to implicit-bias studies.Source: §5 · p. 9
• Although the theory assumes stationary datasets, CNNs with global average pooling learn sinusoidal filters on non-stationary datasets, suggesting an extension is possible for constrained architectures.Source: §5 · p. 9

6. Conclusion

• Stationarity lets the authors predict optimal weights from a dataset’s expected power spectrum, while the choice of images and augmentations can strongly affect representation utility.Augmentations can change which frequencies are measured; synthetic images should have a power spectrum close to real images to encourage measurement of the same frequencies.Source: §6 · p. 9
• Partial whitening reduces representation redundancy and provides some noise robustness; the results suggest contrastive augmentations can guide this representation rather than simply teach transformation invariance.The paper notes that partial whitening had previously been suggested as useful by domain experts.Source: §6 · p. 9

A. Comparing the InfoNCE and LGUPA contrastive losses

• On CIFAR10, ResNet18 recognition accuracy is nearly identical with InfoNCE and LGUPA, using either KNN or linear readout.The comparison varies performance by epoch and reports the near-match for both readout types.Source: A. Comparing the InfoNCE and LGUPA… · p. 13

B. From Uniformity to Cross Entropy

• With Gaussian y and infinite batch size, InfoNCE equals LGUPA up to an additive constant.The proof uses the Gaussian assumption to simplify the uniformity term.Source: B. From Uniformity to Cross Entropy · pp. 13, 15
• For Gaussian y with covariance Σ_y, smoothing its distribution with Gaussian covariance εI yields covariance Σ_y + εI, enabling the uniformity term to be expressed through Gaussian cross-entropy.The smoothing distribution q is a Parzen-window estimate of p(y).Source: B. From Uniformity to Cross Entropy · p. 15

C. Properties of the LGUPA Contrastive Loss

• Under a centered-embedding norm constraint, the LGUPA uniformity loss is minimized by a whitened representation with equal variance in every direction.Without the constraint, increasing variance along principal directions can decrease the uniformity loss arbitrarily.Source: C. Properties of the LGUPA Contrastive… · p. 15

D. Optimal Weights in the Last Layer

• For y(x) = W^T Φ(x), the optimal last-layer weights are rescaled generalized eigenvectors of (B, Σ), selected by a spectral algorithm and waterfilling under the norm constraint.The theorem identifies eigenvectors with minimal eigenvalue; the optimal power allocation is found by waterfilling.Source: D. Optimal Weights in the Last Layer · pp. 16–17

E. Optimal Representation with ReLU nonlinearity

• Under Theorem 2.1 dataset conditions, the specified architecture, no biases, circular boundaries, and figure 4 augmentations, sub-cutoff non-DC sinusoids with whitening attain the global LGUPA optimum.The first-layer filters span the full image, and the final-layer weights are unit vectors scaled to whiten.Source: E. Optimal Representation with ReLU… · p. 17
• These filters yield zero alignment loss by making the representation invariant to cyclic crops, ideal blur, and linear jitter after normalization.Circular convolution provides crop invariance, the frequency cutoff provides blur invariance, and no biases ensure scale equivariance before normalization.Source: E. Optimal Representation with ReLU… · p. 17
• Under Theorem 2.1’s conditions, the output neurons are pairwise independent, giving diagonal covariance that becomes white after each unit is divided by its activity standard deviation.Each neuron depends on one DFT coefficient; for complex Gaussian coefficients, the amplitude is Rayleigh-distributed and its standard deviation is proportional to that of the coefficient.Source: E. Optimal Representation with ReLU… · pp. 17–18
• With sinusoidal filters, ReLU and global average pooling produce features proportional to the amplitudes of individual Fourier coefficients, rather than their powers.The proof uses circular translation and the symmetry of Fourier coefficients for real-valued images.Source: E. Optimal Representation with ReLU… · p. 18

F. Cyclic Translations and Crop Translations

• Crop translations are modeled in the continuous domain using a window function that defines the crop.The analysis compares the power spectra of two cropped signals.Source: F. Cyclic Translations and Crop… · p. 18
• The second crop’s Fourier transform is a phase-shifted first-crop transform plus a perturbation bounded by translation size; unlike independent noise, this perturbation depends on the signal.Source: F. Cyclic Translations and Crop… · p. 18
• For both crop translations and cyclic translations plus noise, squared-DFT features are highly correlated, with larger differences at frequencies with greater power.Figure 14 compares three CIFAR10 images under crop translations and cyclic translations plus noise.Source: F. Cyclic Translations and Crop… · p. 18, Figure 14 · p. 19

G. Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur

• For stationary signals, the stated CNN achieves the optimal contrastive loss with sinusoidal filters and positive frequency weights selected by waterfilling.Its representation measures signal power at K frequencies.Source: G. Optimal Weights for simple CNN with… · p. 19
• Diagonal B and covariance matrices make the generalized eigenvectors unit vectors in frequency space, yielding frequency-wise weights for the augmentation cases.The CNN features include the squared DFT; the covariance is diagonal because DFT coefficients are pairwise independent.Source: G. Optimal Weights for simple CNN with… · pp. 19–20

H. Computing sensitivities of CNNs

• Sensitivity is the ℓ1 norm of the representation on a pure sinusoid at a given frequency, computable from the simple CNN’s weights.The reported sensitivities are nearly identical when using the ℓ2 norm instead.Source: H. Computing sensitivities of CNNs · p. 20

I. Partial Whitening using PCA

• Figure 15 shows partial-whitening sensitivity concentrated on a ring for fractal images and a diamond for CIFAR10.These patterns are described as similar to the predicted optimal-CNN and trained-CNN sensitivities.Source: Figure 15 · p. 21
• PCA-based partial whitening maps centered inputs so directions with variance large relative to λ have unit variance, while directions with variance small relative to λ approach zero variance.The PCA construction uses dataset covariance eigenvectors and eigenvalues.Source: §I · pp. 20–21
• 47% KNN accuracy was obtained with horizontal flips, versus 45% without them.The setup followed the code of (Thiry, 2021).Source: §I · p. 21

J. Experiments with different augmentations

• Learned filters are highly localized in frequency across augmentations; horizontal-flip filters are also horizontally symmetric, consistent with networks learning sinusoids.The paper attributes sinusoidal filters to the uniformity loss measuring individual DFT coefficients when alignment loss is low for simple functions of those coefficients.Source: J. Experiments with different… · p. 21, Figure 17 · p. 22
• Across augmentations, sensitivity increases with frequency but ignores frequencies with low expected power, indicating partial whitening.For squaring, sensitivity profiles look like diamonds whose size varies with augmentation and learning details such as iterations and weight decay; ReLU profiles are noisier but follow the same general trend.Source: J. Experiments with different… · p. 21, Figure 17 · p. 22
• CelebA leads to approximately Gaussian DFT coefficients.Source: Figure 18 · p. 23

K. Experiments with nonstationary datasets

• Despite CelebA’s highly location-dependent statistics, its DFT coefficients are approximately Gaussian and squared-coefficient covariance approximately diagonal, and the CNN learns sinusoidal filters.The location dependence is shown in Figure 18.Source: K. Experiments with nonstationary… · p. 23

L. Visualizing all learned filters

• With cyclic and crop translations, almost all 256 learned filters are sinusoids, and their differences from best-fitting sinusoids are mostly noise.The figures show filters in spatial and Fourier domains, alongside sinusoidal fits and their differences.Source: L. Visualizing all learned filters · p. 23, Figure 20 · p. 24, Figure 21 · p. 25
• With horizontal flips, filters remain highly localized in Fourier space but sometimes combine two sinusoids; their fit residual is another sinusoid at a horizontally flipped frequency.Figures 22 and 23 show CIFAR10 filters; Figure 23 uses ReLU nonlinearities, valid convolutions, 256 channels, and bias terms in both layers.Source: L. Visualizing all learned filters · pp. 23, 27, Figure 22 · p. 26, Figure 23 · p. 27

M. Non-Uniqueness of the Optimal Solution

• For the augmentations in Figure 4, measuring any K frequencies and whitening is optimal, and any orthogonal transform of an optimal representation remains optimal.More generally, if y*(x) is optimal, then Qy*(x) is also optimal for any orthogonal matrix Q.Source: M. Non-Uniqueness of the Optimal… · p. 28
• With crop-translation augmentation, more training iterations yield higher-frequency sinusoids, while recognition accuracy peaks early even as loss continues to decrease.For CIFAR10, performance peaks around epoch 100 in the reported training run.Source: M. Non-Uniqueness of the Optimal… · p. 28, Figure 25 · p. 29, Figure 24 · p. 29
• In the Figure 5 architecture, many settings of projection weights and filters can yield the same representation, and with many channels either component can be optimized while the other is frozen.With high-variance random initialization of projection weights, gradient descent barely changes those weights and instead adjusts first-layer filters.Source: M. Non-Uniqueness of the Optimal… · p. 28
• When projection weights are frozen to an orthogonal matrix under Theorem 3.2's assumptions, the optimal loss can only be attained by sinusoidal first-layer filters.Theorem 3.2 permits sinusoidal filters at a global minimum but does not make them unique; Theorem M.1 establishes necessity under the frozen-weight condition.Source: M. Non-Uniqueness of the Optimal… · p. 28

N. Extensions to Color Images

• For CIFAR10 color-image training, kernels learned with all simple augmentations are sinusoids in approximately gray, red-green, and blue-yellow uncorrelated channels, as predicted by the theory.Source: Figure 26 · p. 30
• For multichannel signals with stationary, mutually independent channels, the theory extends directly and predicts that contrastive CNNs measure sinusoids in each channel.The prediction follows because each channel's DFT coefficients are asymptotically Gaussian and independent.Source: N. Extensions to Color Images · p. 30
• For natural RGB images, a 3×3 linear transform decorrelates the channels; the theory then predicts sinusoids in those channels, consistent with the reported ImageNet and CIFAR10 figures.The approximately uncorrelated channels are described as gray, red-green, and blue-yellow.Source: N. Extensions to Color Images · p. 30, Figure 26 · p. 30