A Theory of Contrastive Learning with Natural Images

Antonio Torralba; Yair Weiss

Source-linked AI summary

A Theory of Contrastive Learning with Natural Images

Antonio Torralba, Yair Weiss

arXiv:2607.07470v1Jul 8, 2026cs.CV

TL;DR

The paper asks why simple images and augmentations can produce useful contrastive-learning representations. It analytically characterizes optimal representations and CNN weights, finding sinusoidal frequency measurements and partial whitening whose utility depends strongly on augmentation and image power spectra.

Problem
The paper investigates why contrastive learning with simple images and augmentations can yield useful representations for downstream recognition tasks.
Method
The paper analytically computes optimal representations and simple-CNN weights for stationary datasets, using expected power spectra to predict measured frequencies and whitening.
Results
Augmentation and image choices greatly influence representation utility: many augmentations induce partial whitening, while their details alter measured frequencies and recognition performance.
Takeaways & Limitations
In this setting, stationarity connects dataset power spectra and augmentation details to the frequencies learned by contrastive representations.
Takeaways & Limitations
The analysis focuses on simple CNN architectures and augmentations whose performance is far from state-of-the-art, and uses LGUPA rather than InfoNCE.

Abstract

from arXiv · show

Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening

1. Contrastive Losses and their Optima

The section reformulates InfoNCE as a Gaussian uniformity-plus-alignment objective and characterizes its optima through whitening, generalized eigenvectors, and waterfilling. The resulting optimal projection is typically low-rank.

Contrastive loss: InfoNCE encourages augmented-image representations to be close while separating representations of different images.Its loss uses an inverse-temperature parameter and an ℓ2 distance formulation, with dot-product similarity equivalent under unit-sphere constraints.
Gaussian reformulation: For Gaussian representations and infinite batch size, InfoNCE equals the Gaussian Uniformity Plus Alignment loss up to an additive constant.The equivalence follows from expressing the Gaussian cross-entropy through covariance traces and determinants.
Gaussian reformulation: Under a unit expected squared norm, Gaussian uniformity is minimized when the representation is whitened, with equal variance in every direction.The Gaussian uniformity term therefore encourages a white representation covariance matrix.
Optimal weights: With fixed penultimate features, the optimal projection uses generalized eigenvectors of (B, Σ), with their rescaling determined by a globally optimal waterfilling algorithm.The allocation has nonnegative channel power under a total norm constraint, and diminishing returns make waterfilling applicable.
Optimal weights: The optimal projection matrix W ∗ will be low-rank because waterfilling typically assigns nonzero power to only a subset of channels.This prediction is noted as consistent with prior observations in contrastive learning.

2. Natural Image Statistics

Natural image datasets are modeled as translation-invariant (stationary) signals whose expected power spectrum decreases with spatial frequency. Under stationarity, DFT coefficients are asymptotically Gaussian and pairwise independent, properties found to hold approximately for CIFAR10.

Stationarity: Natural image statistics are assumed to be translation invariant, or stationary, across image locations.The covariance of neighboring pixels is expected not to depend on whether they appear on the left or right side of an image.
Power spectrum: The expected power spectrum of natural images decreases as spatial frequency increases.For CIFAR10, the expected power spectrum is highest at low frequencies near the plot center and decreases toward higher frequencies.
Fourier-domain consequences: For stationary signals under regularity conditions, DFT coefficients are asymptotically complex Gaussian and pairwise independent.The coefficient distribution is characterized by expected power g(k) = E[|xF [k]|2], and independence holds for distinct frequencies as image size tends to infinity.
Fourier-domain consequences: For stationary signals, the variance of each squared DFT coefficient equals its expected power squared.The theorem states V ar(|xF [k]|2) = g2(k).
Empirical validation: These asymptotic Fourier properties hold approximately for real image datasets, including CIFAR10.Across 50,000 CIFAR10 images, most DFT-coefficient histograms are highly Gaussian, implying that most coefficient pairs are independent.

3. Optimal Representations for Contrastive Learning with Natural Images

For several basic augmentations, globally optimal contrastive representations measure selected squared DFT coefficients with inverse-variance scaling, and a shallow sinusoidal CNN computes them. With harder augmentations, the optimal CNN weights remain sinusoidal and implement partial whitening over selected spatial frequencies.

Basic augmentations: For circular crops, brightness and contrast jitter, and ideal blur, a globally optimal representation measures K non-DC squared DFT coefficients below the cutoff and rescales them inversely to expected variance.The output vector is additionally rescaled to unit norm.
Basic augmentations: The representation is invariant to these augmentations because squared DFT coefficients ignore cyclic translations, non-DC frequencies ignore brightness changes, unit normalization removes contrast changes, and sub-cutoff frequencies ignore ideal blur.Its alignment loss is therefore zero, while inverse-standard-deviation scaling makes the covariance isotropic and white.
Basic augmentations: The optimum applies to any stationary signal and is non-unique across frequency choices and arbitrary orthogonal transformations of the output.White noise, filtered noise, and real images yield the same optimum up to scalar output rescaling.
CNN implementation: A globally optimal representation for the basic augmentations is computed by a shallow CNN with sinusoidal convolutional filters, squaring, global average pooling, and a whitening linear projection.Replacing squaring with ReLU also yields an alternative optimal representation, and deeper networks are unnecessary for this augmentation class.
Harder augmentations: For cyclic crops plus noise, linear jitter, and arbitrary blur, complete invariance is impossible for the simple CNN, but its optimal first-layer filters are still sinusoids measuring power at K frequencies.The theorem applies to any stationary signal satisfying theorem 2.1's conditions.
Harder augmentations: The optimal CNN performs partial whitening by selecting a subset of spatial frequencies and equalizing representation variance across directions, with sensitivity concentrated on mid-range frequencies in the shown datasets.This follows from diagonal covariance and augmentation matrices in DFT space.

4. Experiments

Experiments with single-layer CNNs broadly confirmed the theory: learned filters were sinusoids across many augmentations, while SOTA augmentations produced more localized filters. The learned representations improved CIFAR10 recognition, with much of the gain attributable to partial whitening, and transfer improved when training-image power spectra matched CIFAR10.

Experimental setup: Single-layer CNNs trained with the LGUPA contrastive loss used 256 11 × 11 filters, nonlinearities, global average pooling, and projection to 32 dimensions across synthetic and real datasets.The datasets included six synthetic datasets and CIFAR10, CIFAR100, and ImageNet.
Filter structure: For random-crop augmentations, learned first-layer weights converged to sinusoids in all cases, as predicted by the theory.The two augmentations were equal-sized random crops covering 84% of the original image and differed by a crop translation.
Filter structure: Across a wide range of augmentations, including no augmentation and mixed augmentation, networks learned sinusoidal filters or sums of a small number of sinusoids.The mixed augmentation combined random cropping, contrast inversion, and blur.
Filter structure: SOTA augmentations instead produced much more localized filters, presumably because localization better minimizes the alignment loss.These augmentations included horizontal flips, randomized crop size and aspect ratio, and nonlinear color jitter.
Color images: With color images, the theory remained applicable after decorrelating color channels into approximately gray-level and red–green and blue–yellow opponent channels.The learned filters for color images were shown for different augmentations.
Downstream recognition: CIFAR10 KNN recognition accuracy improved considerably over original weights, but most improvement came from partial whitening; transfer also improved as training power spectra approached CIFAR10’s.The latter result was obtained by training on synthetic images and testing on CIFAR10.

5. Limitations and Extensions

The analysis is limited by its simple architectures, augmentations, stationary-data setting, and use of LGUPA rather than InfoNCE. The authors identify extensions to deeper networks, broader augmentations and losses, constrained non-stationary datasets, and gradient-descent implicit bias.

Architectures and augmentations: The theory studies simple CNN architectures and augmentations whose performance is far from state-of-the-art, but it may extend to deeper networks and more augmentations.The authors chose this basic setting because it is easier to analyze.
Contrastive losses: The theoretical analysis uses LGUPA rather than InfoNCE, although the losses are identical only when y is Gaussian.The authors state that the analysis can extend to covariance-based contrastive losses and that experiments found the same representations with InfoNCE and LGUPA.
Optimization: The contrastive-loss minimum is non-unique, and gradient-descent dynamics influence which minimum a CNN reaches.The authors suggest relating this effect to theoretical work on implicit bias in deep-model gradient descent.
Non-stationary datasets: Although the theory assumes stationary datasets, CNNs with global average pooling still learn sinusoidal filters on non-stationary datasets.The authors hypothesize that global average pooling prevents exploiting non-stationarity and that the theory can extend to constrained architectures.

6. Conclusion … D. Optimal Weights in the Last Layer

The paper analytically characterizes contrastive-learning representations and shows that augmentation choices influence which frequencies are measured and how useful representations become. Its appendices relate InfoNCE to LGUPA, establish whitening properties, and derive optimal last-layer weights through generalized eigenvectors and waterfilling.

6. Conclusion: The analysis computes optimal representations and CNN weights for simple and additional augmentations, using stationarity to predict weights from the dataset’s expected power spectrum.The main results concern the contrastive loss LGUPA and simple CNNs under stationary image statistics.
6. Conclusion: Augmentation and image choices can greatly influence representation utility, while many augmentations produce partial whitening but measure different frequencies.The paper argues that small augmentation details can substantially affect the resulting representation.
A. Comparing the InfoNCE and LGUPA contrastive losses: With Gaussian y, InfoNCE and LGUPA yield nearly identical CIFAR10 recognition performance using either KNN or linear readout.Figure 11 compares the two losses across epochs; the losses are identical when y(x) is Gaussian.
B. From Uniformity to Cross Entropy: For infinite batches and Gaussian y, the InfoNCE loss equals the stated loss up to an additive constant.The appendix derives this equivalence by simplifying the infinite-batch InfoNCE expression under Gaussian representations.
C. Properties of the LGUPA Contrastive Loss: Under a centered-embedding norm constraint, LGUPA’s uniformity term is minimized when the representation is whitened, with equal variance in every direction.The unconstrained loss decreases as principal-direction variance increases, but the constraint yields a finite optimum.
D. Optimal Weights in the Last Layer: The optimal last-layer weights are rescaled generalized eigenvectors of (B, Σ) selected by minimal eigenvalue.The derivation identifies the columns of the optimal weight matrix as generalized eigenvectors and permits column-wise rescaling.
D. Optimal Weights in the Last Layer: Waterfilling globally optimizes nonnegative channel power allocations under the total representation-norm constraint because each channel’s marginal benefit decreases and is strictly convex.The allocation satisfies P_k ≥0 and total power P_k ≤1.

E. Optimal Representation with ReLU nonlinearity

With ReLU, full-image non-DC sinusoidal filters below the cutoff frequency yield representations invariant to cyclic crops and ideal blur. After ReLU and global average pooling, each feature measures the amplitude of one Fourier component, while independent channels produce diagonal covariance that can be whitened by standard-deviation scaling.

Architecture: ReLU CNNs can use full-image, non-DC sinusoidal filters below the cutoff frequency under circular boundary conditions.The architecture assumes no bias terms and the dataset conditions of theorem 2.1.
Invariances: The resulting representation is invariant to cyclic crops and ideal blur.Periodic convolution provides crop invariance, while selecting frequencies below the cutoff provides ideal-blur invariance.
Covariance: Because each neuron depends on one channel and one DFT coefficient, neurons are pairwise independent and the representation covariance matrix is diagonal.This conclusion holds when the conditions of theorem 2.1 are satisfied.
Feature interpretation: After ReLU and global average pooling, each feature is proportional to the amplitude of one Fourier coefficient rather than its power.For real images, conjugate Fourier coefficients have equal magnitudes, yielding amplitude-based features.
Whitening: Whitening is achieved by dividing each output feature by its standard deviation.The standard deviation is determined by the Rayleigh-distributed magnitude of a complex Gaussian Fourier coefficient.

F. Cyclic Translations and Crop Translations

Crop translations affect Fourier representations like cyclic translations plus a perturbation: the second crop’s transform is a phase-shifted first transform with translation-dependent error. Experiments show squared DFT power spectra remain highly correlated, with larger differences at high-power frequencies.

Formalization: The analysis compares the power spectra |X1(ω)|2 and |X2(ω)|2 of two windowed signals to characterize crop-translation effects.Crop translations are formalized in the continuous domain using a window function W(t).
Crop translations: The Fourier transform of a translated crop equals the first crop’s transform with a phase shift plus a perturbation ηF(ω) whose magnitude depends on translation size.Unlike cyclic translation plus independent noise, the perturbation ηF(ω) is not independent of the original signal.
Empirical comparison: For three CIFAR10 images, squared DFT power spectra from different crop translations are highly correlated, with larger differences at frequencies carrying more power.The same qualitative pattern appears for cyclic translation plus noise.

G. Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur · H. Computing sensitivities of CNNs · I. Partial Whitening using PCA

The paper derives frequency-domain optimal weights for simple CNNs under several augmentations, defines frequency sensitivity, and interprets PCA-based linear mappings as partial whitening. Experiments and figures connect these analyses to sinusoidal filters, dataset-dependent frequency sensitivity, and KNN accuracy.

G. Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur: For cyclic crop plus noise, linear jitter, and blur, the optimal CNN first-layer filters are sinusoids measuring signal power at K frequencies.The theorem applies to any stationary signal satisfying theorem 2.1 and uses the architecture with a squaring nonlinearity.
G. Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur: The CNN output can be written as y(x) = WΦ(x), with Φ(x) including squared DFT features such as |xF[k]|2.This representation links the network to frequency-power measurements.
G. Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur: For all three augmentations, B and Σ are diagonal, so generalized eigenvectors are unit vectors in frequency space.The corresponding generalized eigenvalues are computed from the diagonal matrices.
H. Computing sensitivities of CNNs: Sensitivity is defined as the ℓ1 norm of a representation’s response to an image that is a pure sinusoid at frequency k.The paper reports nearly identical results when replacing the ℓ1 norm with the ℓ2 norm.
I. Partial Whitening using PCA: The reported partial-whitening mapping uses y(x) = W^T(x − µ), retaining unit variance when λ << Λ(j,j) and suppressing variance when λ >> Λ(j,j).The covariance eigenvectors and eigenvalues determine the mapping through the first K components.
I. Partial Whitening using PCA: PCA-based linear partial whitening is mostly sensitive to ring-shaped frequencies for fractal images and diamond-shaped frequencies for CIFAR10.This pattern resembles predicted optimal-CNN and SGD-trained-CNN sensitivities.
I. Partial Whitening using PCA: Quadratic-nonlinearity experiments found sinusoidal first-layer filters across datasets, with sensitivity patterns qualitatively predicted by theory.Fractal-image sensitivity forms a ring whose radius decreases with α, while real-image sensitivity follows a different dataset-dependent pattern.
I. Partial Whitening using PCA: 47% KNN accuracy resulted when training examples were augmented with horizontal flips, compared with 45% without horizontal flips.The experiment followed Thiry (2021)’s code.

J. Experiments with different augmentations

Experiments across augmentations show that learned filters are highly localized in frequency space and often become sinusoids. The reported results also examine CelebA’s DFT coefficients and larger-resolution training setups.

Learned filters: Across many augmentations, the network learns filters that are highly localized in frequency space.This behavior matches the proof of theorem 3.2.
Learned filters: Sinusoidal filters emerge because uniformity is minimized by measuring individual DFT coefficients when simple coefficient functions keep alignment loss low.The passage states this mechanism explicitly for simple functions of DFT coefficients.
Experimental results: Figure 17 summarizes results for different augmentations and quadratic or ReLU nonlinearities after 2000 training epochs.It extends Figure 8 by adding network sensitivities to the learned filters.
CelebA: CelebA produces approximately Gaussian DFT coefficients.This finding is presented in Figure 18.
CelebA: With CelebA scaled to 128×128, experiments compare square nonlinearity with cyclic translations and circular convolutions against ReLU with crop translations and valid convolutions.These configurations correspond to the left and right panels of Figure 19.

K. Experiments with nonstationary datasets

Experiments on the highly nonstationary CelebA dataset show that sinusoidal filters are still learned, despite strongly location-dependent image statistics. This follows because DFT coefficients remain approximately Gaussian and squared-coefficient covariance remains approximately diagonal.

K. Experiments with nonstationary datasets: On highly nonstationary CelebA images, the CNN still learns sinusoidal filters despite strongly location-dependent statistics.The DFT coefficients are approximately Gaussian, and the covariance of squared DFT coefficients is approximately diagonal.

L. Visualizing all learned filters

Across all 256 learned filters, cyclic and crop translations produce almost exclusively sinusoidal kernels, while horizontal flips can produce paired sinusoids. The filters remain highly localized in Fourier space, and fitting errors reflect the augmentation-induced frequency structure.

Cyclic and crop translations: Almost all 256 filters learned with cyclic or crop translations are sinusoids, evidenced by two Fourier-domain peaks and small differences from best-fitting sinusoids.The filters are visualized in both spatial and Fourier domains, alongside their best-fitting sinusoids and residual differences.
Horizontal flips: With horizontal-flip augmentation, filters remain highly localized in Fourier space but can combine two sinusoids, producing four Fourier-domain peaks.For these filters, the residual from the sinusoidal fit is another sinusoid with a horizontally flipped frequency.
Experimental configurations: The translation experiments use 256-channel networks with either squaring and circular convolutions or ReLU and valid convolutions, respectively.The cyclic-translation network has no bias terms, whereas the crop-translation network has bias terms in both layers.
Experimental configurations: The horizontal-flip experiments likewise use 256-channel networks, comparing squaring with circular convolutions against ReLU with valid convolutions.The squaring/circular-convolution configuration has no bias terms, while the ReLU/valid-convolution configuration includes bias terms in both layers.

M. Non-Uniqueness of the Optimal Solution · N. Extensions to Color Images

The optimal contrastive representation is generally non-unique: multiple frequency sets, orthogonal transformations, and parameter settings can attain the same minimum. For color images, the theory extends to independent channels and predicts sinusoidal filters after decorrelating natural-image color channels.

M. Non-Uniqueness of the Optimal Solution: For the augmentations in figure 4, measuring any K frequencies followed by whitening yields an optimal representation.More generally, if y*(x) is optimal, then Qy*(x) is also optimal for any orthogonal matrix Q.
M. Non-Uniqueness of the Optimal Solution: Gradient-descent dynamics influence which minimum a CNN finds, with longer training selecting higher-frequency sinusoids.In theorem 3.1, the chosen frequencies depend on the number of iterations.
M. Non-Uniqueness of the Optimal Solution: The architecture in figure 5 also permits many weight settings to compute the exact same representation.The output is y(x) = WHΦ(x), so different pairs of projection weights and filter weights can produce the same effective mapping.
M. Non-Uniqueness of the Optimal Solution: When channels are numerous, optimization can freeze either the projection weights or the filters while optimizing the other component.Both procedures can compute the optimal representation under suitable parameterizations.
M. Non-Uniqueness of the Optimal Solution: The global minimum can be attained with sinusoidal first-layer filters, but the solution is not unique.With a frozen orthogonal final layer, however, the minimum can only be attained when the filters are sinusoids.
N. Extensions to Color Images: In color images with stationary, independent channels, contrastive-trained CNNs learn to measure sinusoids in each channel.The extension relies on asymptotically Gaussian, independent DFT coefficients for each channel.
N. Extensions to Color Images: Natural-image R, G, and B channels can be made uncorrelated by a 3 × 3 linear transformation before applying the channel-wise theory.The resulting channels correspond approximately to radiance, blue–yellow, and red–green chromatic opponent mechanisms.