The role of locality and weight sharing in convolutional neural networks

1 Introduction

• CNNs encode translation invariance through weight sharing and locality through small filters, while natural images also exhibit spatial correlations.The introduction notes that noisy labels add to the interplay between architectural constraints and SGD noise.Source: §1 · pp. 1–2

1.1 Related works

• Prior work establishes statistical benefits of symmetry, but disentangling locality from weight sharing in feature learning and understanding correlated inputs remain open challenges.Related work also studies locality, architecture-dependent sample complexity, and optimization, but does not resolve the specific questions addressed here.Source: §1 · p. 2, §1.1 · pp. 3–4

1 Introduction

• The analysis derives closed ODEs for convolutional and locally connected students in the large-D limit, separating the effects of locality and weight sharing.It applies to Gaussian inputs, arbitrary kernel sizes, and input covariance.Source: §1 · p. 2
• For uncorrelated positions, the LCN follows the CNN’s trajectory but needs exactly P times more SGD steps; matching the teacher’s kernel requires O_D(D) samples, versus O_D(D log(D)) for a larger kernel.The extra logarithmic factor comes from a reflection-symmetric saddle point escaped through random-initialization fluctuations.Source: §1 · p. 2
• For c > 0 at the same learning rate, the CNN’s asymptotic error grows with correlation strength, while the LCN’s is independent of c and lower.With learning rates tuned to the training horizon, the LCN has lower error early and the CNN wins late, with the transition time predicted analytically.Source: §1 · pp. 2–3

2 Setup and notation

• The model studies a convolutional teacher and either a convolutional or locally connected student, with teacher–student learning used to recover the teacher’s output.Inputs contain P positions, each with D features, and the analysis takes P fixed as D tends to infinity.Source: §2 · p. 4, §2.1 · p. 5

2.1 Data structure

• Each input position has D features; the Gaussian input covariance encodes spatial structure, with compound symmetry giving unit variance and pairwise correlation c.At c = 0, positions are independent.Source: §2.1 · p. 5

2.2 Student architectures

• The teacher applies a kernel of size kt across consecutive positions with cyclic boundaries; students use kernel size ks ≥ kt and matched receptive fields, but only the CNN shares weights.The LCN has P times as many trainable parameters as the CNN; setting ks = kt makes the CNN student match the teacher architecture.Source: Figure 1 · p. 5, §2.1 · p. 6, §2.2 · p. 6

2.3 Learning algorithm

• Both students train by online SGD on squared loss, drawing a fresh independent sample at each step and updating weights with learning rate η.Weights are randomly initialized, and generalization error depends on low-dimensional summary statistics.Source: §2.3 · p. 7

3 Learning dynamics

• As D and SGD steps grow with α = t/D fixed, summary order parameters concentrate and follow closed ODEs for both student architectures.The theory matches numerical simulations at D = 1000; terms linear in η represent population-loss gradient flow, while η² terms describe SGD noise.Source: §3 · pp. 7–8

3.1 Benefits of translational invariance

• At c = 0, rescaling LCN time by P makes its error curve coincide with the CNN’s; the CNN gains an exact P-fold speedup, even at finite learning rate η.The shared kernel accumulates gradient contributions from all P positions, whereas each LCN kernel receives contributions only from its own position.Source: §3.1 · pp. 8–9
• For c > 0, the time-rescaled trajectories differ, and the LCN converges faster than the CNN even without rescaling.The rescaling failure and faster LCN convergence are illustrated at c = 0.5.Source: Figure 2 · p. 8, §3.1 · p. 9

3.2 Benefits of locality

• Matched kernels (ks = kt) require O_D(D) samples, while larger CNN kernels (ks > kt) require O_D(D log D) samples for specialization.The larger-kernel case has a long error plateau before an abrupt drop to zero.Source: §3.2 · pp. 9–10, Figure 3 · p. 10
• The OD(D) matched-locality result is expected for activations with information exponent one; higher information exponents may require more samples even when ks = kt.The paper leaves the role of locality in that higher-exponent setting for future analysis.Source: §3.2 · p. 11
• For ks > kt, reflection symmetry preserves a nonzero-error plateau, which unstable asymmetric perturbations from random initialization break after time of order log D.When ks = kt, the unique solution is itself symmetric and specialization occurs in O_D(1) time; the plateau and logarithmic growth also persist for c > 0.Source: §3.2 · pp. 11–12

4.1 Early-time expansion

• At c > 0, correlated CNN preactivations make output errors add coherently, whereas independent LCN parameters make errors across positions tend to cancel.The CNN consequently starts from a larger error at initialization.Source: §4.1 · pp. 12–13
• The early-time expansion predicts a transition boundary c*(α) = c0 α + O(α^2), with c0 depending on P, ks, T, and σ.It predicts that the CNN has lower error at small correlations and the LCN at large correlations; this boundary matches the numerical phase diagram well.Source: §4.1 · pp. 12–13

4.2 Asymptotic generalization error

• For c > 0, the asymptotic CNN noise floor grows with c and quadratically with P, while the LCN floor is independent of c and linear in P.At c = 0, the two floors coincide and are both linear in P.Source: Figure 5 · p. 13, §4.2 · p. 14
• At a fixed small learning rate, the LCN has lower asymptotic error because its position-wise errors are uncorrelated, unlike the CNN’s data-correlated errors.With learning rates optimized separately for each training horizon, the CNN instead has lower error at long horizons; its optimal learning rate is lower for every c > 0.Source: §4.2 · p. 14, §4.3 · p. 15

4.3 Phase diagram at the optimal learning rate

• At optimized learning rates, the LCN has lower error at short horizons and the CNN at long horizons, with an analytically predicted transition matching the numerical phase boundary.The same single-transition pattern holds for other tested values of P, kernel size, T, and σ.Source: §4.1 · p. 13, §4.3 · p. 14

5 Conclusions

• For uncorrelated positions, weight sharing speeds learning by exactly P; matching the teacher’s kernel size needs O_D(D) samples, versus O_D(D log D) for a larger kernel.At small constant learning rates, correlations make the CNN less robust to noise; with learning rates optimized for long horizons, it outperforms the LCN.Source: §5 · p. 15
• The analysis assumes Gaussian inputs without localized structure, regression, and one-pass SGD; multiple passes on finite data would require tools that address overfitting and memorization.The authors also identify classification as an extension beyond the regression setting studied.Source: §5 · p. 15
• The equations allow arbitrary position covariance, opening study of distance-decaying correlations; the analysis could also extend beyond translations to other position symmetries.The paper gives permutation invariance as an example of another group action.Source: §5 · p. 16
• The study uses constant learning rates; schedules could further reduce error and may be designed with optimal control.Source: §5 · p. 16

Statement on the use of AI

• Claude Opus 5.5 assisted with literature review, analytical expressions, simulation code, and drafting; the authors verified all results and take responsibility for the paper.Source: Statement on the use of AI · p. 16

A ODE derivation: the CNN

• In the D → ∞ limit, the CNN’s order-parameter ODE is derived generically for data covariance S and kernel size ks.The appendix computes the teacher–student overlap drift R from the SGD update.Source: A ODE derivation: the CNN · p. 20, §A.3 · p. 21

A.1 Setup and building blocks

• The setup defines student updates and position-indexed fields, with covariances built from input covariance and teacher and student overlap matrices.For convenience, the teacher kernel can be zero-padded to the student kernel size.Source: §A.1 · pp. 20–21

A.2 Saad–Solla integrals

• The ODEs use averages of activation functions and derivatives over jointly Gaussian fields; erf is chosen because these averages have closed forms.Source: §A.2 · p. 21

A.4 ODE for Q

• The Q dynamics account for updates to both student vectors, with two cross terms of order η and one term of order η^2.At leading order in D, input inner products concentrate around their means.Source: §A.4 · p. 22

B ODE derivation: the LCN

• For the LCN, each position’s kernel receives gradients only from its own position, while student–student blocks couple positions whose windows overlap.Source: B ODE derivation: the LCN · p. 24

B.1 Setup

• The LCN setup indexes fields by position and kernel weight, and its SGD update is specified separately for each position’s kernel.Source: §B.1 · p. 24

B.2 ODEs

• The LCN ODE lacks the CNN’s position sum in the R drift, producing the factor-of-P time rescaling when S = I.The p-th LCN kernel receives gradients only from position p.Source: §B.2 · p. 25

• For either student, the population squared error against the clean teacher is written in terms of averages over positions.Source: C Generalization error in terms of… · p. 26
• The averages reduce to sums of two-point integrals over jointly Gaussian fields.Source: C Generalization error in terms of… · p. 26
• The error expression is the same for both architectures; they differ through the preactivation covariances used to evaluate its integrals.Source: C Generalization error in terms of… · p. 26

D Exact P-fold rescaling at S = I

• For S = I, coinciding initial conditions, and position-symmetric LCN initialization, CNN and LCN dynamics coincide after rescaling time by P.The exact rescaling applies for kt ≤ ks ≤ P.Source: D Exact P-fold rescaling at S = I · pp. 26, 28
• The LCN generalization error depends on aligned entries of Q, and the aligned component and R form a closed dynamical description.Under position-symmetric initialization, this reduces the LCN description by a factor P.Source: D Exact P-fold rescaling at S = I · p. 26
• Position-symmetric initialization makes the reduced LCN ansatz satisfy its ODEs, which then match the CNN equations after time rescaling by P.Source: D Exact P-fold rescaling at S = I · p. 27

E Reflection symmetry of the CNN dynamics

• When the input covariance is reflection-invariant and teacher overlaps are symmetric, reflecting both kernels leaves the CNN ODEs equivariant and the generalization error invariant.Source: E Reflection symmetry of the CNN… · pp. 28–29
• A reflection-symmetric initial condition remains in the reflection-fixed subspace throughout the flow.Source: E Reflection symmetry of the CNN… · p. 29

F Plateau phase

• At ks = kt = 2, specialization time is flat in D; at ks = 3 and ks = 4, it grows logarithmically.Figure 7(a) uses a teacher with kt = 2.Source: F Plateau phase · p. 29, Figure 7 · p. 30
• For a teacher with kt = 1 and c = 0.5, specialization time remains D-independent at ks = kt and grows with D when ks > kt.The plateau for wider kernels persists under this correlation.Source: F Plateau phase · p. 29, Figure 7 · p. 30
• Figure 7 plots CNN specialization time against feature dimension, with student kernel size distinguished by color.Markers average over random initializations; lines fit a + b ln D.Source: Figure 7 · p. 30

G Early-time dynamics

• At c = 0, the early-time coefficients satisfy A_CNN = P A_LCN and B_CNN = P B_LCN, consistent with the stated equivalence of the students’ dynamics.The four coefficients are positive, independent of kt, and affine in 1/ks.Source: G Early-time dynamics · p. 32
• For c > 0, the LCN has lower error at short training times and the CNN at long times, with a transition at c*(α).The transition is determined by equality in the comparison condition; its bracket is reported positive for the tested parameter range.Source: G Early-time dynamics · p. 33
• At small α, the transition correlation follows c*(α) = c0 α + O(α^2).The expansion evaluates A and B at c = 0.Source: G Early-time dynamics · p. 33

H Noise-floor calculation

• For η → 0, near-perfect recovery is asymptotic; at finite but small η, the appendix calculates the fixed point and generalization error under label noise.The calculation covers CNN and LCN students with kt ≤ ks ≤ P.Source: H Noise-floor calculation · p. 33

H.1 CNN student

• The CNN calculation defines a symmetric ks × ks deviation matrix M that vanishes at perfect recovery and uses it to represent preactivation error.Near perfect recovery at small η, M is of order η and the error fields are of order √η.Source: §H.1 · p. 34
• Setting the CNN deviation dynamics to zero yields a Lyapunov equation with a unique solution for c < 1.The calculation then recovers the stated CNN noise-floor result, reported as independent of both kernel sizes.Source: §H.1 · p. 35

H.2 LCN student

• The LCN assigns a separate deviation kernel to each position, and its reported result is independent of c and both kernel sizes.Source: §H.2 · pp. 35–36

I Phase diagram: numerical method

• The analysis integrates the ODEs over correlation values, 80 learning rates, and 400 training horizons, then refines each grid minimum with a parabola fit.The learning-rate grid spans 10^-4 to 3, and the horizon grid spans 10^-2 to 10^4.Source: I Phase diagram: numerical method · p. 36
• For c > 0, the CNN’s optimal learning rate is lower than the LCN’s at every training horizon because it approaches the teacher faster at small learning rates.Source: I Phase diagram: numerical method · p. 37
• For every c > 0, the LCN has lower error at short training horizons and the CNN at long horizons, with a single transition.Figure 8 compares optimal learning rate and corresponding generalization error against training horizon for three correlation values.Source: I Phase diagram: numerical method · p. 37, Figure 8 · p. 37

J Activation dependence: ReLU

• With ReLU, the CNN’s rescaled noise floor grows with correlation c, while the LCN’s stays flat.The comparison uses online SGD from a specialized initial condition, with ks = kt = 2, P = 4, and D = 400.Source: J Activation dependence: ReLU · p. 37, Figure 10 · p. 39
• The shared-kernel noise-floor mechanism is coherent accumulation of correlated SGD noise, and is not specific to the activation.Source: J Activation dependence: ReLU · p. 37
• The P-fold learning-curve rescaling holds for ReLU at c = 0 but fails at c = 0.5.The rescaling test compares the CNN with the LCN after time rescaling by P.Source: J Activation dependence: ReLU · pp. 37, 39