Source-linked AI summary
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea
TL;DR
The paper addresses why teacher-student learnability varies even when students have sufficient expressive capacity, focusing on teacher geometry and optimization dynamics. It formalizes learnability, analyzes loss landscapes and convergence modes, and finds that dissimilar teacher nodes and differential learning rates improve training success.
Problem
Existing teacher-student studies often assume normally distributed teacher parameters, leaving variation in the probability that students reach zero loss insufficiently characterized.
Method
The paper formalizes learnability as global-minimum convergence probability and studies teacher distributions, loss landscapes, convergence classes, and optimizer dynamics.
Results
Teacher-node dissimilarity strongly correlates with convergence probability, and increasing readout learning rates while decreasing inner-bias learning rates substantially improves success rates.
Takeaways & Limitations
Teacher geometry and differential layerwise learning rates provide a framework for understanding and improving learnability in teacher-student networks.
Takeaways & Limitations
The convergence classification assumes that the probability of reaching a saddle point exactly is vanishingly small.
Abstract
from arXiv · showhide
Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.
1 Introduction
The paper asks why learnability varies beyond expressibility and formalizes teacher-student learnability around teacher geometry, optimization, initialization, and overparameterization. It finds that teacher-node dissimilarity strongly affects convergence and studies loss-landscape mechanisms and optimizer interventions.
- Motivation: Learnability depends not only on whether a network can express a function, but also on whether optimization avoids suboptimal local minima.This motivates studying factors beyond architectural capacity.
- Problem formulation: The paper formalizes teacher-student learnability as a function of teacher distribution, initialization, data, overparameterization, and optimizer.The teacher generates labels, while the student is trained to match them.
- Teacher geometry: Teacher-node similarity strongly negatively correlates with student convergence probability across minimally dissimilar, standard, and maximally dissimilar teacher distributions.These distributions are ordered by increasing degree of learnability.
- Loss landscapes: The study analytically characterizes ReLU loss landscapes to examine how teacher and student orientations, signs, and biases influence local minima.It combines analytical reduction with controlled experiments across teacher distributions.
- Convergence behavior: Minimally dissimilar and standard teachers more often drive neurons out of bounds than maximally dissimilar teachers.The paper distinguishes these convergence behaviors as part of the explanation for differing success rates.
- Optimization: Increasing readout-weight learning rates or decreasing inner-bias learning rates substantially improves success rates across all teacher distributions.The intervention targets optimizer dynamics rather than teacher expressibility.
2 Teacher-student learnability
The teacher-student framework defines teacher-generated targets and trains an overparameterized student to match them. Learnability is measured by the probability of reaching zero loss, while convergence can instead produce OOB or interior local minima.
- Teacher network: A one-hidden-layer teacher maps inputs to labels using embedding weights, readout weights, inner biases, an outer bias, and an activation function.Teacher parameters are drawn from a teacher distribution.
- Student network: The student uses analogous parameters and is trained to match teacher outputs, with N student neurons compared with M teacher neurons.The overparameterization ratio is ρ = N/M.
- Convergence classes: ReLU neurons are out of bounds when their kink lies outside the data range, preventing the unit from having datapoints on both sides.The kink is k = −b/||w|| along the weight direction.
- Convergence classes: Converged students are classified as global minima, OOB local minima with L > 0, or interior local minima with L > 0.Interior minima retain in-bounds neurons but remain suboptimal.
- Learnability: Teacher-student learnability is the expected conditional probability that training reaches the global minimum over student initializations and training data.The probability is conditioned on the teacher, overparameterization ratio, and optimizer.
- Learnability: Empirical learnability is estimated from repeated random-seed trainings, counting runs whose loss falls below 10^-18.This produces a binomial estimate for each teacher.
3 Comparing three different teacher distributions
The paper compares maximally dissimilar, standard, and minimally dissimilar teacher distributions to test how teacher geometry shapes student learnability. Maximally dissimilar teachers are generally easiest to learn, while minimally dissimilar teachers are hardest across varied settings and activations.
- Teacher distributions: Maximally dissimilar teachers use mutually dissimilar weight directions with balanced readout signs, whereas standard teachers use independent normal weights and zero biases.The minimally dissimilar distribution is referenced but its defining passage is truncated in the supplied text.
- Learnability across settings: As teacher width increases, global-minimum convergence decreases, while increasing input dimensionality makes fitting easier.These trends are reported while varying one parameter at a time around M = 4 or D = 2.
- Learnability across settings: Across almost all settings, success rates rank maximally dissimilar teachers highest, standard teachers next, and minimally dissimilar teachers lowest.The comparison covers changes in teacher-node count and input dimensionality.
- Activation functions: Similar success-rate patterns occur with ReLU, softplus, and tanh activation functions.The paper repeats the experiment for softplus and tanh and reports similar results.
4 Loss landscapes of one- and two-node ReLU systems
Loss-landscape analyses reduce small ReLU teacher-student systems under infinite data and fast readout weights to low-dimensional dynamics governed by teacher-student angles. Teacher signs and orientations determine whether interior local minima appear and how strongly they attract trajectories.
- Single-neuron system: For a single neuron, infinite Gaussian data and fast readout weights produce a landscape without spurious local minima, but finite-data gradient flow can drive dissimilar or wrongly signed starts out of bounds.The infinite-data setting has no OOB minima because Gaussian data is unbounded; finite-data simulations show strong dependence on starting similarity.
- Two-neuron system: The two-neuron bias-free landscape is reduced to the angles between student weights and a reference teacher weight using teacher–teacher, student–student, and teacher–student kernels.Rotational invariance expresses the relevant angles through the two student angles and the teacher-node angle.
- Two-neuron system: Same-sign readouts yield only the global minimum under the analytical assumptions, whereas mixed-sign readouts create local minima in which both students align with one teacher neuron.The attraction basin of these local minima decreases as the angle between teacher nodes increases.
- Finite-data comparison: With finite data and standard gradient flow, increasing the angle between opposite-sign teacher neurons increases the fraction of students trapped above the global minimum.Same-sign teachers instead develop a significant number of local minima, producing a slightly lower overall success rate than opposing-sign teachers.
- Interpretation: The changing loss landscapes help explain why local minima are more prominent for dissimilar teacher nodes and motivate faster readout learning.The empirical-versus-analytical difference also motivates examining out-of-bounds minima caused by bounded data.
5 Analysis of the local minima
The paper distinguishes global, out-of-bounds, and interior local minima and relates their frequencies to teacher geometry and initialization overlap. Maximally dissimilar teachers show fewer OOB outcomes but more interior local minima, while initialization overlap is highest for this distribution.
- Types of minima: Student training reaches three outcome types: global minima, OOB local minima, and interior local minima.The paper illustrates these outcomes for networks with more than two hidden neurons.
- Minima by teacher geometry: Maximally dissimilar teachers produce fewer OOB neurons and more interior local minima than the other teacher distributions.These teachers also reach the global minimum most frequently.
- Minima by teacher geometry: Changing readout signs substantially affects success rates, but neither tested sign modification matches the success of the maximally dissimilar distribution.The result indicates that both readout signs and teacher-node orientations contribute to learnability differences.
- Initialization overlap: The study tests whether initial overlap predicts which minima students reach by using it as an AUROC decision score.The supplied passage introduces this evaluation but does not report its resulting AUROC values.
- Initialization overlap: Initial teacher–student overlap is highest for maximally dissimilar teachers and tends to increase with the number of teacher nodes.The overlap roughly trends toward zero as input dimensionality increases.
6 Differential learning rates
Differential learning speeds are motivated by the distinct roles of local minima and OOB failures. Faster readout learning and slower inner-bias learning improve global-minimum success, whereas Adam does not provide the same improvement.
- Motivation: Faster readout weights and slower inner biases are tested as a strategy for improving student success.The motivation comes from analytical disappearance of local minima with fast readout weights and empirical OOB failures from poor starts.
- Figure 6: Figure 6 compares global-minimum success across differential learning speeds, minimum types under several rate settings, and gradient flow versus Adam.The 10x differential speed increases readout speed while decreasing inner-bias speed by the same factor.
- Learning-rate effects: Increasing the inner-bias learning rate decreases success by driving more neurons out of bounds.The changes in success track corresponding changes in OOB-neuron frequency.
- Optimizer comparison: Changing from gradient flow to Adam does not increase success rates or prevent neurons from going out of bounds.The paper suggests momentum may mitigate other critical points or stochastic-gradient noise without removing OOB failures.
7 Discussion
The paper frames learnability as global-minimum convergence shaped by teacher geometry, initialization, data, overparameterization, and optimizer dynamics. It links teacher geometry to distinct local-minimum types and shows that differential learning rates can improve optimization success.
- The paper defines teacher-student learnability as the probability that gradient-based optimization finds the global minimum, depending on teacher geometry and other training factors.
- Minimally dissimilar and standard teachers more often drive neurons out-of-bounds, whereas maximally dissimilar teachers produce more interior minima.
- Increasing readout-weight learning rates while decreasing inner-bias learning rates substantially improves success rates across teacher distributions.
- The resulting framework aims to explain why some functions are easier to learn and how smaller networks might be trained by limiting network failure.
- The paper connects its focus to prior work on expressibility, approximation speed, overparameterization, and loss-landscape symmetries.
B.1 Training
The training procedure samples structured teacher networks and evaluates many independently initialized students using two-stage optimization and explicit convergence classifications. Teacher configurations vary by dimensionality, width, geometry, biases, and readout signs.
- Each teacher is evaluated with 50 independently seeded student fits using Gaussian inputs, followed by ODE-based training and second-order optimization.
- Runs are excluded when no output file is generated, and larger settings have more nonterminating runs; completed runs receive no additional convergence filter.
- Global-minimum success is defined by final loss L < 10^-18, while training stops at L = 10^-30, patience, or a two-hour stage limit.
- Runs stopping above the global-minimum threshold are classified using the minimum Hessian eigenvalue as strict minima, non-strict critical regions, or optimizer failures.
- Teacher-student overlap combines readout-weight products with embedding-vector inner products, treats double-negative ReLU pairs as non-overlapping, and maximizes total assignment with the Hungarian algorithm.
- The three teacher distributions use approximately equidistant, Gaussian, or shared-direction embedding weights, with specified bias and readout-weight constructions.
C.2 The different teacher distributions maximize and minimize expected initial similarity
Expected initial student-teacher similarity is controlled by the angle between teacher weight vectors: aligned minimally dissimilar teachers minimize it, while opposite maximally dissimilar teachers maximize it. Readout-sign differences reinforce the maximally dissimilar advantage under the paper’s settings.
- Rotational invariance reduces the two-neuron comparison to the angle between teacher vectors and permits fixed coordinate representatives without loss of generality.
- The derivation uses the norm relation ∥v∥ = 2|sin(θ/2)|, making the expected assignment difference increase with angular separation.
- The simplified expected initial similarity is minimized at teacher angle θ = 0 and maximized at θ = π, corresponding to minimally and maximally dissimilar teachers.
- In the paper’s settings, maximally dissimilar teachers also differ in readout signs, yielding greater random-initialization similarity than minimally dissimilar teachers.
- The analytic result is limited to a simplified D = 2 setting, although empirical differences in starting similarity appear in additional figures.
- For finite ReLU data domains, out-of-bounds neurons can become dead or always-on linear units, and wrong readout signs can trap them in local minima.
E.1 Derivation
The derivation reduces ReLU teacher-student loss landscapes by optimizing fast readout weights, fixing homogeneous weight norms, and averaging over Gaussian inputs. Under these assumptions, small-network landscapes become low-dimensional functions of angles and biases.
- The paper fully characterizes selected ReLU loss landscapes by reducing their parameters to two variables using simplifying assumptions and symmetry.
- Instantaneously optimized readout weights remove N student degrees of freedom because the readout optimization is convex for fixed embedding weights.
- ReLU positive homogeneity fixes student weight norms to one and absorbs their norms into readout weights, leaving angular parameters.
- Infinite-data Gaussian averaging eliminates finite-domain boundary effects and therefore excludes out-of-bounds minima from the analytical setting.
- For a single teacher and student neuron, the reduced coordinates are the teacher-student angle θ and the student bias b.
- For two unbiased neurons in D = 2, the landscape is a closed-form function over the two student angles relative to the first teacher neuron.
- A timescale hierarchy makes readout weights fastest, embedding angles intermediate, and biases slowest, so trajectories first explore the zero-bias landscape.
E.2 Results
The loss-landscape analysis finds that same-sign readouts have only global minima, whereas mixed-sign readouts create persistent local minima. Across broader settings, these qualitative patterns remain similar for other activations and Adam optimization.
- Loss-landscape analysis: Same-sign readouts produce no local minima beyond the global minimum, and all gradient-flow trajectories converge regardless of initialization.This result is shown for the two-node ReLU system with the stated teacher geometry.
- Loss-landscape analysis: Mixed-sign readouts create local minima where both student neurons align with one teacher neuron, leaving the other unmatched.Slowly evolving biases do not enable escape, indicating these minima are robust rather than artifacts of the zero-bias constraint.
- Loss-landscape analysis: Across teacher-angle sweeps from π/16 to π, same-sign systems remain free of trapped trajectories, while mixed-sign systems retain local minima at every tested angle.The basin size varies with teacher angle, but the local minima persist throughout the sweep.
- Experimental settings: The experiments vary input dimensionality and teacher-node count while keeping student width and activation equal to the teacher in the main convergence settings.The sweeps use D ∈ {2, 4, 8, 16} with M = 4, or M ∈ {2, 4, 8, 16} with D = 2.
- Classification criterion: The global-versus-local classification uses a 1 × 10^-18 loss threshold, supported by a large gap between losses below and above the threshold.The threshold is reported as comfortably above the precision reached by converged runs.
- Generalization checks: The qualitative results remain similar for softplus and tanh activations, and for Adam optimization instead of gradient flow.Additional experiments also examine readout signs, parallel hyperplanes, and starting similarity.
F.6 Extended analysis of the local minima
The extended analysis examines which local minima students reach and how initialization similarity relates to success. It reports distinct minima patterns across teacher distributions and decreasing overlap terms with increasing dimensionality.
- Minima types: Figure 17 compares the types of minima reached by students across the different teacher distributions.The figure varies teacher-node count at fixed D = 2 and input dimensionality at fixed M = 4.
- Overlap analysis: Overlap terms tend towards zero as input dimensionality increases.This trend is summarized in Figure 18 as average starting overlap versus dimensionality.
- Success prediction: Figure 19 relates starting similarity to final success using AUROC.The analysis plots individual overlaps against losses for teachers in a single setting.