Implicit Regularization in Deep Learning

• Nathan Srebro served as thesis advisor, alongside committee members Yury Makarychev, Ruslan Salakhutdinov, and Gregory Shakhnarovich.Source: TOYOTA TECHNOLOGICAL INSTITUTE AT… · p. 1

• Nathan Srebro is identified as the thesis advisor.Source: Implicit Regularization in Deep Learning · p. 3
• Nathan Srebro’s title is Professor.Source: Implicit Regularization in Deep Learning · p. 3
• The dedication is to the author’s dear friend, Sina Masihabadi.Source: Implicit Regularization in Deep Learning · p. 4

Introduction

• Overparameterized networks can be computationally hard to learn yet train and generalize in practice, while optimization choices select among training-error minima with different generalization behavior.The introduction asks what bias and capacity control explain this success.Source: Introduction · pp. 12–13

1.1 Main Contributions

• Experiments are designed to demonstrate the role of implicit regularization in deep-learning success.Source: §1.1 · p. 13
• The dissertation proves generalization bounds for bounded-norm fully connected feedforward networks, with some bounds independent of hidden-unit count.Source: §1.1 · p. 13
• It derives neural-network generalization bounds using PAC-Bayes by connecting sharpness with PAC-Bayes theory.Source: §1.1 · p. 13
• It shows that networks learned by SGD satisfy several conditions leading to flat minima.Source: §1.1 · p. 13
• Experiments compare complexity measures for explaining implicit regularization and generalization in deep learning.Source: §1.1 · p. 13
• The work characterizes rescaling invariances in feedforward and recurrent networks and proposes the rescaling-invariant Path-norm.Source: §1.1 · p. 13
• Path-SGD approximates steepest descent with respect to the Path-norm and achieves better generalization error than SGD on fully connected and recurrent network benchmarks.Source: §1.1 · p. 14

2.1 The Statistical Learning Framework

• With i.i.d. training data from unknown D, classification seeks low expected error, but zero training error can coexist with near-random expected loss, making the population–training gap central.Training error estimates expected loss, which the learner cannot directly evaluate without access to D.Source: §2.1 · p. 15

2.2 Feedforward Neural Networks with Shared Weights

• A feedforward network is a directed acyclic graph with edge weights and an activation; mapping edges to shared parameters defines a class covering standard FCNNs and shared-weight CNNs and RNNs.Network size is the number of edges, depth the longest directed path, and width the maximum vertex in-degree.Source: §2.2 · pp. 16–17
• ReLU is 1-Lipschitz, idempotent, and positively homogeneous; the latter permits reciprocal rescaling of adjacent layers without changing the represented function.The section focuses mostly on hinge, or ReLU, activation.Source: §2.2 · p. 16

Part I

• The passage contains only the fragment “Implicit Regularization and,” with no stated generalization finding.Source: Part I · p. 18

Generalization and Capacity Control

• Restricting a high-capacity class to low-complexity predictors can support generalization; training-set-dependent measures also work if chosen carefully, and capacity comparisons depend on the model class.Monotone transformations of a complexity measure are equivalent, while capacity provides only a crude yardstick for trends and relative phenomena.Source: Generalization and Capacity Control · p. 19

3.1 VC Dimension: A Cardinality-Based Arguments

• Finite-class union bounds make ln |F| a complexity measure, while VC dimension captures intrinsic cardinality; parameter-count measures are too weak to explain overparameterized generalization.In practice, networks can fit random labels, and increasing hidden units can reduce generalization error without reducing parameter count.Source: §3.1 · p. 20

3.2 Norms and Margins: Counting Real-Valued Functions

• Rademacher-complexity bounds for feedforward networks depend on layerwise maximum incoming-weight ℓ1 norms, motivating norm-based capacity control.The discussion extends capacity control to a large family of norms.Source: §3.2 · p. 21

3.3 Robustness: Lipschitz Continuity with Respect to Input

• The bounded-Lipschitz capacity result is weak (exponential in input size) [29], while network Lipschitz bounds grow exponentially in input dimension and depth, so Lipschitz control alone is insufficient.Robustness capacity scales with the number of covering sets, which can itself be exponential in input dimension [30].Source: §3.3 · pp. 21–22

3.4 PAC-Bayesian Framework: Sharpness with Respect to Parameters

• In the PAC-Bayes view, expected sharpness is only one bound term and must be balanced with norm; together they provide capacity control and explain observed phenomena.The bound also depends on KL divergence to a data-independent prior and holds for any prior, perturbation distribution, and training-dependent choice of weights.Source: §3.4 · pp. 22–23

On the Role of Implicit Regularization in Generalization

• Because fitting even small networks is computationally intractable, the authors argue that representational capacity alone cannot explain learning success, and propose hidden capacity control instead.They suggest optimization implicitly favors low-complexity solutions; experiments find this behavior without explicit regularization, and weight decay brings only slight further improvement.Source: On the Role of Implicit Regularization… · p. 25, §4.1 · p. 27

4.1 Network Size and Generalization

• On MNIST and CIFAR-10, test error kept decreasing as network size grew beyond zero training error, even past the number of training examples.The reported behavior contradicts the expectation that larger networks should overfit once they fit the training set.Source: §4.1 · p. 26, Figure 4.1 · p. 27
• With censored data and with five percent random-label noise, test error still decreased beyond zero training error as network size increased.The censored data were constructed so approximation error was already zero at a small network size; the noisy-label experiment found no significant overfitting.Source: §4.1 · pp. 26–27, Figure 4.2 · p. 28

4.2 A Matrix Factorization Analogy

• In matrix factorization, limiting hidden units imposes a rank constraint, whereas norm control can yield tractable learning, generalization, and convergence to global minima.The authors use this contrast to motivate low factorization norm as a more suitable inductive bias than dimensionality.Source: §4.2 · p. 29
• The authors propose that optimization may implicitly favor low-norm network solutions, making finite networks approximations to infinite-width bounded-norm models rather than capacity controls.They cite evidence that gradient descent on full-dimensional matrix factorizations can converge to a minimum trace-norm solution under suitable initialization and step size.Source: §4.2 · pp. 29–30

Norm-based Capacity Control

• Per-unit regularization controls capacity independently of network size only with per-unit ℓ1 norms, while overall ℓp regularization does so only for p ≤ 2; exponential depth dependence remains unavoidable.This depth dependence persists even when the sum of all weight magnitudes is bounded.Source: Norm-based Capacity Control · pp. 31–32

5.1 Group Norm Regularization

• The group norm takes an ℓp norm over each unit’s incoming weights and an ℓq norm across units, unifying per-unit (q = ∞) and overall (q = p) regularization.For ReLU networks, the two weight measures define the same model classes.Source: §5.1 · pp. 32–33

5.1.1 Generalization and Capacity

• When q ≤ p* (equivalently, 1/p + 1/q ≥ 1), the group-norm Rademacher bound loses width dependence, allowing infinite-width capacity control.This includes overall ℓ1 and ℓ2 regularization and per-unit ℓ1 regularization.Source: §5.1.1 · p. 34

5.1.2 Tightness

• When 1/p + 1/q < 1, width dependence is unavoidable and capacity can be infinite; more generally, group-norm control entails sample complexity that grows exponentially with depth.In this regime, increasing width and depth can lower the magnitude control without lowering capacity.Source: §5.1.2 · p. 35

5.1.3 Convexity

• Magnitude-controlled, unbounded-width model classes can be convex, although the analysis also identifies a regime that is only quasi-convex.This is function-space convexity; the empirical-risk objective over weights remains nonconvex for depth ≥ 2.Source: §5.1.3 · pp. 35–36

5.2 Per-Unit and Path Regularization

• For layered graphs, path and per-unit regularization are equivalent after weights are balanced; although this equivalence fails for non-layered graphs generally, it holds again over all DAGs with depth at most d.Source: §5.2 · pp. 36–37
• Under the stated complexity assumptions, efficient PAC learning is ruled out for unit-margin functions in the per-unit class at specified norm bounds, though sublinear bounds may still permit poly(n_in)-time learning.The passages give lower bounds under strong random CSP assumptions and, for another norm regime, intractability of a unique shortest vector problem; they do not rule out poly(n_in)-time learning when ψ_1,∞ is sublinear.Source: §5.2 · pp. 37–38
• Per-unit regularization can represent any function with an optimal-norm network whose non-input nodes form a tree, so it does not encourage sharing lower-level units.The paper argues that this misses an inductive bias of deep learning: sharing features across downstream units or tasks.Source: §5.2 · p. 38

5.3 Overall Regularization

• For overall ℓp regularization, p ≤ 2 gives width-independent generalization guarantees, whereas p > 2 yields infinite-capacity classes at depth d ≥ 3 and even an ℓ1 bound has complexity exponential in depth.The width-independent guarantee requires the corresponding dual norm of the inputs to be bounded.Source: §5.3 · p. 38
• Among the overall-norm classes discussed, F^2_{2,2} is the viable convex case: it is equivalent to a convex neural net with ℓ2-constrained bottom-layer units and an ℓ1-regularized output layer.Although the convexity conditions extend to p ≥ d, for depth greater than two those values lead to infinite capacity without width control, while width control removes convexity.Source: §5.3 · p. 39

5.4 Depth Independent Regularization

• For unbounded-depth networks, ψ_p,q remains a sensible complexity measure, but the available generalization bound retains exponential depth dependence; the stated theorem covers anti-symmetric 1-Lipschitz activations.For ReLU, the paper shows that the inductive approach cannot avoid exponential depth dependence because [conv(H)]+ can increase Rademacher complexity.Source: §5.4 · p. 40

5.5.1 Rademacher Complexities

• The Rademacher-complexity analysis bounds the depth-d, width-H ReLU class with bounded layer-wise group norm by induction on depth, starting from linear separators.Source: §5.5.1 · p. 41

ℓp-regularized Linear Predictors

• The proof bounds network Rademacher complexity by reducing matrix maximization to H independent vector maximizations, applying contraction across layers, and using Lemma 17 to bound first-layer complexity.The cited statement of Lemma 17 specifies 1 ≤ p ≤ 2.Source: ℓp-regularized Linear Predictors · pp. 41, 43, 44

5.5.2 Proof that ψd p,q(w) is a semi-norm in Fd

• The paper proves that ψ^d_{p,q} is a seminorm in F_d by establishing convex sublevel sets and non-negative homogeneity, which together imply the triangle inequality.The stated convexity argument applies under the theorem’s conditions on d, p, and q.Source: §5.5.2 · pp. 46–47

5.5.3 Path Regularization

• A weight-rescaling procedure preserves the represented function while balancing incoming ℓp norms across layers without increasing ψp,∞; Theorem 7 gives a path-regularization bound for p ≥ 1.The balancing procedure reaches equal incoming norms for each internal layer, and the result covers finite or infinite width.Source: §5.5.3 · p. 48
• Any depth-d DAG can be transformed into an equivalent layered network by inserting vertices on edges while preserving its function and φ.The construction terminates after at most ρd − T0 steps, where ρ is the number of input-output paths and T0 is the initial sum of path lengths.Source: §5.5.3 · pp. 49–50

5.5.4 Hardness of Learning Neural Networks

• Theorem 24 shows that bounded-norm neural networks can realize intersections of homogeneous halfspaces with unit margin; therefore, hardness of learning those intersections transfers to any class that represents them.The cited hardness results are conditional: Daniely et al. [11] use a strong random CSP assumption, and uses an intractability assumption.Source: §5.5.4 · p. 50

5.6 Discussion

• For depth d > 2, per-unit control with p > 1 and overall regularization with p > 2 do not control capacity without network-size control; linear models have weak dimensional dependence for any finite p, while two-layer networks suffice with per-unit p = 2.The discussion also reports ℓ2 generalization guarantees, but notes that they retain an exponential dependence on depth.Source: §5.6 · p. 51
• Although additive µ-measure and multiplication ψ-measure are equivalent at the optimum, they can behave differently during optimization; the path-based regularizer may also help practical neural-network regularization.The claim about optimization dynamics is based on anecdotal empirical experience.Source: §5.6 · p. 51
• The polynomial dependence of margin-based classification bounds on the norm may not be tight, and Theorem 3 gives the same bound for per-unit and overall ℓ1 regularization despite an expected lower capacity for the latter.Improving the bound may require going beyond Rademacher-complexity bounds for the real-valued class.Source: §5.6 · p. 51
• Open questions include whether ψ-based capacity control can be depth-independent and how much expressive power additional depth provides: F2 already contains all binary functions over {±1}^n and is dense among continuous real-valued functions.The ψ-measure is monotonically non-increasing with depth, but whether it keeps decreasing for any functions—and which functions need high depth for efficient representation—remains unresolved and relates to circuit-complexity problems.Source: §5.6 · p. 51

6.1 Spectrally-Normalized Margin Bounds

• Theorem 26 gives a PAC-Bayes generalization guarantee for ReLU feedforward networks using a perturbation bound controlled by layer spectral norms.The proof first chooses perturbations that satisfy the margin condition, then computes the corresponding PAC-Bayes KL term.Source: §6.1 · p. 52

6.2 Generalization Bound based on Expected Sharpness

• Conditions C1–C3 bound expected sharpness by preventing weak layer interactions, extreme activation changes, and large-weight inactive units from activating.These conditions are used to establish a ReLU-network generalization bound in terms of average sharpness and norm.Source: §6.2 · pp. 53–54
• The expected-sharpness generalization bound depends linearly on depth, not exponentially, and worsens as μ decreases or C2 and C3 increase.Choosing the perturbation scale σ balances the sharpness and norm terms in the bound.Source: §6.2 · p. 54
• Experiments report that trained networks satisfy C1–C3, with μ ≥ 1/4, C2 ≤ 5, and C3 ≤ 3.The conditions were numerically checked on a 10-layer, 1000-hidden-unit-per-layer MNIST perceptron with more than 10,000,000 parameters.Source: §6.2 · p. 54, Figure 6.1 · p. 54

6.3.1 Supporting Lemma

• Lemma 30 considers matrices A and B and an entrywise Gaussian random matrix u with dimensions conforming to their matrix products.The passage specifies A and B as n1 × n2 and n3 × n4, and u as n2 × n3.Source: §6.3.1 · p. 56

6.3.2 Conditions in Theorem 27

• The learned network is reported to be very similar to random initialization under conditions C1, C2, and C3.The comparison uses a 10-layer feedforward network with 1000 hidden units per layer trained on MNIST.Source: §6.3.2 · p. 56, Figure 6.2 · p. 57, Figure 6.3 · p. 57

6.4.1 Proof of Lemma 25

• An induction argument has base case Δ₀ = |x − x|₂ = 0; for i ≥ 1, activation Lipschitzness and φ(0) = 0 support an ℓ₂ layer-output bound by |x|₂ times a product over layers.Source: §6.4.1 · pp. 56–57

6.4.2 Proof of Lemma 28

• For Lemma 28, the proof defines a network variant by replacing the weights in two layers with U_i and U_j.The supplied passage does not show the resulting bound.Source: §6.4.2 · p. 58
• The Lemma 28 proof first handles terms with one noisy layer, using Lemma 30 and condition C1 for two steps.The base-case passage introduces the network with weights in one layer replaced by u_k.Source: §6.4.2 · p. 58
• The induction step assumes the claim for k noisy layers and proves it for k + 1 by substituting into equation (6.4.1).The supplied passages outline this step but omit the expressions being substituted.Source: §6.4.2 · pp. 58–59

Proof of Lemma 29

• The proof proceeds by induction on k, tracking activation-pattern changes and extending the bound from k to k + 1.It represents perturbed activation patterns with diagonal indicator matrices.Source: §6.4.3 · pp. 59, 61
• For k = 1, the proof uses Gaussian perturbation u1 and ||x||≤1, then bounds D1’s change separately because it depends on u1.The expectation for this bound is taken only with respect to u1.Source: §6.4.3 · p. 59
• At k = 2, the proof expresses Err2 as the difference between perturbed and unperturbed activation paths, then splits it into two expected terms.The terms separate contributions involving changes in both activation patterns from those involving only the first-layer change.Source: §6.4.3 · p. 61
• At the induction step, the proof substitutes bounds for the activation-pattern change and Errk, then considers all replacements of Wi with Ui as in Lemma 28.This replacement-based bounding approach yields the result.Source: §6.4.3 · p. 62

7.1 Complexity Measures

• For zero-one loss, norms alone can be made arbitrarily small by scaling weights, so meaningful comparison requires accounting for output scale, such as with margins; PAC-Bayes combines norm and sharpness.The bound’s norm–sharpness balance depends on perturbation variance, so sharpness alone does not control capacity.Source: §7.1 · pp. 63–64

7.2 Experiments Settings

• Experiments use two-layer ReLU perceptrons for varying network sizes and a modified VGG for other settings, with norms divided by the fifth-percentile training margin.Batch-normalized networks are reparameterized to preserve the same function when calculating norms.Source: §7.2 · pp. 64–65

7.3 True Labels Vs. Random Labels

• With true versus random CIFAR10 labels, complexity separates the models for all four norms, while joint sharpness and norm better distinguish generalization than sharpness alone.At a fixed expected sharpness, random-label networks require higher norm; the bi-criteria plots also show a larger capacity increase as sample size grows with random labels.Source: §7.3 · pp. 65–66, Figure 7.1 · p. 66, Figure 7.2 · p. 66

7.4 Different Global Minima

• As the random-label confusion set grows, test error rises despite zero training error; norm and joint PAC-Bayes sharpness–KL behavior are more predictive than sharpness alone.For a fixed expected sharpness, networks with higher generalization error have higher norms.Source: §7.4 · p. 67, Figure 7.3 · p. 67

7.5 Increasing Network Size

• For larger MNIST networks, margin/norm measures and joint PAC-Bayes complexity decrease up to 128 hidden units, but none of the examined measures explains generalization across all sizes.Beyond 512 units, the spectral norm increases; the joint PAC-Bayes measure also fails to explain the observed generalization behavior at larger sizes.Source: §7.5 · pp. 67–68, Figure 7.4 · p. 68

Part II

• The Part II passage begins “Geometry of Optimization and” but provides no substantive generalization claim.Source: Part II · p. 69

8.1 Invariances in Feedforward and Recurrent Neural Networks

• Gradient descent is not invariant to function-preserving node-wise rescalings, so equivalent ReLU networks can follow different updates and become functionally different after one step.Scaling down an edge also scales up its gradient, contrary to what a rescaling-invariant update would require.Source: §8.1 · pp. 71–72
• For ReLU RNNs, every feasible node-wise rescaling is captured by the stated Tα transformation, and the network is invariant to these transformations.Weight sharing restricts which node-wise rescalings are feasible.Source: §8.1 · p. 72

8.2 Understanding Invariances

• Infinitesimal node-wise rescalings span all model invariances when dim N(w) equals the number of internal nodes, but parameter-specific degeneracies can create additional invariances.A ReLU node whose output is always zero is insensitive to changes in its outgoing weights; considering invariance uniformly over inputs can also miss transformations that hold for many, but not all, inputs.Source: §8.2 · p. 73

8.2.1 Path-based characterization of the network

• The path-based characterization separates parameter and input dependence: model degrees of freedom are at most rank J(w), with equality when the path-feature span is full.For distribution-dependent degrees of freedom, equality requires a full-rank path covariance matrix.Source: §8.2.1 · pp. 74–75
• The full path-feature span is not guaranteed: inactive ReLU units or inserted linear layers can increase apparent path counts without increasing the span.The passage conjectures that configurations with full path-feature span exist and become more numerous as networks grow.Source: §8.2.1 · p. 75

8.2.2 Combinatorial characterization of the rank of path Jacobian

• Generically, rank J(w) equals |E| − |Vinternal|; when the path-feature span is full, node-wise rescaling is the only invariance for the fixed architecture.The generic-rank claim excludes a parameter set of zero Lebesgue measure.Source: §8.2.2 · p. 75, §8.2.1 · p. 75

8.3.3 Proof of Theorem 33

• Generically, rank J(w) equals rank of the path-incidence matrix M, whose rank counts independent paths and equals parameters minus internal nodes.The proof counts paths inductively in a layered graph: paths over final-layer edges are independent, while each additional edge from the penultimate internal layer adds only one independent path because other paths can be formed by combinations.Source: §8.3.3 · p. 78

9.1 Path-regularizer

• The path-regularizer is the minimum ℓ2 per-unit regularizer over rescaling-equivalent feedforward networks, equivalently the sum over input-output paths of products of squared weights.It can be computed by dynamic programming in a single forward step, avoiding an explicit sum over exponentially many paths.Source: §9.1 · p. 81

9.2 Path-SGD for Feedforward Networks

• Path-SGD approximates steepest descent with respect to the path-norm using a diagonal quadratic approximation, and its stochastic version estimates gradients from a subset of training examples.The feedforward formulation assumes networks without shared weights; the section then extends the method to shared-weight networks.Source: §9.2 · p. 81

9.3 Extending to Networks with Shared Weights

• For shared-weight networks, the path-regularizer steepest-descent update is invariant to feasible node-wise rescalings, as is a simpler update using only κ(1).The κ(2) term captures interactions between edges sharing a parameter on the same path: it vanishes when no such pair exists, as in feedforward and convolutional networks, but may be nonzero in RNNs.Source: §9.3 · p. 82

9.3.1 Simple and Efficient Computations

• For RNNs, standard minibatch-gradient work requires O(BdTH2), with T the propagation length, d the layer count, H hidden units per layer, and B minibatch size; κ(1) has minimal per-minibatch cost, while κ(2) might be expensive for large networks.Source: §9.3.1 · p. 83

10.1 Experiments on Fully Connected Feedforward Networks

• Path-SGD performs similarly across balanced and unbalanced initializations, while unbalanced initialization hurts SGD and AdaGrad; it often reaches comparable accuracy faster and sometimes has lower final generalization error, even at zero training error.The reported learning curves show the initialization contrast, faster convergence, and cases of lower final test error.Source: §10.1 · pp. 87–89, Figure 10.2 · p. 89

10.2.1 The Contribution of the Second Term

• Path-SGD with and without κ(2) performs similarly and both versions outperform SGD; κ(2) contributes more for smaller hidden size and longer RNNs, but is often negligible, so later experiments use κ(1) alone.The second term can be computationally expensive for large networks.Source: Figure 10.3 · p. 90, §10.2.1 · p. 90

10.2.2 Addition problem

• On addition, path-normalized identity-initialized RNNs achieve 0% error at sequence length 750, while all other methods, including LSTMs, fail; on sequential MNIST, RNN-Path often outperforms other RNN methods except LSTMs.RNNs, including those trained with Path-SGD, fail to learn addition sequences of length 1000.Source: Table 10.2 · p. 91, §10.2.2 · p. 91

10.2.3 Language Modeling Tasks

• Path-normalization outperforms RNN-ReLU and RNN-tanh, narrowing the performance gap to LSTM by 57% on PTB and 54% on text8.The experiments evaluate language modeling on PTB-c and text8 using bits-per-character (BPC).Source: Table 10.3 · p. 91, §10.2.3 · p. 92

Data-Dependent Path Normalization

• Choosing a per-node complexity measure and normalization matrix R lets the framework interpolate between data-independent Path-SGD and data-dependent batch normalization.The framework also permits mixing the extent of data-dependence with either normalized reparameterization or approximate steepest descent.Source: Data-Dependent Path Normalization · p. 93

11.3 DD-Path-SGD: A Steepest Descent Approach

• DDP-SGD takes approximate steepest-descent steps with respect to the overall complexity measure, and its updates can be computed efficiently with forward and backward propagation.Using stochastic estimates of the loss gradient gives the DDP-SGD updates.Source: Data-Dependent Path Normalization · p. 93, §11.3 · p. 98

Relation to the Natural Gradient

• With full data-dependence, DDP-SGD is an efficient approximation to the natural gradient using only the diagonal of the Fisher information matrix.The diagonal natural-gradient update is equivalent to DDP-SGD for a second-moment choice of Rv.Source: Data-Dependent Path Normalization · p. 93, Relation to the Natural Gradient · p. 99

11.1 A Unified Framework

• The unified framework defines node complexity γv through a positive semidefinite Rv, ranging from incoming-unit complexities to data-dependent covariance or second-moment matrices.Summing node complexities yields a network measure that agrees with the ℓ2-path regularizer, the sum of squared weight products over paths [58].Source: §11.1 · p. 95

11.2 DD-Path Normalization: A Batch Normalization Approach

• With Rv equal to covariance, DDP-Normalization is essentially batch normalization and reparameterizes weights so internal-node complexity is fixed at one.The reparameterization changes optimization geometry while retaining the standard feed-forward model class; its gradients can be computed through the chain rule on a batch.Source: §11.2 · p. 96, §11.5.1 · p. 101

11.4 Node-wise invariance

• DDP-SGD is invariant to node-wise rescaling, whereas DDP-Normalization does not have favorable invariance properties.For feed-forward ReLU networks, DDP-SGD's node-wise rescaling invariance follows from the corresponding incoming- and outgoing-weight updates.Source: §11.4 · p. 100, §11.4.1 · p. 100

11.4.2 SGD on DDP-Normalized networks

• DDP-Normalized networks remain unchanged when incoming weights to an internal node are rescaled, but the SGD update rule is not invariant to that transformation.The network invariance follows because each node's output is normalized.Source: §11.4.2 · p. 101

11.5.1 Implementation of DDP-Normalization

• DDP-normalization calculations can be implemented as vector operations with negligible extra memory and computation.Source: §11.5.1 · p. 101

11.5.2 Implementation of DDP-SGD

• The DDP-SGD implementation calculates first derivatives and propagates the loss through γ2 to obtain second-order terms for parameter partials.Source: §11.5.2 · pp. 101–102

11.5.3 Natural Gradient

• Natural gradient applies the inverse Fisher information matrix at the current parameters to the negative gradient direction to achieve invariance.The Fisher matrix's entries can be computed efficiently by forward and backward propagation on a minibatch.Source: §11.5.3 · pp. 102–103

Conclusion

• The dissertation identifies optimization-induced implicit regularization as central to generalization and reports algorithms that generalize better than SGD on feedforward and recurrent networks.It also proves generalization guarantees based on different neural-network complexity measures and designs algorithms to implicitly regularize measures it considers more suitable for neural networks.Source: Conclusion · p. 104