PaperlayerView exact revision on arXiv

Source-linked AI summary

Dive into Deep Learning

Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola

arXiv:2106.11342v5cs.LGcs.AIcs.CLcs.CV

TL;DR

As deep learning advances across diverse practical fields, accessible resources must connect its concepts with context and runnable practice. Dive into Deep Learning addresses this through an open-source, notebook-based treatment, while highlighting that vision Transformers outperform ResNets by a large margin when scaling to 300 million images.

  • Problem

    Rapid progress across diverse deep-learning fields creates a need for resources that connect concepts, context, and practical problem solving.

    Source: Preface · p. 25

  • Method

    The book integrates exposition, figures, mathematics, interactive examples, and self-contained runnable code in Jupyter notebooks.

    Source: §18.3 · p. 886

  • Results

    Vision Transformers outperform ResNets by a large margin on image classification with 300 million images, demonstrating superior scalability.

    Source: §11.8.6 · p. 511

  • Takeaways & Limitations

    The book provides a practical entry point for learning deep learning concepts and implementing models across modern applications.

    Source: Preface · p. 25, §18.3 · p. 886

    Linear models can fail for image classification because pixel significance depends on complex interactions among surrounding pixels.

    Source: Limitations of Linear Models · p. 211

Abstract

from arXiv · show

This open-source book represents our attempt to make deep learning approachable, teaching readers the concepts, the context, and the code. The entire book is drafted in Jupyter notebooks, seamlessly integrating exposition figures, math, and interactive examples with self-contained code. Our goal is to offer a resource that could (i) be freely available for everyone; (ii) offer sufficient technical depth to provide a starting point on the path to actually becoming an applied machine learning scientist; (iii) include runnable code, showing readers how to solve problems in practice; (iv) allow for rapid updates, both by us and also by the community at large; (v) be complemented by a forum for interactive discussion of technical details and to answer questions.

About This Book … 2 Preliminaries

This section presents Dive into Deep Learning as a free, technically substantial, runnable resource that connects concepts, context, mathematics, code, and community-supported learning for practical deep learning. It develops the foundations needed throughout the book, covering machine learning formulations, deep learning’s context and applications, data and computational tools, linear algebra, calculus, automatic differentiation, probability, statistics, and uncertainty.

One Medium Combining Code, Math, and HTML

  • Its goals are free access, sufficient depth for aspiring applied machine learning scientists, runnable practical code, rapid community updates, and an interactive discussion forum.GitHub, Jupyter notebooks, Sphinx, and Discourse combine code, equations, text, publishing, and community contributions.Source: One Medium Combining Code, Math, and… · pp. 26–27

Learning by Doing

  • The book teaches concepts just in time through self-contained working examples, generally presenting tools before fully explaining them so readers can solve problems quickly.The one-example-per-notebook policy is intended to make code easy to copy and modify for research projects.Source: Learning by Doing · p. 27
  • Basic tutorials explain deep learning from scratch and often pair NumPy-like, automatically differentiated implementations with concise high-level framework APIs.After explaining a component, later tutorials rely on the framework API.Source: Learning by Doing · pp. 27–28

Content and Structure

  • The book progresses from preliminaries and fundamental models to modern deep learning techniques, then to scalability, efficiency, and applications in real systems.Its coverage includes linear algebra, calculus, probability, regression, classification, CNNs, RNNs, attention mechanisms, optimization, computational performance, computer vision, and language models.Source: Content and Structure · pp. 28–29

Code

  • Most sections provide executable code because trial-and-error remains useful where deep learning theory does not yet fully explain why techniques work.The lightweight d2l package reduces repetition, while most code is based on PyTorch and tested against its latest stable version at writing.Source: Code · pp. 29–30

Target Audience

  • The intended audience includes students, engineers, and researchers with modest linear algebra, calculus, probability, and Python knowledge, but no prior deep-learning or machine-learning background.The book prioritizes intuition and ideas over mathematical rigor and provides an online mathematics refresher.Source: Target Audience · p. 30

Notebooks, Website, GitHub, and Forum

About This Book

  • The book unifies problem formulation, mathematics, optimization, generalization, engineering, prose, figures, and code in one comprehensive deep-learning resource.It addresses the lack of resources combining technical depth, current practice, textbook-quality exposition, and clean runnable code.Source: One Medium Combining Code, Math, and… · p. 26, Summary · p. 32

Probability and Information Theory

  • Machine learning improves performance from observational data or environmental interactions, addressing changing, highly complex, or poorly understood tasks that resist explicit programming.Deep learning is presented as a powerful family of techniques applied across computer vision, natural language processing, healthcare, and genomics.Source: Probability and Information Theory · p. 42

1.1 A Motivating Example

  • A wake-word system illustrates programming with data: choose a parameterized model family, use labeled audio examples to tune parameters, and iteratively improve predictions.The model maps audio snippets to {yes, no} outputs, while the learning algorithm selects parameter values according to a performance measure.Source: §1.1 · p. 43

1.2 Key Components

  • Machine-learning systems combine data, a model, an objective function, and an algorithm that adjusts model parameters to optimize that objective.The book introduces supervised learning as prediction of unknown labels from known inputs and uses these components across learning problems.Source: §1.2 · p. 44

1.2.1 Data

  • Datasets contain examples represented by features, with supervised targets called labels; deep learning handles varying-length data comparatively gracefully, but requires large and appropriate datasets.Poor, biased, or unrepresentative data can cause predictive failure and automate historical injustices, especially in sensitive applications.Source: §1.2.1 · pp. 45–46

1.2.2 Models

  • Models are computational machinery that transform inputs into predictions, while deep models chain many successive data transformations and can address problems beyond classical methods.The book also discusses traditional methods alongside these deeper model architectures.Source: §1.2.2 · p. 46

1.2.3 Objective Functions

  • Objective functions formalize model quality, conventionally making lower loss better; squared error is common for numerical prediction, whereas classification often minimizes error rate or a surrogate.Training loss is minimized on training data, but held-out test data is needed to assess generalization and detect overfitting.Source: §1.2.3 · pp. 46–47

1.2.4 Optimization Algorithms

  • Gradient descent searches for parameters by estimating how small perturbations affect training loss and updating each parameter in the loss-reducing direction.This optimization step follows the selection of data, model representation, and objective function.Source: §1.2.4 · p. 47

1.3 Kinds of Machine Learning Problems

  • Machine-learning problems include supervised learning, in which labeled feature–label examples support models that predict labels from input features; the surrounding preliminaries define calculus and probability notation.The notation includes derivatives and integrals, random variables, distributions, conditional probabilities, expectations, covariance, correlation, entropy, and KL-divergence.Source: Functions and Operators · p. 39, Probability and Information Theory · p. 39, §1.3.1 · pp. 47–48

1.3.1 Supervised Learning

  • Supervised learning maps labeled input features to predictions, but modeling choices vary with the type, size, and quantity of inputs and outputs.The workflow uses labeled examples to train a model, which then predicts labels for previously unseen inputs.Source: §1.3.1 · p. 48

Regression

  • Regression predicts arbitrary numerical targets, commonly by minimizing squared error, which corresponds to assuming Gaussian noise.Examples include house prices, ratings, hospital stays, surgery duration, and rainfall; linear regression can model a base charge plus an hourly rate.Source: Regression · p. 49

Classification

  • Classification predicts a class from a discrete set, using probabilities to express uncertainty and cross-entropy as the common loss.Problems may be binary or multiclass, and decisions should account for the differing benefits and harms of possible outcomes.Source: Classification · pp. 50–51

Tagging

  • Multi-label classification handles examples belonging to multiple nonexclusive classes, making it suitable for auto-tagging and large ontologies such as MeSH.PubMed indexing uses roughly 28,000 MeSH tags, and provisional machine-generated tags can reduce the typical one-year lag before manual review.Source: Tagging · pp. 51–52

Search

  • Search ranks relevant items by prominence, whereas recommender systems emphasize personalized item relevance using explicit or implicit user feedback.Search scores can be query-dependent, while recommendation models may estimate ratings or purchase probabilities and retrieve the highest-scoring items.Source: Recommender Systems · p. 53

Recommender Systems

  • Recommender systems face censored feedback and algorithm-induced feedback loops, leaving incentives and their interaction with learning as open research questions.Observed ratings reflect selective user behavior, while purchases influenced by current recommendations can make promoted items appear better and receive further promotion.Source: Recommender Systems · pp. 53–54

Sequence Learning

  • Sequence learning models variable-length inputs or outputs while retaining context, covering tagging, speech recognition, text-to-speech, translation, and other temporal tasks.Sequence-to-sequence problems may involve unequal lengths, missing alignment, reordered correspondence, or long-range conversational and medical history.Source: Sequence Learning · pp. 54–55

1.3.2 Unsupervised and Self-Supervised Learning

  • Unsupervised learning uses unlabeled data for questions such as clustering, subspace estimation, symbolic representations, causal discovery, and density modeling.Self-supervised learning creates supervision from the data itself, such as masked-word prediction or image-based objectives, often producing representations later fine-tuned for downstream tasks.Source: §1.3.2 · pp. 56–57

1.3.3 Interacting with an Environment

  • Offline learning disconnects models from the environment after data collection, whereas intelligent agents must choose actions whose effects alter future observations.This interaction raises questions about environmental memory, cooperation or opposition, and shifting dynamics.Source: §1.3.3 · pp. 57–58

1.3.4 Reinforcement Learning

  • Reinforcement learning trains agents that map observations to actions while receiving rewards, addressing delayed credit, partial observability, exploration, and distribution shift.Its framework includes applications such as robotics, dialogue systems, and video games; DQN beat humans at Atari, and AlphaGo defeated the world champion (Mnih et al., 2015; Silver et al., 2016).Source: §1.3.3 · p. 58, §1.3.4 · pp. 59–60

1.4 Roots

  • Deep learning grew from statistics, information theory, computation, neuroscience, and psychology, with layers and backpropagation central to modern neural networks.Its progress accelerated after larger datasets, cheaper computation, GPUs, improved capacity control, attention, generative models, and distributed training shifted advantages toward deep neural networks.Source: §1.4 · pp. 60–62

1.5 The Road to Deep Learning

  • Transformers scale with dataset size, model size, and training compute, enabling successes across language, vision, speech, reinforcement learning, and graph neural networks.A single Transformer pretrained across text, images, joint torques, and button presses can play Atari, caption images, chat, and control a robot.Source: §1.5 · p. 64
  • Parallel and distributed training reduced ResNet-50 training on ImageNet to under 7 minutes, compared with initial training times of days.Work using up to 1,024 GPUs and aggregate minibatches of about 32,000 images enabled this reduction.Source: §1.5 · p. 65

1.6 Success Stories

  • Deep learning now supports pervasive AI applications, while ImageNet top-five error fell from 28% in 2010 to 2.25% by 2017.Applications include speech recognition reaching human parity for some tasks, assistants, fraud detection, search, recommendation, and personalization.Source: §1.6 · p. 66
  • AI progress spans increasingly difficult games, from TD-Gammon and DeepBlue to AlphaGo and Libratus, while self-driving systems remain only partially autonomous.Deep learning combined with Monte Carlo tree sampling helped AlphaGo reach human parity in 2015, whereas full autonomy still requires perception, reasoning, and rule integration.Source: §1.6 · p. 66
  • The section emphasizes that AI’s immediate societal risks include job automation and biased consequential decisions more than malevolent superintelligence.Routine work may be automated, while statistical models can produce racial, gender, or age bias when applied without care.Source: §1.6 · p. 67

1.7 The Essence of Deep Learning

  • Deep learning replaces both shallow models and manual feature engineering with jointly optimized representations, providing unified tools across diverse application areas.Its advantages include automatically tuned filters and reduced domain-specific preprocessing, while abundant data, GPU computation, and efficient frameworks enable high performance.Source: §1.7 · p. 68, §1.8 · p. 69
  • The book is freely distributed through open-source notebooks designed to lower barriers to learning deep learning.The notebooks integrate exposition, figures, mathematics, interactive examples, and runnable code, continuing the community’s practice of sharing libraries, models, and trained networks.Source: §1.7 · p. 68

2 Preliminaries

  • The preliminaries provide the data, mathematical, differentiation, probability, and documentation skills needed to follow the book’s technical content.They begin with tensors, which support multidimensional data manipulation, automatic differentiation, and GPU-accelerated computation in modern frameworks.Source: §2 · p. 70, §2.1 · p. 70

2.1 Data Manipulation

  • Tensor fundamentals cover construction, indexing, elementwise and linear-algebra operations, broadcasting, in-place updates, and shared-memory conversion with NumPy.Broadcasting expands dimensions of length 1 to enable elementwise operations across compatible shapes, while slice assignment preserves allocated memory during updates.Source: §2.1.1 · p. 70, §2.1.2 · p. 73, §2.1.3 · p. 74

2.1.7 Summary

  • The tensor interface supports construction, indexing, slicing, mathematical operations, broadcasting, memory-efficient assignment, conversion to Python objects, and scalar extraction.A size-1 tensor can be converted to a Python scalar with item or built-in functions, while exercises reinforce conditional comparisons and broadcasting across alternative shapes.Source: §2.1.7 · pp. 77–78, §2.1.6 · p. 77, §2.1.8 · p. 78

2.2 Data Preprocessing

  • Data preprocessing converts messy real-world inputs into tensors by separating targets, handling missing values, encoding categorical fields, imputing numerical fields, and converting pandas data.Missing values may be imputed or deleted; categorical NaN values can be represented as categories, while numerical NaN values can be replaced with column means.Source: §2.2 · p. 78, §2.2.2 · p. 79, §2.2.3 · p. 80

2.2.4 Discussion

  • Real-world data processing can become a major machine-learning bottleneck because datasets may span files and data types, contain quality errors, exceed memory limits, and require visualization.The discussion mentions relational tables, text, images, audio, point clouds, outliers, faulty sensor measurements, and recording errors.Source: §2.2.4 · pp. 80–81

2.3 Linear Algebra

  • Linear algebra progresses from scalars and vectors to matrices and higher-order tensors, whose axes and shapes organize data such as records, attributes, images, and image collections.Scalars are 0th-order tensors, vectors 1st-order tensors, matrices 2nd-order tensors, images 3rd-order tensors, and collections of images 4th-order tensors.Source: §2.3 · p. 81, §2.3.1 · p. 82, §2.3.2 · p. 82

2.3.3 Matrices

  • Matrices represent m × n arrangements of scalars, support reshaping and transposition, and commonly encode datasets with rows as records and columns as attributes.A transpose exchanges rows and columns, and symmetric matrices equal their own transposes.Source: §2.3.3 · pp. 83–84

2.3.5 Basic Properties of Tensor Arithmetic

  • Elementwise tensor operations preserve operand shapes, scalar operations apply to every element, and the Hadamard product multiplies corresponding matrix entries.These operations provide basic shape-aware arithmetic across scalars, vectors, matrices, and higher-order tensors.Source: §2.3.5 · pp. 85–86

2.3.6 Reduction

  • Reductions sum or average tensor elements across all or selected axes, removing reduced dimensions and producing scalars or lower-order tensors.Summing a matrix across axis 0 produces a vector over columns, while axis 1 produces a vector over rows; mean supports analogous reductions.Source: §2.3.6 · pp. 86–87

2.3.7 Non-Reduction Sum

  • Keeping reduced axes unchanged is useful for broadcasting when calculating tensor sums or means.Non-reduction sums preserve the number of axes, including singleton dimensions in the output shape.Source: §2.3.7 · pp. 87–88
  • Non-reduction operations preserve axes: broadcasting A by row sums produces rows summing to 1, while cumsum accumulates values without reducing an axis.These operations illustrate tensor manipulation through axis-aware sums and broadcasting.Source: §2.3.7 · p. 88, §2.3.12 · p. 93

2.3.8 Dot Products

  • Dot products sum pairwise products, express weighted sums or averages, and equal cosine similarity after unit-length normalization.They can be computed through elementwise multiplication followed by summation.Source: §2.3.8 · pp. 88–89

2.3.9 Matrix–Vector Products

  • Matrix–vector multiplication forms an m-dimensional vector of row-wise dot products and transforms vectors from R^n to R^m, including in neural-network layers.The implementation requires A’s column dimension to match x’s length; the example produces tensor([ 5., 14.]).Source: §2.3.9 · p. 89

2.3.10 Matrix–Matrix Multiplication

  • Matrix multiplication computes each output entry as a row–column dot product, assembling m matrix–vector products into an n × m result distinct from the Hadamard product.Multiplication of a 2 × 3 matrix by a 3 × 4 matrix yields a 2 × 4 matrix, and generally costs cubic rather than quadratic time.Source: §2.3.10 · p. 90, §2.3.12 · p. 93

2.3.11 Norms

  • Norms encode vector or matrix magnitude and distances, including ℓ1, ℓ2, spectral, and Frobenius norms, which often define deep-learning objectives.The ℓ1 norm is less sensitive to outliers than the ℓ2 norm, and norms support objectives involving prediction errors and representation distances.Source: §2.3.11 · pp. 91–92, §2.3.12 · p. 93

2.3.12 Discussion

  • The discussion consolidates scalars, vectors, matrices, and tensors; contrasts elementwise and shape-changing products; and uses exercises to reinforce these concepts.It intentionally covers the linear algebra needed for much of modern deep learning while deferring broader mathematics and decomposition topics.Source: §2.3.12 · pp. 92–93, §2.3.13 · pp. 93–94

2.4 Calculus

  • Calculus underlies deep-learning optimization because parameter updates decrease loss, while generalization concerns performance on unseen data.The section motivates calculus through the limit-based construction of circle area and distinguishes optimization from generalization.Source: §2.4 · p. 94, Figure 2.4.1 · p. 94

2.4.1 Derivatives and Differentiation

  • A derivative measures a function’s rate of change, indicates how loss changes with parameters, supports optimizing differentiable surrogates, and equals 2 at the cited location x = 1.Numerical limits approach 2 as h approaches 0, including 2.00003 when h = 0.00001; derivatives give the slope at a particular location.Source: §2.4.1 · pp. 95–96

2.4.2 Visualization Utilities

  • Visualization utilities standardize SVG output, figure sizes, axes properties, and overlays for plotting functions and tangent lines.The saved d2l helpers support reusable plotting code, including labels, ranges, scales, legends, grids, and input-shape handling.Source: §2.4.2 · pp. 96–97

2.4.3 Partial Derivatives and Gradients; 2.4.4 Chain Rule; 2.4.5 Discussion

  • Gradients concatenate all partial derivatives, while matrix calculus and the chain rule reduce multivariate differentiation to vector–matrix products along a dependency graph.The section gives standard gradient identities, including ∇x||x||2 = 2x, and connects backward gradient computation to linear algebra and later backpropagation.Source: §2.4.3 · pp. 98–99, §2.4.4 · p. 99, §2.4.5 · p. 99

2.4.5 Discussion; 2.5 Automatic Differentiation; 2.5.5 Discussion

  • The section frames gradients as the central quantity for optimization: they determine parameter updates that lower loss, and every optimization step requires them.Automatic differentiation removes tedious, error-prone hand calculations and enables models whose gradients would be prohibitively time-consuming to derive manually.Source: §2.4.5 · p. 99, §2.5 · p. 100, §2.5.5 · p. 104

2.4.6 Exercises; 2.5.6 Exercises

  • Exercises ask readers to derive elementary differentiation rules, compute derivatives and gradients, plot tangent lines, apply the multivariate chain rule, and consider higher-order derivatives.The problems reinforce first-principles calculus, matrix-gradient practice, inverse-function differentiation, and the greater cost of second derivatives.Source: §2.4.6 · pp. 99–100, §2.5.6 · p. 105

2.5 Automatic Differentiation; 2.5.4 Gradients and Python Control Flow

  • Automatic differentiation builds computational graphs and applies the chain rule backward, making gradients practical even for deeply nested functions and dynamic Python control flow.For dynamic control flow, each input realizes a specific graph before backward is run, and the demonstrated piecewise-linear function satisfies f(a) / a equal to its gradient.Source: §2.5 · p. 100, §2.5.4 · p. 104

2.5.1 A Simple Function

  • PyTorch computes and verifies the gradient of y = 2x⊤x as 4x, while requiring explicit gradient-buffer management because repeated backward calls accumulate gradients.The example stores gradients in x.grad, verifies the expected result elementwise, and resets the buffer with x.grad.zero_() before a new calculation.Source: §2.5.1 · pp. 101–102

2.5.2 Backward for Non-Scalar Variables

  • For vector-valued outputs, PyTorch’s backward requires a supplied reduction vector, commonly summing per-example gradients to obtain a vector matching the input shape.The Jacobian is the general derivative representation, but batch losses often require summing component gradients; y.sum().backward() provides the faster equivalent.Source: §2.5.2 · p. 102

2.5.3 Detaching Computation

  • Detaching an intermediate removes its ancestors from the downstream graph, so gradients do not flow through it, while the detached computation’s own graph remains available for differentiation.With u = y.detach() and z = u * x, differentiating z yields u rather than the full derivative through y; y’s gradient with respect to x can still be computed.Source: §2.5.3 · p. 103

2.5.6 Exercises

  • The exercises apply automatic differentiation through repeated backpropagation, vector- or matrix-valued functions, dependency graphs, chain-rule gradients, and choices between forward and backward differentiation.The final example sets up Bayesian reasoning for an HIV test with a 1% false-positive rate among healthy patients and no missed detections.Source: §2.5.6 · p. 105

2.6 Probability and Statistics

  • Probability and statistics support uncertainty-aware prediction, anomaly detection, and decision-making across supervised, unsupervised, and reinforcement learning.Probability models event likelihoods, while statistics uses data to infer properties of the processes that generated them.Source: §2.6 · pp. 105–106

2.6.1 A Simple Example: Tossing Coins

  • As the number of repeated coin tosses grows, empirical frequencies converge to true probabilities, with typical errors decreasing at rate 1/√n.This illustrates the law of large numbers and the central limit theorem, while distinguishing theoretical probabilities from data-derived statistics and estimators.Source: §2.6.1 · pp. 107–109

2.6.2 A More Formal Treatment

  • Formal probability represents outcomes with a sample space, events as subsets, and a probability function satisfying nonnegativity, normalization, and countable additivity.These axioms imply that complementary events have total probability 1 and impossible events have probability 0.Source: §2.6.2 · pp. 109–110

2.6.3 Random Variables

  • Random variables map sample-space outcomes to values, with discrete variables assigning probabilities to values and continuous variables using densities over intervals.Different random variables can share one underlying sample space, allowing observations of one variable to inform beliefs about another.Source: §2.6.3 · pp. 110–111

2.6.4 Multiple Random Variables

  • Joint probabilities describe combinations of variable values, while conditional probabilities update one event after another and Bayes’ theorem reverses the conditioning order.Bayesian reasoning combines prior beliefs and likelihoods to obtain posterior beliefs, and marginalization recovers distributions by summing over variables.Source: §2.6.4 · pp. 111–112
  • Independence means conditioning on one variable does not change the distribution of another, whereas conditional independence can appear or disappear after conditioning on a third variable.Independence simplifies statistical models, but estimating dependencies is often the learning objective, such as inferring disease from symptoms.Source: §2.6.4 · p. 113

2.6.5 An Example

  • A second, less accurate test raises HIV confidence beyond the first test’s 13.06%, provided the tests are conditionally independent; repeating the same test adds no insight.The diagnosis behaves like a classifier whose ability to decide whether a patient is healthy increases with more test-result features.Source: §2.6.5 · pp. 114–115

2.6.6 Expectations

  • Expectations aggregate probabilistic outcomes, while variance and standard deviation quantify spread; the investment example has expected return 1.8×, expected utility 0.7, and variance 8.36.For vector-valued variables, expectations apply elementwise and covariance captures coordinate correlations while determining the variance of linear functions.Source: §2.6.6 · pp. 115–116

2.6.7 Discussion

  • Machine-learning uncertainty includes irreducible aleatoric randomness and reducible epistemic uncertainty, but estimation often improves only at the 1/√n rate, making later gains data- and computation-intensive.Bayes’ theorem separates likelihood from prior information, while expectations and variance support distributional reasoning such as Chebyshev’s inequality, which bounds deviations from the mean.Source: §2.6.7 · p. 117

1. How does the variance scale with the number of observations? … 5. Why is the K-fold cross-validation error estimate biased?

This section develops foundational tools for deep learning, covering probability, variance, linear regression, optimization, reusable implementations, and minibatch training. It then extends these ideas to regularization and classification while examining generalization, model selection, validation procedures, and cross-validation.

1. How does the variance scale with the number of observations?

  • The exercises apply Chebyshev’s inequality and probe how averaging random variables relates to the central limit theorem and dependence assumptions.They also ask learners to derive probability bounds, simplify Markov-chain joint probabilities, and analyze dependent diagnostic-test outcomes.Source: §1 · p. 118, §3 · p. 118

3. How does it relate to the central limit theorem?

  • The portfolio exercise connects expected return and covariance to constrained quadratic optimization, identifying the Markowitz portfolio formulation.It asks for portfolio return and variance calculations and for an optimization maximizing return subject to an upper variance bound.Source: §3 · pp. 118–119

2.7 Documentation

  • The PyTorch API can be explored by listing module contents with dir and inspecting specific callable behavior with help or notebook question marks.The examples use torch.distributions and the tensors’ ones function, while noting that official documentation and source code extend beyond the book’s coverage.Source: §2.7 · p. 119, §2.7.1 · pp. 119–120, §2.7.2 · pp. 120–121

3.1.1 Basics

  • Linear regression models targets as approximately linear functions of features with Gaussian observation noise, establishing a simple baseline for later neural-network methods.The setup uses weights and a bias to represent the conditional mean and acknowledges measurement or observation error.Source: §3.1.1 · p. 123, Model · pp. 123–124

Model

  • The model predicts with a weighted feature sum plus bias, represented compactly by vectors and by a design matrix for a dataset.The weights determine feature influence, the bias permits general affine functions, and training seeks parameters that minimize expected prediction error on new samples from the same distribution.Source: Model · pp. 123–124

Loss Function

  • Squared error measures regression loss, penalizing large deviations quadratically; averaging these losses over examples defines empirical error, which training minimizes.The quadratic form discourages large errors but can make the model excessively sensitive to anomalous data.Source: Loss Function · pp. 124–125, Figure 3.1.1 · p. 125

Analytic Solution

  • Linear regression has an analytic optimum obtained from the normal equations after absorbing the bias into the design matrix, provided the matrix has full rank.The optimum is unique when X⊤X is invertible, whereas reliance on analytic solutions would exclude most deep-learning models.Source: Analytic Solution · pp. 125–126

Minibatch Stochastic Gradient Descent

  • Gradient descent iteratively updates parameters in directions that reduce loss, but full-dataset updates can be slow and wasteful when data are redundant.This motivates practical minibatch-based optimization for models that lack tractable analytic solutions.Source: Minibatch Stochastic Gradient Descent · p. 126
  • Minibatch stochastic gradient descent balances computational efficiency and statistical considerations by updating parameters from randomly sampled batches rather than full datasets or individual observations.Batch size and learning rate are hyperparameters, and validation data is used to assess solution quality.Source: Minibatch Stochastic Gradient Descent · pp. 126–127
  • Training is nondeterministic because finite-step minibatch optimization generally does not reach exact minimizers and uses randomly selected batches.For deep networks, accurate predictions on unseen data—generalization—matter more than finding an exact parameter set.Source: Minibatch Stochastic Gradient Descent · p. 127

3.1.2 Vectorization for Speed

  • Vectorization often yields order-of-magnitude speedups while reducing implementation errors and improving code portability.The approach replaces costly Python loops with optimized linear-algebra library operations.Source: §3.1.2 · p. 128

3.1.3 The Normal Distribution and Squared Loss

  • Changing a normal distribution’s mean shifts it along the x-axis, whereas increasing its variance spreads it out and lowers its peak.The normal distribution provides the probabilistic noise model underlying the squared-loss motivation.Source: §3.1.3 · p. 129
  • Minimizing mean squared error is equivalent to maximum likelihood estimation for a linear model with additive Gaussian noise of fixed variance.The equivalence follows by minimizing the negative log-likelihood; the variance contributes only a multiplicative constant to the squared-error term.Source: §3.1.3 · pp. 129–130

3.1.4 Linear Regression as a Neural Network

  • Linear regression can be represented as a single-layer fully connected neural network whose input features connect directly to one numerical output.This representation introduces the connectivity pattern later generalized to deeper networks.Source: §3.1.4 · p. 130, Figure 3.1.2 · p. 131

3.1.5 Summary

  • The section summarizes linear regression as a compact introduction to parametric models, differentiable objectives, minibatch optimization, and evaluation on previously unseen data.It also emphasizes that deep learning draws inspiration from mathematics, linguistics, psychology, statistics, computer science, and biology rather than biology alone.Source: Biology · pp. 131–132, §3.1.5 · p. 132

3.2 Object-Oriented Design for Implementation

  • The chapter’s implementation design organizes models, losses, optimizers, and data loading into Module and DataModule abstractions, combined by Trainer for hardware-aware training.Utilities also allow class methods to be registered incrementally, keeping notebook code readable across separate fragments.Source: §3.2 · p. 134, §3.2.1 · p. 134

3.2.1 Utilities

  • Utilities support extensible classes, automatic hyperparameter attributes, and interactive progress visualization for experiments.The utilities include add_to_class, HyperParameters.save_hyperparameters, and ProgressBoard, whose every_n option smooths plotted curves by averaging neighboring points.Source: §3.2.1 · p. 135

3.2.2 Models

  • The framework separates models, data, and training: Module defines model behavior, DataModule supplies training and validation batches, and Trainer fits a model for max_epochs.Module supports learnable parameters, training and optional validation steps, while DataModule prepares data and loaders; Trainer coordinates both objects.Source: §3.2.2 · p. 136, §3.2.3 · p. 137, §3.2.4 · p. 138

3.2.4 Training

  • D2L’s modular toolkit lets projects reuse components such as optimizers, models, and datasets with minimal changes, improving conciseness and simplicity.The toolkit’s fully implemented classes are saved in the D2L library and are enriched throughout the book via @add_to_class.Source: §3.2.4 · p. 139

3.2.6 Exercises

  • The exercises direct readers to inspect the complete class implementations and investigate how removing save_hyperparameters affects access to stored attributes.They also encourage explaining the behavior by examining HyperParameters’ full implementation.Source: §3.2.6 · p. 139

3.3 Synthetic Regression Data

  • Synthetic regression data provides a controlled testbed for evaluating learning algorithms and checking whether implementations recover known parameters.The example generates 1000 two-dimensional examples from a standard normal distribution, applies a ground-truth linear function, and adds independent normal noise with µ = 0 and σ = 0.01.Source: §3.3 · p. 139, §3.3.1 · p. 140

3.3.1 Generating the Dataset

  • The synthetic dataset uses true parameters w = [2, −3.4]⊤ and b = 4.2, enabling direct comparison between estimates and ground truth.Labels are generated from the linear function plus additive noise, with configurable dataset and batch-size hyperparameters.Source: §3.3.1 · p. 140

3.3.2 Reading the Dataset

  • The custom data loader yields feature-label minibatches, shuffling training examples while preserving validation order for debugging.The inspected minibatch has X shape torch.Size([32, 2]) and y shape torch.Size([32, 1]); the didactic iterator is less efficient because it loads all data into memory and performs extensive random access.Source: §3.3.2 · p. 141

3.3.3 Concise Implementation of the Data Loader

  • The concise loader replaces the custom iterator with the framework’s TensorDataset and DataLoader APIs, which efficiently handle batching and shuffling.Training and validation are selected through slices over the stored feature and label tensors.Source: §3.3.3 · pp. 141–142
  • The data loader abstracts loading and manipulation across data sources and can compose transformations into an end-to-end processing pipeline.The framework data loader also supports querying the number of batches through its built-in __len__ method.Source: §3.3.4 · p. 142, §3.3.3 · p. 142

3.4 Linear Regression Implementation from Scratch

  • The from-scratch implementation combines a linear model, squared loss, minibatch SGD, and a training loop using tensors and automatic differentiation.The model computes Xw + b, averages squared loss over minibatches, estimates gradients, and updates parameters with a learning rate.Source: §3.4 · p. 143, §3.4.1 · p. 144, §3.4.2 · p. 144

3.4.4 Training

  • The training loop iterates through minibatches, backpropagates loss, updates parameters, and evaluates validation performance once per epoch when validation data is available.Hyperparameters such as the learning rate and number of epochs are selected separately from final evaluation in the recommended three-way split.Source: §3.4.4 · pp. 145–146
  • The trained model closely recovers the synthetic dataset’s known parameters, with errors of 0.1163 and -0.2020 for w and 0.2383 for b.The text cautions that exact parameter recovery is not generally guaranteed, whereas accurate prediction is often the primary objective.Source: §3.4.4 · p. 147

3.4.5 Summary

  • The resulting implementation assembles a data loader, model, loss function, optimizer, training procedure, and monitoring tool into a functional small-problem system.The book presents this as a foundation for later implementations that reduce boilerplate and improve efficiency, including GPU use.Source: §3.4.5 · p. 147

3.5 Concise Implementation of Linear Regression

  • Modern frameworks provide predefined data iterators, losses, optimizers, and layers, allowing the same linear regression model to be implemented more concisely than from scratch.The concise version uses PyTorch’s LazyLinear layer and MSELoss, whose default behavior averages mean squared error over examples.Source: §3.5 · p. 149, §3.5.1 · pp. 149–150, §3.5.2 · p. 150

3.5.4 Training

  • The trained linear regression model recovers parameters close to those that generated the finite synthetic dataset, with weight errors [0.0031, -0.0099] and bias error [0.0127].The text explicitly compares learned parameters with the true parameters and reports the resulting estimation errors.Source: §3.5.4 · p. 151

3.5.3 Defining the Optimization Algorithm / 3.5.4 Training

  • Framework defaults simplify optimization and training while providing tested, optimized components, but researchers must still understand and implement these components when developing new ones.The implementation uses PyTorch’s data, layer, loss, optimizer, and training-loop facilities; high-level APIs reduce boilerplate as models become more complex.Source: §3.5.4 · p. 151, §3.5.5 · p. 152

3.6 Generalization / 3.6.1 Training Error and Generalization Error

  • Generalization asks whether models discover patterns that transfer beyond finite training data, under the standard IID assumption that training and test examples share a distribution.Training error is computed on observed examples, whereas generalization error is the expected loss over the underlying distribution and is estimated using withheld data.Source: §3.6 · p. 153, §3.6.1 · p. 154

3.6.1 Training Error and Generalization Error

  • Training error is generally biased because the fitted model depends on the training sample, whereas evaluating a fixed classifier on independent test data reduces error estimation to mean estimation.The generalization error itself cannot be calculated exactly because the population density and an infinite stream of examples are unavailable.Source: §3.6.1 · p. 154

Model Complexity

  • A sufficiently expressive model can fit arbitrary labels, so low training error alone neither certifies low generalization error nor rules out high generalization error.Deep neural networks exemplify this limitation; holdout or validation data is therefore needed to assess generalization after training.Source: Model Complexity · p. 155

Polynomial Curve Fitting / 3.6.2 Underfitting or Overfitting?

  • Higher-degree polynomials can always match or improve training error and may fit every example perfectly, illustrating how increasing model complexity produces underfitting and overfitting tradeoffs.The relationship between polynomial degree, underfitting, and overfitting is illustrated in Fig. 3.6.1.Source: Polynomial Curve Fitting · p. 156, Figure 3.6.1 · p. 157

Dataset Size / 3.6.4 Summary

  • Increasing dataset size typically decreases generalization error and makes overfitting less likely, while more complex models generally require more data.For fixed tasks and distributions, model complexity should not increase faster than data availability; deep learning often surpasses linear models when many thousands of examples are available.Source: Dataset Size · p. 157, §3.6.4 · p. 158

3.6.3 Model Selection / Cross-Validation / 3.6.4 Summary

  • Model selection should use validation data or K-fold cross-validation rather than training or repeatedly reused test data, with K-fold training and validation averaged across K held-out subsets.The book notes that its reported experimental accuracies are generally validation accuracies rather than true test-set accuracies.Source: §3.6.3 · pp. 157–158, Cross-Validation · p. 158, §3.6.4 · p. 158

3.7.1 Norms and Weight Decay

  • Weight decay adds an ℓ2 penalty to the loss, shrinking weights toward zero and continuously controlling function complexity through λ.λ = 0 recovers the original loss, whereas larger λ values impose stronger constraints; biases may be excluded from regularization.Source: §3.7.1 · pp. 160–161
  • ℓ2 regularization penalizes large weight components and tends to distribute weight across features, whereas ℓ1 regularization can drive weights to zero and perform feature selection.The distinction can affect robustness to measurement error and whether unused features need to be collected, stored, or transmitted.Source: §3.7.1 · p. 161

3.7.2 High-Dimensional Linear Regression

  • In a synthetic regression problem with d = 200 inputs and only 20 training examples, weight decay was used to expose and mitigate overfitting.The labels followed an underlying linear function corrupted by Gaussian noise with standard deviation 0.01.Source: §3.7.2 · pp. 161–162

3.7.3 Implementation from Scratch

Using Weight Decay

  • Weight decay increased training error while decreasing validation error in the high-dimensional regression example, correcting the overfitting observed without regularization.Without weight decay, training error fell but validation error did not; with substantial weight decay, the opposite validation trend appeared.Source: Using Weight Decay · p. 163, Training without Regularization · p. 163

3.7.4 Concise Implementation

  • The concise optimizer implementation produced a similar plot while running faster and requiring less code than the from-scratch version.The framework applies weight decay directly through the optimizer and can assign different decay policies to weights and biases without additional computational overhead.Source: §3.7.4 · p. 164

4.1 Softmax Regression

  • Classification asks which category applies rather than how much, and may require either hard assignments, class probabilities, or multiple simultaneous labels.The section distinguishes ordinary classification from soft assignments and multi-label classification.Source: §4.1 · pp. 167–168

4.1.1 Classification

  • For three unordered classes, one-hot encoding represents each label as a three-dimensional vector, while the linear model uses one affine output per class.With four input features and three categories, the model has 12 weight scalars and 3 bias scalars, represented compactly as o = Wx+b.Source: §4.1.1 · p. 168, Linear Model · p. 169

The Softmax

  • Softmax transforms logits into nonnegative class probabilities that sum to 1, while preserving their ordering for prediction.Direct vector-valued regression provides no guarantee of valid probabilities, motivating the exponential-and-normalization construction.Source: The Softmax · pp. 169–170

Vectorization

  • Vectorization computes minibatch logits with XW and applies softmax rowwise, while numerical safeguards prevent overflow and underflow.For n examples with d input dimensions and q categories, X ∈ R^n×d, W ∈ R^d×q, and b ∈ R^1×q.Source: Vectorization · pp. 170–171

4.1.2 Loss Function

  • Maximum likelihood yields the negative log-likelihood, or cross-entropy loss, for one-hot labels under independent class draws.The loss is bounded below by 0 for probability predictions, reaches 0 only with certainty on the true class, and can become infinite for an assigned probability of 0.Source: §4.1.2 · p. 171, Log-Likelihood · pp. 171–172

Softmax and Cross-Entropy Loss

  • The softmax–cross-entropy gradient is the difference between predicted probabilities and one-hot outcomes, making gradients easy to compute.This mirrors the prediction-minus-observation gradient in regression and follows the exponential-family log-likelihood structure.Source: Softmax and Cross-Entropy Loss · p. 172

Cross-Entropy Revisited

  • The lowest possible cross-entropy from P to Q occurs when P = Q, where it equals the entropy H(P).Entropy is the expected surprisal under the true distribution, and one nat equals 1/log(2) ≈ 1.44 bits.Source: Entropy · p. 173, Surprisal · p. 173, Cross-Entropy Revisited · p. 174
  • Cross-entropy is both the expected surprisal under predicted probabilities and the objective obtained by maximizing observed-data likelihood.It measures the number of bits needed to encode observed labels relative to predictions.Source: Cross-Entropy Revisited · p. 174

4.1.4 Summary and Discussion

  • Fully connected layers with d inputs and q outputs require O(dq) parameters and computation, which can be prohibitive in practice.Approximation and compression methods can reduce this cost, including a quadratic-to-log-linear reduction, but may trade accuracy and storage against execution efficiency on modern GPUs.Source: §4.1.4 · p. 174

4.1.5 Exercises

  • The exercises extend the section by asking readers to connect softmax with exponential families, compute the cross-entropy Hessian, and analyze softmax variance.They also include an equal-probability three-class example with vector (1/3, 1/3, 1/3).Source: §4.1.5 · p. 174

1. What is the problem if we try to design a binary code for it? … 3. How do researchers typically determine the stopping criterion?

This section develops core approaches to image classification, from softmax regression and reliable evaluation to multilayer perceptrons, backpropagation, initialization, and dropout. It also examines distribution shift, generalization, numerical stability, and practical methods for training and deployment.

4.2 The Image Classification Dataset

  • Fashion-MNIST replaces MNIST as a more challenging, practical benchmark, containing 10 clothing categories at 28 × 28 pixels with 60,000 training and 10,000 test images.MNIST is now mainly a sanity check because simple models exceed 95% accuracy, whereas ImageNet is too large for the book’s interactive examples.Source: §4.2 · p. 176, §4.2.1 · p. 177

4.2.1 Loading the Dataset

  • The dataset loader represents grayscale images as c × h × w tensors, supports built-in preprocessing, and separates training data from test data for evaluation.The broader image convention also accommodates RGB and hyperspectral channels, while Fashion-MNIST uses one grayscale channel.Source: §4.2.1 · p. 177

4.2.2 Reading a Minibatch

  • Training data are read through shuffled minibatch iterators, and a sampled minibatch contains 64 images with shape [64, 1, 32, 32] and 64 integer labels.The loader is not blazingly fast, but it is sufficient because deep-network computation takes substantially longer, so training is not I/O constrained.Source: §4.2.2 · p. 178

4.2.3 Visualization

  • Visualization of images with human-readable labels provides a safeguard against data and experiment-design errors before training.The section concludes that the Fashion-MNIST data are ready for subsequent work, with visualization used to inspect initial examples.Source: §4.2.3 · pp. 178–179

4.2.4 Summary

  • Fashion-MNIST is adopted for evaluating network designs from linear models through residual networks, while high-performance data iterators help prevent preprocessing from slowing training.The summary emphasizes its 10 categories, grayscale tensor format, and the importance of efficient iterators for operations such as image decompression and preprocessing.Source: §4.2.4 · p. 179

4.3.1 The Classifier Class

  • The base Classifier class standardizes validation by reporting loss and accuracy, averaging updates across validation batches, and using minibatch stochastic gradient descent by default.The validation averages are slightly inaccurate when the final batch is smaller, a difference ignored for simplicity.Source: §4.3.1 · p. 180

4.3.2 Accuracy

  • Classification accuracy selects the highest-scoring class with argmax and measures the fraction of predictions matching the labels, making it a central empirical metric despite being nondifferentiable.Training may minimize other objectives for statistical or computational reasons, but accuracy remains the performance measure commonly reported in benchmarks.Source: §4.3.2 · p. 181, §4.3.3 · p. 181

4.4.1 The Softmax

  • Softmax converts class scores into row-normalized probabilities by exponentiating scores, summing each row, and dividing by its normalization constant.The denominator’s logarithm is the log partition function; the illustrative implementation is numerically unsafe for very large or small inputs, so built-in protected versions are preferred.Source: §4.4.1 · pp. 182–183, §4.4 · p. 182

4.4.2 The Model

  • From scratch, softmax regression flattens 28 × 28 images into 784-dimensional vectors, uses 10 outputs, and applies cross-entropy to predicted class probabilities.Weights have shape 784 × 10, biases have shape 1 × 10, and cross-entropy selects true-label probabilities before averaging their negative logarithms.Source: §4.4.2 · p. 183, §4.4.3 · p. 184

4.4.4 Training

  • Training performance depends on adjustable epochs, minibatch size, and learning rate, which should be selected using validation data before final testing.The section uses Fashion-MNIST’s test data as a validation split and reports validation loss and accuracy there.Source: §4.4.4 · p. 185

4.5.2 Softmax Revisited

  • Combining softmax with cross-entropy computes loss directly from logits, avoiding overflow and underflow through numerically stable operations such as LogSumExp.Subtracting the largest logit prevents overflow, while the combined computation avoids taking log 0 during backpropagation.Source: §4.5.2 · p. 188

4.5.3 Training

  • The concise softmax-regression implementation trains on flattened 784-dimensional Fashion-MNIST images for 10 epochs and converges to a reasonably accurate solution.The framework implementation uses a built-in fully connected layer and hides implementation details behind a shorter interface.Source: §4.5.1 · p. 187, §4.5.3 · pp. 188–189

4.5.4 Summary

  • High-level APIs improve accessibility and concision but can hide numerical hazards and discourage users from understanding or extending model components.The authors therefore recommend reviewing both bare-bones and elegant implementations, which are generally performant except for convolutions.Source: §4.5.4 · p. 189

4.6 Generalization in Classification

  • Generalization concerns arise because perfect training accuracy can result from memorization, while a model’s real objective is performance on previously unseen data.The section frames test-set size and repeated evaluation as central questions for assessing classifier accuracy on the underlying population.Source: §4.6 · p. 190

4.6.1 The Test Set

  • Test error estimates population error by averaging mistakes on fresh, representative examples, whereas population error is an unobservable expectation over the underlying distribution.The distinction motivates treating the test set as a statistical estimator rather than the population itself.Source: §4.6.1 · p. 191
  • O(1/√n) is the convergence rate for test error toward population error, so twice the precision requires four times as many test examples.Reducing error by a factor of one hundred requires ten thousand times as large a test set.Source: §4.6.1 · p. 191

4.6.1 The Test Set / 4.6.2 Test Set Reuse

  • Test-set error estimates converge at O(1/√n), but evaluating many models or reusing a holdout introduces false discovery and adaptive overfitting.For a single classifier, approximately 10,000 samples support 95% confidence within ±0.01 asymptotically, while finite-sample guarantees require roughly 15,000; test sets should therefore be consulted sparingly and multiple hypothesis testing should be considered.Source: §4.6.1 · p. 192, §4.6.2 · p. 193, §4.6.4 · p. 195

4.6.3 Statistical Learning Theory

  • Statistical learning theory seeks uniform convergence bounds linking empirical and population error to model-class complexity and sample size, but VC-dimension guarantees are often pessimistic for deep networks.Uniform convergence allows selection among models while preserving generalization guarantees; however, deep networks can have millions of parameters, fit random labels, and still generalize well despite high VC dimension.Source: §4.6.3 · pp. 194–195, §4.6.4 · p. 196

4.7 Environment and Distribution Shift

  • Distribution shift can cause models with excellent test accuracy to fail catastrophically in deployment, especially when model-based decisions alter future data.A footwear-based loan policy could cause applicants to change behavior, making the predictive relationship disappear.Source: §4.7 · p. 197

4.7.1 Types of Distribution Shift

  • Without assumptions relating training pS(x, y) and test pT(x, y), robust classification is impossible; under restricted assumptions, algorithms can detect or adapt to shift.Arbitrary label flipping can leave pS(x) = pT(x) while reversing pS(y | x) and pT(y | x), making the shift indistinguishable from no shift.Source: §4.7.1 · pp. 197–198

Covariate Shift / Label Shift

  • Covariate shift changes P(x) while preserving P(y | x), whereas label shift changes P(y) while preserving P(x | y), with the appropriate assumption depending partly on causal direction.Label-shift methods can be advantageous when labels cause inputs because they manipulate lower-dimensional label-like quantities rather than high-dimensional inputs.Source: Covariate Shift · p. 198, Label Shift · p. 199

Concept Shift

  • Concept shift occurs when label definitions or P(y | x) change across time or geography, as with changing diagnostic criteria, job meanings, soft-drink names, or translation patterns.Gradual temporal or geographic change may provide structure that can be exploited, but concept shift can be difficult to detect.Source: Concept Shift · p. 199

Medical Diagnostics

  • Medical-diagnostic data can exhibit extreme, uncorrectable covariate shift when healthy controls differ systematically from real patients, producing misleading near-perfect accuracy and wasted resources.The startup’s student blood donors differed from sick, predominantly older men in age, hormone levels, activity, diet, alcohol consumption, and other disease-irrelevant factors.Source: Medical Diagnostics · pp. 200–201

Self-Driving Cars

  • Synthetic-rendering artifacts and collection-time confounds can make autonomous-driving and tank-detection classifiers appear successful while learning irrelevant environmental features.The roadside detector learned a uniform rendered texture, while the tank classifier learned the time-of-day shadows distinguishing the photographs.Source: Self-Driving Cars · p. 201

Nonstationary Distributions

  • Distribution shift occurs when training and test distributions differ, causing systems to fail on novel spam, seasonal preferences, unseen face views, new markets, or nonuniform real-world labels.Examples include spam filters missing newly crafted messages, recommendation systems suggesting Santa hats after Christmas, face detectors failing on close-ups absent from training, and classifiers facing changed label frequencies.Source: Nonstationary Distributions · p. 201, More Anecdotes · pp. 201–202

Empirical Risk and Risk

  • Empirical risk minimizes average training loss as a practical approximation to population risk, whose expectation over the true data distribution is usually unavailable.This approximation underlies empirical risk minimization but can become unreliable when training and test distributions differ.Source: Empirical Risk and Risk · p. 202, §4.7.6 · p. 208

Covariate Shift Correction

  • Under covariate shift, unchanged P(y | x) permits correction by weighting training examples by p(x)/q(x), estimated by a classifier distinguishing target from source features.Logistic regression yields βi = exp(h(xi)); capped weights can be used, and every target example must have nonzero probability under the training distribution.Source: Covariate Shift Correction · pp. 202–204

Label Shift Correction

  • Under label shift, unchanged q(x | y) enables estimating target label frequencies from a source-validation confusion matrix and mean target predictions, then weighting examples by p(yi)/q(yi).The method requires a reasonably accurate classifier, previously observed target categories, and the label-shift assumption; when C is invertible, p(y) = C−1µ(ŷ).Source: Label Shift Correction · pp. 204–205

Concept Shift Correction

  • Concept shift is harder to correct when the task changes, but gradual changes can be handled by updating existing network weights with new data rather than retraining from scratch.Examples include evolving product popularity, degrading camera lenses, and changing news content; adaptation speed should reflect how quickly the environment changes.Source: Concept Shift Correction · p. 205

4.7.4 A Taxonomy of Learning Problems

  • Learning settings differ by data availability and environmental dynamics: batch learning uses a fixed model, online learning updates after sequential feedback, bandits choose among finite actions, and reinforcement learning addresses responsive environments.Control problems and reinforcement-learning environments may retain memory or react strategically, so algorithms must account for prior actions and adaptation.Source: Batch Learning · p. 206, Online Learning · p. 206, Reinforcement Learning · p. 207

4.7.5 Fairness, Accountability, and Transparency in Machine Learning

  • Deployed predictive systems automate decisions with technical and ethical consequences, requiring attention to affected populations, asymmetric error costs, fairness, transparency, and feedback loops.Predictive policing illustrates a runaway loop in which intensified patrols generate more recorded crime, reinforcing future predictions.Source: §4.7.5 · pp. 207–208

4.7.6 Summary

  • The section concludes that covariate and label shift can be corrected under their assumptions, while live systems require monitoring because automated actions can entangle models with changing environments.The next chapter moves from softmax regression and linear classifiers to multilayer perceptrons, gradient calculation, numerical stability, initialization, regularization, generalization, and house-price prediction.Source: §4.7.6 · p. 208, §5.1 · p. 210, §4.7.7 · p. 210

Limitations of Linear Models

  • Nonlinear interactions make linear models inadequate for tasks such as image classification, so deep neural networks jointly learn representations and predictors through hidden layers.Pixel importance depends on surrounding context, and deep networks learn these feature interactions from observational data rather than requiring them to be handcrafted.Source: Limitations of Linear Models · p. 211

From Linear to Nonlinear

Universal Approximators

  • A single-hidden-layer network can represent any function with enough nodes and suitable weights, although deeper networks can represent many functions more compactly.The text emphasizes that representational capacity does not make learning the function easy, and kernel methods may solve some problems more effectively.Source: Universal Approximators · p. 214

5.1.2 Activation Functions

  • ReLU is favored because its derivatives either vanish or pass the input through, improving optimization and mitigating vanishing gradients.Sigmoid remains useful for probability-like binary-classification outputs but is harder to optimize because its gradients vanish for large positive and negative inputs; tanh likewise has derivatives approaching zero away from zero.Source: ReLU Function · p. 215, Sigmoid Function · p. 216, Tanh Function · p. 217

5.2.1 Implementation from Scratch

  • The implementation section builds a one-hidden-layer MLP for Fashion-MNIST with 784 input features, 256 hidden units, and 10 output classes.Images are flattened into vectors, weights and biases are stored as trainable tensors, and the forward pass applies ReLU before the output layer.Source: Initializing Model Parameters · p. 220, Model · p. 220

Training

  • Training an MLP uses the same training loop as softmax regression: define the model and data, create the trainer, and invoke fit.The implementation relies on framework autograd while exposing the layer computations directly.Source: Model · p. 220, Training · p. 221

Model and Training

  • Deep networks can reuse training algorithms and data loaders, while framework abstractions simplify model construction and avoid the parameter-management and optimization difficulties of from-scratch implementations.Sequential layers abstract the forward process, and the same training loop used for softmax regression applies to MLPs.Source: Model · p. 222, Training · p. 222

5.3 Forward Propagation and Computational Graphs

  • Forward propagation computes and stores intermediate variables from input to output, while computational graphs visualize the dependencies among these variables and operators.The illustrated one-hidden-layer MLP includes hidden activations, output variables, loss, and ℓ2 regularization in its objective function.Source: §5.3.1 · pp. 224–225, §5.3.2 · p. 225, Figure 5.3.1 · p. 225

5.3.3 Backpropagation

  • Backpropagation reverses the computational graph and applies the chain rule to calculate gradients for intermediate variables and model parameters.For the one-hidden-layer network, it computes gradients with respect to both W(1) and W(2), proceeding from the objective toward the input layer.Source: §5.3.3 · pp. 225–226

5.3.4 Training Neural Networks

  • Training alternates forward propagation with backpropagation, reusing stored intermediates for gradient computation but requiring substantially more memory than prediction.Memory use grows roughly with network depth and batch size, making deeper networks and larger batches more prone to out-of-memory errors.Source: §5.3.4 · pp. 226–227, §5.3.5 · p. 227

5.4 Numerical Stability and Initialization

  • Poor initialization and activation choices can make gradients numerically unstable, producing updates that are either excessively large or too small for effective learning.Products of layerwise matrices may become very large or very small, causing exploding or vanishing gradients and undermining optimization.Source: §5.4 · p. 228, §5.4.1 · pp. 228–229

Vanishing and Exploding Gradients

  • Sigmoid gradients vanish for large-magnitude inputs and can be cut off across many layers, whereas ReLUs have become the default because they are more stable.Exploding gradients can likewise prevent convergence; multiplying 100 Gaussian matrices at variance σ^2 = 1 produced a matrix with entries on the order of 10^25.Source: Vanishing Gradients · p. 229, Exploding Gradients · p. 230

Xavier Initialization

  • Random initialization breaks hidden-unit symmetry, while Xavier initialization balances forward-activation and backward-gradient variances across layer widths.Identical hidden-unit parameters receive identical gradients and remain indistinguishable; Xavier initialization uses variance σ2 = 2 nin+nout and works well in practice despite simplified assumptions.Source: Breaking the Symmetry · p. 230, Xavier Initialization · p. 232

5.5 Generalization in Deep Learning

  • Deep-learning theory still lacks a comprehensive account of optimization and generalization, although practitioners have accumulated heuristics that often work across diverse prediction problems.The practical difficulty is greater for generalization because fitting training data is usually straightforward.Source: §5.5 · p. 234

5.5.1 Revisiting Overfitting and Regularization

  • Deep networks can fit training data perfectly, yet greater expressivity can sometimes improve generalization and produce non-monotonic double-descent behavior.Architectures may all achieve zero training error, while adding layers, nodes, or epochs can further reduce generalization error; classical complexity-based bounds do not explain this behavior.Source: §5.5.1 · p. 235

5.5.2 Inspiration from Nonparametrics

  • Deep neural networks often behave like nonparametric interpolators: 1-nearest neighbors can have zero training error yet be consistent, and infinitely wide randomly initialized networks converge to neural tangent kernel methods.Distance metrics and kernels encode inductive biases, so generalization depends on compatibility between these assumptions and the data.Source: §5.5.2 · p. 236

5.5.3 Early Stopping

  • In noisy-label settings, networks fit clean examples before mislabeled ones, so fitting clean data without interpolating random labels guarantees generalization and motivates early stopping.Researchers typically monitor validation error after each epoch and stop when it fails to decrease by more than ϵ for a specified patience interval.Source: §5.5.3 · pp. 236–237

3. How do researchers typically determine the stopping criterion?

  • Early stopping is crucial with label noise or intrinsic label variability but usually offers little generalization improvement for noiseless, realizable datasets.Training until noisy data are interpolated is typically harmful, whereas stopping can also save substantial training time for large models.Source: §5.5.3 · p. 237

5.5.4 Classical Regularization Methods for Deep Networks

  • Classical weight decay often cannot prevent deep networks from interpolating data, so its apparent regularization benefit may instead reflect inductive bias or interaction with early stopping.Why particular choices improve generalization remains largely unresolved, even though classical regularizers remain widely used.Source: §5.5.4 · pp. 237–238

5.6 Dropout

  • Dropout regularizes by randomly zeroing internal activations during training and rescaling retained activations so their expectation remains unchanged.The technique was developed by Srivastava et al. (2014) and is widely implemented, although its co-adaptation explanation remains debatable.Source: §5.6 · p. 239

5.6.1 Dropout in Practice

  • Dropout removes hidden units during training so outputs do not depend excessively on any single activation, while it is typically disabled during testing.At test time, some researchers instead retain dropout to estimate predictive uncertainty from agreement across stochastic outputs.Source: §5.6.1 · p. 240

5.6.2 Implementation from Scratch

  • From scratch, dropout samples Bernoulli keep/drop masks with drop probability p and rescales surviving activations by 1/(1-p).The implementation handles p=1 by returning zeros and tests probabilities 0, 0.5, and 1.Source: §5.6.2 · p. 240

Defining the Model

  • The scratch model applies separately configurable dropout probabilities after the activation of each hidden layer and only when the model is training.A lower dropout probability is commonly used nearer the input layer.Source: Defining the Model · p. 241

Training

  • Training follows the previously described MLP procedure using a DropoutMLPScratch model, Fashion-MNIST data, and example dropout probabilities of 0.5 for both hidden layers.The supplied setup uses batch size 256 and learning rate 0.1.Source: Training · p. 241

5.6.3 Concise Implementation

  • With high-level APIs, Dropout layers are inserted after fully connected layers, randomly dropping outputs during training and passing data through unchanged during testing.The concise model places dropout after each ReLU-activated hidden layer.Source: §5.6.3 · p. 242

5.6.4 Summary

  • Dropout is another overfitting-control tool used during training, replacing an activation h with a random variable whose expected value is h.It can be combined with methods that control dimensionality and weight-vector size.Source: §5.6.4 · p. 242

5.6.5 Exercises

  • The exercises investigate layer-specific dropout probabilities, longer training with and without dropout, and how dropout changes hidden-layer activation variance over time.They ask learners to report quantitative results and summarize qualitative takeaways where applicable.Source: §5.6.5 · p. 243

4. Why is dropout not typically used at test time? … 4. What is the computational cost for the backpropagation?

This section develops practical and reusable neural-network workflows, covering preprocessing, model selection, modular composition, parameter management, custom layers, and deployment-oriented saving. It then focuses on efficient computation and convolutional neural networks, including device use, spatial inductive biases, learned filters, multi-channel and 1 × 1 convolutions, computational costs, and pooling.

5.7 Predicting House Prices on Kaggle

  • A Kaggle house-price workflow combines data preprocessing, model design, hyperparameter selection, and prediction submission on Ames, Iowa housing data from 2006–2010.The competition data include 1,460 training examples, 80 features, and 1,459 validation examples; Kaggle enables quantitative comparisons and code sharing.Source: §5.7 · p. 244, §5.7.2 · pp. 244–245, §5.7.3 · p. 246

5.7.4 Data Preprocessing

  • Preprocessing removes identifiers, imputes missing numerical values, standardizes numerical features, and one-hot encodes categorical features, increasing the feature count from 79 to 331.Standardization supports optimization and avoids penalizing coefficients unevenly across features, while one-hot encoding represents discrete categories as indicators.Source: §5.7.4 · p. 247

5.7.5 Error Measure

  • The competition evaluates relative price accuracy with root-mean-squared error on logarithmic predictions and labels rather than absolute price error.Logarithmic discrepancies make errors comparable across inexpensive and expensive houses, and small log error bounds translate into multiplicative bounds on predicted prices.Source: §5.7.5 · p. 248

5.7.6 K-Fold Cross-Validation

  • K-fold cross-validation trains models on complementary folds, returns average validation error, and supports selecting model designs and hyperparameters.The presented implementation uses an untuned hyperparameter set, while the text cautions that excessive option testing can make validation performance unrepresentative.Source: §5.7.6 · p. 249, §5.7.7 · p. 249

5.7.7 Model Selection

  • A large gap between low training error and substantially higher cross-validation error indicates overfitting and may motivate stronger regularization.Monitoring both training and validation behavior helps assess whether the data support a more powerful model.Source: §5.7.7 · p. 250

5.7.8 Submitting Predictions on Kaggle

  • The workflow ensembles the K models’ exponentiated log-price predictions, saves them as a CSV file, and submits the resulting predictions to Kaggle for test-set comparison.The competition separates training and test data, provides labels only for training, and requires uploaded predictions for official evaluation.Source: §5.7.3 · p. 246, §5.7.8 · p. 250

5.7.9 Summary and Discussion

  • The chapter’s practical summary recommends zero-mean/unit-variance scaling, mean imputation, one-hot encoding, logarithmic error measurement, and K-fold cross-validation for real tabular data.These techniques address mixed data types, missing values, relative-error objectives, and model or hyperparameter selection.Source: §5.7.9 · p. 251

6.1 Layers and Modules

  • Neural-network modules provide a recursive abstraction for representing single layers, multi-layer components, or entire models, enabling compact implementations of complex architectures.The abstraction addresses repeated layer patterns such as those in ResNet and supports composing modules into larger artifacts.Source: §6.1 · p. 253

6.1.1 A Custom Module

  • Custom modules subclass nn.Module and typically implement only initialization and forward propagation, while the system supplies parameter initialization and backpropagation.The example MLP uses 256 hidden units, a ReLU activation, and a 10-dimensional output layer.Source: §6.1.1 · p. 255

6.1.2 The Sequential Module

  • Sequential modules append constituent modules and pass inputs through them in the order added, reproducing the default Sequential functionality.The custom MySequential implementation produces the same functionality as the default class and can reimplement the earlier MLP.Source: §6.1.2 · p. 256

6.1.3 Executing Code in the Forward Propagation Method

  • Defining a custom forward method enables Python control flow, arbitrary mathematical operations, constant weights, parameter sharing, and nested module composition.The FixedHiddenMLP keeps a randomly initialized weight constant, reuses a fully connected layer, and divides its output until its ℓ1 norm is at most 1.Source: §6.1.3 · pp. 257–258

6.1 Layers and Modules

  • Modules can represent layers, complete models, or intermediate components, with framework support for parameter initialization and backpropagation.A module defines forward propagation and stores required parameters; automatic differentiation generally supplies backpropagation.Source: §6.1 · p. 254, §6.1.4 · p. 258

6.1.4 Summary

  • The module abstraction supports arbitrary nesting and organization, from individual layers to larger modules and complete architectures.Sequential concatenations handle chained layers and modules, while more flexible architectures can define their own blocks.Source: §6.1.3 · p. 256, §6.1.4 · p. 258

6.2 Parameter Management

  • Parameter management supports accessing learned values for debugging, diagnostics, visualization, reuse, saving, and examination, especially when architectures depart from standard stacked layers.The framework normally handles parameter declaration and manipulation, but custom architectures may require explicit management.Source: §6.2 · p. 259

6.2.1 Parameter Access

  • Parameters can be accessed through indexed Sequential layers or recursively across nested models using named-parameter listings, which expose parameter names and shapes.An example lists weights and biases with shapes [8, 4], [8], [1, 8], and [1].Source: §6.2.1 · pp. 259–260, All Parameters at Once · pp. 260–261

6.2.2 Tied Parameters

  • Tied parameters are the same tensor rather than merely equal values, so changing one changes the other and their gradients are added during backpropagation.The example confirms identical values before and after modifying one shared parameter.Source: §6.2.2 · p. 261

6.3 Parameter Initialization

  • PyTorch supports built-in, constant, block-specific, custom, and direct parameter initialization methods.Examples include Gaussian weights with zero biases, constant values, Xavier initialization for one layer and 42 for another, and custom thresholded uniform weights.Source: §6.3.1 · pp. 262–263, Custom Initialization · pp. 263–264

6.4 Lazy Initialization

  • Lazy initialization infers parameter shapes when data first passes through the model, simplifying architecture changes and eliminating a common source of errors.The framework initializes layers sequentially after learning the input dimensionality, and dummy inputs can trigger a dry run before applying custom initialization.Source: §6.4 · pp. 265–266

6.5 Custom Layers

  • Custom layers can omit parameters or define trainable weights and biases, then operate within arbitrary model architectures like built-in layers.The examples implement a mean-centering layer and a parameterized fully connected layer with a default ReLU activation; built-in parameter utilities handle access, initialization, sharing, saving, and loading.Source: §6.5.1 · p. 267, §6.5.2 · pp. 268–269, §6.5.3 · p. 269

6.6 File I/O

  • PyTorch can save and load individual tensors, tensor collections, dictionaries, and model parameter state dictionaries.Restoring an entire network requires recreating its architecture in code because serialization stores parameters rather than arbitrary model code.Source: §6.6.1 · p. 270, §6.6.2 · p. 271

6.7.2 Tensors and GPUs

  • GPU computation requires tensors involved in an operation to share a device, and careless transfers can substantially reduce performance or cause failures.Frameworks therefore require explicit movement between devices rather than silently copying incompatible tensors.Source: §6.7.2 · p. 274, Copying · p. 275, Side Notes · p. 276

Side Notes

  • Data transfers are slower than computation and complicate parallelization, so many small or interleaved copies should be avoided in favor of larger, coordinated operations.Printing tensors or converting them to NumPy may add further transfer overhead and trigger Python’s global interpreter lock.Source: Side Notes · p. 276

6.7.3 Neural Networks and GPUs

  • Models compute efficiently when their parameters and input data reside on the same GPU, and the training framework can be configured to support available GPUs.GPU selection utilities can fall back to the CPU when requested devices are unavailable.Source: §6.7.3 · pp. 276–277

7.1.1 Invariance

  • CNNs exploit spatial structure and translation invariance, offering powerful computer-vision models with fewer parameters and significant performance improvements on ImageNet (LeCun et al., 1995; Deng et al., 2009; Krizhevsky et al., 2012).Convolutions also parallelize efficiently across GPU cores, supporting their broad use in vision and other structured data domains.Source: §2 · p. 279, §7.1.1 · p. 281

7.1 From Fully Connected Layers to Convolutions

  • Fully connected networks become impractical for high-dimensional images: a one-megapixel input with 1,000 hidden units requires 10^9 parameters, motivating CNNs’ structural assumptions.CNNs exploit local image structure instead of treating pixels as unrelated features, reducing the burden of learning massive parameter sets.Source: §2 · p. 279, §7.1 · p. 280

7.1.1 Invariance

  • CNN architectures are designed around translation invariance, locality, and progressively broader receptive fields, aggregating local representations into whole-image predictions.Early layers detect similar patches regardless of location, while deeper layers capture longer-range visual features.Source: §7.1.1 · p. 281

Translation Invariance

  • Translation invariance and locality reduce an image layer from billions of parameters to typically a few hundred by sharing weights and restricting computation to nearby pixels.Translation invariance makes all patches use the same coefficients, while locality limits the kernel to offsets within ∆, reducing parameters from 4 × 10^6 to 4∆^2 when ∆ is typically smaller than 10.Source: Translation Invariance · p. 282, Locality · p. 283

7.1.4 Channels

  • Multiple input and output channels extend convolutions from two-dimensional images to third-order hidden representations, with kernels indexed by spatial offsets, input channels, and output channels.Channels can represent learned feature maps, such as edge- or texture-sensitive patterns, while also restoring complexity restricted by locality and translation invariance.Source: §7.1.4 · pp. 284–285, §7.1.5 · p. 285

7.1.5 Summary and Discussion

  • The section frames convolutional networks as applying the same local computation across image patches, while noting that efficient multi-layer design, dimensionality changes, activations, and output aggregation remain further considerations.The exercises extend these ideas to audio, boundaries, and other data types, but the supplied text does not provide their solutions.Source: §7.1.5 · p. 285, §7.1.4 · p. 285, §1 · p. 286
  • Convolutional inductive biases make previously infeasible image-processing problems tractable, but they can hurt when translation invariance or locality disagrees with the data.The resulting models are sample-efficient and generalize well when their imposed biases match reality, while deeper networks combine repeated convolutions with nonlinearities to represent larger image structures.Source: Locality · p. 283, §7.1.5 · p. 285

7.2.1 The Cross-Correlation Operation

  • Convolutional layers generalize cross-correlation by sliding a kernel across an input, multiplying each window elementwise, summing the products, and adding a scalar bias.The output is smaller than the input when the kernel must fit wholly within the image; padding can preserve spatial size.Source: §7.2.1 · p. 287, §7.2.2 · p. 288

7.2.3 Object Edge Detection in Images

  • A finite-difference kernel detects vertical edges by producing 1 for white-to-black transitions, −1 for black-to-white transitions, and 0 elsewhere.Applying the same kernel to the transposed image makes it vanish because the kernel detects only vertical edges.Source: §7.2.3 · p. 289

7.2.4 Learning a Kernel

  • Training a convolutional kernel from input–output pairs reduces the error to a small value after 10 iterations and learns weights [0.9797, -0.9816], close to the manually specified kernel.The procedure initializes the kernel randomly, minimizes squared error, and updates the weights using gradients.Source: §7.2.4 · pp. 290–291

7.2.5 Cross-Correlation and Convolution

  • Strict convolution differs from cross-correlation only by horizontally and vertically flipping the kernel, and learned kernels make the layer outputs unaffected by which operation is used.Because kernels are learned from data, the model can absorb this orientation difference during training.Source: §7.2.5 · p. 291
  • Strict convolution produces the same output as cross-correlation when its learned kernel is flipped horizontally and vertically, so deep learning commonly calls cross-correlation convolution.The terminology is conventional even though the operations differ strictly.Source: §7.2.5 · p. 291

7.2.6 Feature Map and Receptive Field

  • Learned convolutional filters replace hand-designed feature engineering, while receptive fields expand through deeper layers to detect features over broader image regions.These filters correspond to receptive fields and feature maps observed in the brain, linking CNN representations to biological vision.Source: §7.2.6 · pp. 292–293

7.3.1 Padding

  • Padding prevents repeated convolutions from shrinking feature maps and losing boundary information, while symmetric padding can preserve the input height and width.For an input shape nh × nw, kernel kh × kw, and padding totals ph and pw, the output shape is (nh − kh + ph + 1) × (nw − kw + pw + 1).Source: §7.3 · p. 294, §7.3.1 · p. 295, §7.3.3 · p. 297

7.3.2 Stride

  • Stride moves the convolution window by multiple rows or columns, providing computational efficiency or downsampling; with divisible input dimensions, strides sh and sw yield output shape (nh/sh) × (nw/sw).Setting both strides to 2 halves the input height and width.Source: §7.3.2 · pp. 296–297

7.4.1 Multiple Input Channels

  • Multiple input-channel convolution applies a separate two-dimensional cross-correlation per channel and sums the results, producing a two-dimensional output.With ci input channels and kernel window kh × kw, the kernel has shape ci × kh × kw; the worked example produces tensor([[56., 72.], [104., 120.]]).Source: §7.4.1 · pp. 299–300

7.4.2 Multiple Output Channels

  • CNNs typically increase channel depth while downsampling spatial resolution, with channels jointly optimized to represent feature directions rather than independent per-pixel or per-channel detectors.Each channel can be viewed intuitively as responding to a different feature set, although useful features may correspond to directions in channel space.Source: §7.4.2 · p. 300
  • A convolution with co output channels uses a co × ci × kh × kw kernel, with each output channel combining all input channels.The implementation computes one cross-correlation per output-channel kernel and stacks the results.Source: §7.4.2 · p. 300

7.4.3 1 × 1 Convolutional Layer

  • A 1 × 1 convolution performs a location-wise fully connected transformation across channels, using co×ci weights while preserving height and width.It cannot model adjacent spatial interactions, and nonlinearities prevent simply folding it into other convolutions.Source: §7.4.3 · p. 301
  • 1 × 1 convolution implemented through matrix multiplication is equivalent to the corresponding multi-channel cross-correlation function on sample data.The reported check verifies that the summed absolute difference is less than 1e-6.Source: §7.4.3 · p. 302

computer vision.

  • Convolutional cost scales as O(h · w · k2 · ci · co), reaching over 53 billion operations for a 256 × 256 image with a 5 × 5 kernel and 128 input and output channels.Block-diagonal channel-wise operations, as in ResNeXt (Xie et al., 2017), are cited as a strategy for reducing this cost.Source: computer vision · pp. 302–303

7.5 Pooling

  • Pooling reduces sensitivity to feature location while spatially downsampling representations, supporting progressively larger receptive fields and global image representations.Pooling also accelerates receptive-field growth by reducing spatial resolution.Source: §7.5 · p. 304

7.5.1 Maximum Pooling and Average Pooling

  • Pooling slides a fixed window without parameters and computes either a maximum or average, with max-pooling generally preferred for translation-related invariance.A 2 × 2 max-pooling window can detect a recognized pattern moving by no more than one element vertically or horizontally.Source: §7.5.1 · pp. 304–305

7.5.2 Padding and Stride

  • Pooling supports configurable padding, stride, and rectangular windows; frameworks commonly default the stride to match the pooling-window size.These settings change the output shape while pooling remains parameter-free.Source: §7.5.2 · pp. 306–307

7.5.3 Multiple Channels

  • Pooling processes each input channel independently, so the output has the same number of channels as the input; a common 2 × 2 window quarters spatial resolution.The summary also identifies max-pooling as preferable to average pooling because it provides some output invariance.Source: §7.5.3 · pp. 307–308, §7.5.4 · p. 308

5. Why do you expect max-pooling and average pooling to work differently? … 3. Does causality also apply to text? To which extent?

This section traces the development of convolutional and sequence models, from LeNet and modern CNN architectures through normalization, residual and feature-reuse designs, and efficient model families. It then introduces sequence prediction and language modeling, covering tokenization, probabilistic formulations, evaluation, and the scaling of neural models.

7.6.2 Training

  • LeNet retains spatial structure and uses far fewer parameters than fully connected image models, although its repeated parameter use can make computation more expensive than similarly deep MLPs.Training uses cross-entropy minimized by minibatch stochastic gradient descent, with GPU acceleration available through the trainer.Source: §7.6 · p. 309, §7.6.2 · p. 312

7.6 Convolutional Neural Networks (LeNet)

  • LeNet achieved less than 1% error per digit, matching support vector machines and eventually supporting handwritten-digit recognition in ATM machines.Source: §7.6 · p. 309

7.6.1 LeNet

  • LeNet uses two convolutional layers with sigmoid activation and 2 × 2 stride-2 pooling, followed by three fully connected layers producing 120, 84, and 10 outputs.Each pooling operation reduces dimensionality by a factor of 4 via spatial downsampling; the 10-dimensional output layer corresponds to the possible output classes.Source: §7.6.1 · p. 310
  • The reproduced LeNet preserves the original architecture except that its rarely used Gaussian activation layer is replaced by a softmax layer for simpler implementation.Source: §7.6.1 · p. 311

7.6.4 Exercises

  • Modern CNNs evolved from LeNet toward architectures that became feature generators for vision tasks, with performance shaped by architecture, computation, training techniques, and hyperparameter choices.The chapter’s tour includes AlexNet, VGG, NiN, GoogLeNet, ResNet, ResNeXt, DenseNet, ShiftNet, MobileNet, and RegNet, while noting that Transformers have begun displacing CNNs.Source: §7.6.4 · pp. 315–316

Missing Ingredient: Data

  • The delayed success of deep CNNs reflected missing computational, data, and optimization prerequisites: 1990s accelerators and datasets were limited, while effective initialization, optimizers, nonsaturating activations, and regularization were not yet available.AlexNet benefited from substantially more data and faster GPUs than LeNet had in 1995.Source: §8.1 · p. 316, §8.1.1 · p. 318, Missing Ingredient: Data · p. 319

8.1.1 Representation Learning

  • Before modern CNNs, computer-vision systems typically relied on manually engineered features such as SIFT, SURF, HOG, and bags of visual words.Modern CNNs instead jointly learn multiple layers of features, including edges, colors, textures, larger structures, and compact category-separating representations.Source: §8.1.1 · pp. 317–318
  • AlexNet demonstrated the effectiveness of modern CNNs by learning hierarchical visual features and achieving excellent performance in the 2012 ImageNet challenge.Its lower layers learned filters resembling traditional descriptors, while higher layers represented increasingly complex structures and objects.Source: §8.1.1 · p. 318

8.1.2 AlexNet

Architecture

  • AlexNet used eight layers, larger early convolution windows, more channels, ReLU activations, and max-pooling to process higher-resolution ImageNet images.Its architecture comprised five convolutional layers, two fully connected hidden layers, and one fully connected output layer.Source: §8.1.2 · p. 321, Architecture · pp. 321–322

Activation Functions

  • ReLU simplified computation and improved optimization because its positive-region gradient remains 1, avoiding sigmoid saturation that can halt backpropagation.This made training less sensitive to parameter initialization than sigmoid-based networks.Source: Activation Functions · p. 322

Capacity Control and Preprocessing

8.1.3 Training

  • Training AlexNet on Fashion-MNIST required upsampling 28×28 images to 224×224, a smaller learning rate, and slower training because the network was deeper, wider, and computationally costlier than LeNet.The resizing increased computational complexity without adding information and was used to preserve the AlexNet architecture.Source: §8.1.3 · pp. 323–324

8.1.4 Discussion

  • AlexNet’s final fully connected layers required 164 MB and 81 MFLOPs, limiting efficiency, although dropout helped prevent overfitting and modern tooling reduced implementation effort.Its last two layers contained more than 40 million parameters, yet training and validation losses remained virtually identical.Source: §8.1.4 · p. 324

8.2 Networks Using Blocks (VGG)

  • VGG addressed the rapid loss of spatial resolution by grouping multiple convolutions between max-pooling operations, establishing a reusable block-based template for deeper networks.This design shifted architecture development from individually designed layers toward repeated patterns of layers and blocks.Source: §8.2 · p. 326, §8.2.1 · p. 326

8.2.1 VGG Blocks

  • VGG blocks use padded 3×3 convolutions followed by 2×2 max-pooling, and analyses found deep, narrow networks significantly outperform shallow counterparts.The original VGG-11 used five convolutional blocks, eight convolutional layers, and three fully connected layers.Source: §8.2.1 · pp. 326–327, §8.2.2 · p. 328

8.2.3 Training

  • VGG training on FashionMNIST used a smaller-channel network because VGG-11 is computationally demanding, with closely matching training and validation loss indicating little overfitting.The demonstrated setup trained for 10 epochs on resized 224×224 FashionMNIST images.Source: §8.2.3 · p. 329

8.2.4 Summary

  • VGG introduced repeated convolutional blocks, deep and narrow networks, and model families that trade computational speed against accuracy.Its architecture repeatedly halves spatial dimensions before flattening, while frameworks enable assembling variants directly in Python.Source: §8.2.2 · p. 329, §8.2.4 · p. 330

8.3 Network in Network (NiN)

  • NiN addresses VGG’s costly fully connected layers by applying 1×1 convolutions for local channel-wise nonlinearities and global average pooling across locations.The fully connected portion of VGG-11 alone can occupy almost 400MB in FP32, while earlier fully connected layers would destroy spatial structure.Source: §8.3 · p. 331

8.3.1 NiN Blocks

  • NiN implements pixel-wise fully connected transformations with 1×1 convolutions, replacing VGG’s 3×3 block convolutions and giant final fully connected layer.Each NiN block uses an initial convolution followed by two 1×1 convolutions and ReLU activations.Source: §8.3.1 · p. 331

8.3.2 NiN Model

  • NiN uses max-pooling between blocks, then maps channels to the label classes and applies global average pooling to produce logits without fully connected layers.This significantly reduces parameters, although it may increase training time; the shown network outputs a 10-dimensional vector after adaptive average pooling.Source: §8.3.2 · pp. 332–333

8.3.4 Summary

  • NiN’s global averaging did not harm accuracy and adds translation invariance when applied to low-resolution, many-channel representations.Replacing wide-kernel convolutions with 1×1 convolutions further reduces parameters while supplying channel-wise nonlinearity, influencing later CNN designs.Source: §8.3.4 · pp. 333–334

8.4 Multi-Branch Networks (GoogLeNet)

  • GoogLeNet’s key contribution was concatenating parallel convolutions with 1×1, 3×3, and 5×5 kernels instead of selecting one kernel size.Inception branches also include max-pooling and 1×1 channel reduction, preserve spatial dimensions, and concatenate outputs along channels.Source: §8.4 · p. 335, §8.4.1 · pp. 335–336

8.4.2 GoogLeNet Model

  • GoogLeNet stacks nine Inception blocks in three groups separated by max-pooling, then uses global average pooling in its head for classification.The Inception design recognizes patterns at different spatial extents while allocating different parameter amounts across branches.Source: §8.4.2 · pp. 336, 338, §8.4.1 · p. 336

8.4.3 Training

  • The GoogLeNet example reduces Fashion-MNIST inputs from 224 × 224 to 96 × 96 for reasonable training time, producing a final 10-class linear output after sequential modules.The example trains for 10 epochs with batch size 128, while the LeNet example places batch normalization after linear or convolutional layers and before activations.Source: §8.4.3 · p. 339, §8.4.2 · p. 339, §8.5.4 · p. 346

8.4.4 Discussion

  • GoogLeNet is cheaper to compute than its predecessors while improving accuracy, marking deliberate network design that trades evaluation cost against errors.Its block-level hyperparameter experimentation was still manual, and later exercises propose batch normalization, Inception-block changes, label smoothing, and residual connections.Source: §8.4.4 · pp. 339–340, §8.4.5 · p. 340

8.5 Batch Normalization

  • Batch normalization consistently accelerates deep-network convergence, enables routine training of networks over 100 layers with residual blocks, and provides incidental regularization.It is motivated by controlling drifting intermediate-variable magnitudes, while also conveying preprocessing and numerical-stability benefits.Source: §8.5 · p. 341, §8.5.1 · pp. 341–342

8.5.1 Training Deep Networks

  • Batch normalization subtracts minibatch means, divides by minibatch standard deviations, then learns elementwise scale and shift parameters; training uses minibatch statistics while prediction uses dataset statistics.A small ε prevents division by zero, and moving averages provide stable prediction-time estimates.Source: §8.5.1 · pp. 342–343, Batch Normalization During Prediction · p. 345
  • Moderate minibatches around 50–100 provide useful batch-normalization noise, whereas larger batches regularize less and tiny batches can destroy signal through high variance.Batch normalization therefore requires suitable batch-size calibration; minibatches of size 1 are ineffective for typical fully connected settings.Source: §8.5.1 · pp. 342–343

Convolutional Layers

  • For convolutional layers, batch normalization computes statistics across batch examples and spatial locations separately for each channel, unlike fully connected normalization over the feature dimension.Convolutional batch normalization remains well defined with minibatches of size 1 because spatial locations supply multiple values.Source: §8.5.2 · p. 343, Convolutional Layers · p. 344, Layer Normalization · p. 344

Layer Normalization

  • Layer normalization standardizes one observation independently of minibatch size and training or test mode, with outputs approximately invariant to input rescaling and thus resistant to divergence.Its deterministic transformation differs from batch normalization’s dependence on minibatch statistics.Source: Layer Normalization · p. 344

8.5.3 Implementation from Scratch

  • The from-scratch implementation separates normalization mathematics from layer bookkeeping, maintains learned scale and shift parameters, and tracks moving means and variances for prediction.The implementation computes feature statistics for fully connected layers and channel statistics over dimensions (0, 2, 3) for two-dimensional convolutions.Source: §8.5.3 · pp. 345–346

8.5.4 LeNet with Batch Normalization

  • The book implements both custom and high-level batch-normalization LeNet variants with the same Fashion-MNIST training setup, while the high-level implementation runs faster after compilation.The concise implementation avoids manually supplying dimension arguments required by the custom BatchNorm class.Source: §8.5.4 · p. 347, §8.5.5 · p. 348

8.5.6 Discussion

  • Batch normalization behaves differently in training and prediction, uses minibatch statistics to stabilize intermediate outputs, and differs between fully connected and convolutional layers.Layer normalization can sometimes replace batch normalization for convolutional layers; removing batch normalization may improve robustness to input perturbations (Wang et al., 2022).Source: §8.5.6 · pp. 349–350
  • Batch normalization is useful for regularization and improving optimization convergence, whereas internal covariate shift is not a valid explanation for its success.The text distinguishes guiding intuitions from established scientific explanations and notes that batch normalization is widely used in deployed image classifiers.Source: §8.5.6 · pp. 349–350

8.6.1 Function Classes

  • For non-nested function classes, increasing model size does not guarantee approaching the truth function; nested classes provide the needed expressive-power guarantee.The accompanying figure contrasts these two cases.Source: §8.6.1 · p. 351, Figure 8.6.1 · p. 352
  • Residual networks make deeper models easier to optimize by allowing added layers to represent identity mappings through shortcut connections and residual mappings.If the desired mapping is f(x) = x, the residual mapping g(x) = f(x) − x becomes zero, which is easier to learn.Source: §8.6.1 · pp. 351–352, §8.6.2 · p. 352

8.6.2 Residual Blocks

  • ResNet residual blocks use two 3 × 3 convolutions with batch normalization and ReLU, adding the input directly before the final activation.A 1 × 1 convolution adjusts channels or spatial resolution when the residual output and input shapes differ.Source: §8.6.2 · pp. 352–354

8.6.3 ResNet Model

  • ResNet-18 combines four residual-block modules with an initial 7 × 7 convolution and max pooling, progressive channel increases and downsampling, global average pooling, and a final linear layer.The reported shape sequence is 64 × 24 × 24, 64 × 24 × 24, 128 × 12 × 12, 256 × 6 × 6, 512 × 3 × 3, then 10 outputs.Source: §8.6.3 · pp. 355–356

8.6.4 Training

  • Training ResNet on Fashion-MNIST produces a substantial training–validation loss gap, indicating that additional training data could improve accuracy.The text characterizes ResNet as a powerful and flexible architecture whose training loss is considerably lower than its validation loss.Source: §8.6.4 · p. 357

8.6.5 ResNeXt

  • ResNeXt uses identical transformations across independent groups, reducing convolutional computation and parameters by a factor of g while preserving residual-block structure.Grouped convolution changes cost from O(c_i · c_o) to O(c_i · c_o/g), with 1 × 1 convolutions connecting the groups through a bottleneck block.Source: §8.6.5 · pp. 357–358

8.6.6 Summary and Discussion

  • Residual connections ease identity mappings and forward propagation, enabling substantially deeper networks; ResNet demonstrated networks of up to 152 layers.They also permit layers initialized as identity functions to be added during training, potentially accelerating training of very large networks.Source: §8.6.6 · pp. 359–360

8.7.1 From ResNet to DenseNet

  • DenseNet extends residual ideas by concatenating each layer’s input and output, preserving and reusing progressively transformed features rather than adding them.Dense blocks concatenate along channels, while transition layers control the resulting dimensionality; this design follows DenseNet (Huang et al., 2017).Source: §8.7 · p. 361, §8.7.1 · pp. 361–362

8.7.2 Dense Blocks

  • A dense block concatenates every convolution block’s input and output, so two 10-channel blocks expand a three-channel input to 23 channels.The number of output channels per block is the growth rate controlling channel expansion.Source: §8.7.2 · pp. 362–363

8.7.3 Transition Layers

  • Transition layers limit DenseNet’s channel growth with a 1 × 1 convolution and halve spatial dimensions using average pooling with stride 2.Applying a transition layer can reduce the output to 10 channels while halving height and width.Source: §8.7.3 · p. 363

8.7.4 DenseNet Model

  • The illustrated DenseNet combines an initial ResNet-like convolution and max-pooling stage with four dense blocks, intervening transitions, global pooling, and a fully connected classifier.Each dense block uses four convolutional layers with growth rate 32, adding 128 channels before transition-layer reduction.Source: §8.7.4 · p. 364

8.7.5 Training

  • DenseNet’s feature reuse improves computational efficiency but causes heavy GPU memory consumption, potentially requiring memory-efficient implementations that increase training time.The section trains the deeper model on 96 × 96 Fashion-MNIST inputs for 10 epochs to simplify computation.Source: §8.7.6 · p. 365, §8.7.5 · p. 365

8.8 Designing Convolution Network Architectures

  • Modern CNN design remains highly dependent on human intuition, motivating systematic exploration of architecture design spaces such as those producing RegNetX/Y models.The section also frames DenseNet’s memory cost and alternative memory-reduction strategies as open design questions.Source: §8.6.6 · p. 360, §1 · pp. 365–366, §8.8 · p. 366
  • RegNet derives from network design spaces that optimize distributions of networks rather than a single architecture, combining manual design with neural architecture search to yield performant CNN families.The approach is computationally relatively inexpensive and produces guiding principles for CNN design.Source: §8.8 · pp. 366–367

8.8.1 The AnyNet Design Space

  • The AnyNet design space uses a stem, multistage ResNeXt body, and head, with 17 parameters controlling widths, depths, bottleneck ratios, and group widths.The body progressively reduces spatial resolution across four stages, while the head uses global average pooling and a fully connected classifier.Source: §8.8.1 · pp. 367–368, Figure 8.8.1 · p. 368

8.8.2 Distributions and Parameters of Design Spaces

  • Tying bottleneck ratios and group widths across stages does not affect accuracy, while increasing widths and depths across stages improves performance, reducing the design space by six parameters.These comparisons are reported through empirical error-distribution functions in Fig. 8.8.2.Source: §8.8.2 · pp. 370–371, Figure 8.8.2 · p. 371

8.8.3 RegNet

  • RegNet simplifies AnyNetXE by sharing bottleneck ratio and group width across stages, increasing depth across stages, and using widths that increase approximately linearly with block index.Experiments favor k = 1, meaning no bottlenecks; the principles also extend to larger-scale and SE-augmented RegNetY designs.Source: §8.8.3 · p. 371

8.8.4 Training

  • A 32-layer RegNetX can be trained on Fashion-MNIST with the same training procedure used previously, using 10 epochs and one GPU.The supplied implementation resizes Fashion-MNIST images to 96 × 96 and uses a learning rate of 0.05.Source: §8.8.4 · p. 372

8.8.5 Discussion

  • Vision Transformers now lead large-scale image classification, surpassing CNN accuracy as scalability and large image collections outweigh CNN inductive biases, albeit with higher computational cost.Recent hardware optimizations have further widened the performance gap in favor of Transformers.Source: §8.8.5 · p. 373

8.8.6 Exercises

  • The next chapter turns to sequential data, where recurrent neural networks model temporal dynamics through recurrent connections and shared parameters unfolded across sequence steps.Sequence modeling supports tasks such as captioning, speech synthesis, music generation, translation, dialogue, and robot control.Source: §8.8.6 · pp. 374–375, Figure 9.1 · p. 375

9.1 Working with Sequences

  • Sequence models represent ordered lists of feature vectors whose elements are dependent, supporting tasks with fixed or sequential targets and aligned or unaligned sequence-to-sequence inputs.Examples include documents, patient trajectories, sentiment classification, image captioning, part-of-speech tagging, machine translation, and video captioning.Source: §9.1 · pp. 376–377

9.1.1 Autoregressive Models

  • Autoregressive modeling predicts P(x_t | x_t−1, . . ., x_1), but variable history length requires fixed windows or latent summaries that update over time.Training commonly samples windows and may assume stationary dynamics even when individual observations change.Source: §9.1.1 · p. 378

9.1.2 Sequence Models

  • Language models estimate sequence probabilities by factoring p(x_1, . . ., x_T) into left-to-right conditional distributions, enabling likelihood evaluation, sequence sampling, and optimization.For discrete words, the model outputs a full vocabulary distribution for the next word given leftward context.Source: §9.1.2 · p. 379

Markov Models

  • Markov models discard history beyond τ steps when that preserves predictive power, although real text continues benefiting from longer context with rapidly diminishing gains.Modern RNN- and Transformer-based language models still generally use no more than thousands of words of context.Source: Markov Models · pp. 379–380

The Order of Decoding

  • Left-to-right decoding is preferred because it matches reading and next-word anticipation, supports arbitrarily long sequences, and often aligns with easier forward causal prediction.All factorization orders are valid, but causal structure can make predicting P(x_t+1 | x_t) easier than predicting P(x_t | x_t+1).Source: The Order of Decoding · p. 380

9.1.4 Prediction

  • 4-step-ahead predictions still look good, but predictions beyond that are almost useless because recursively fed errors accumulate and rapidly degrade accuracy.One-step-ahead predictions remain good near t = 1000, whereas multistep predictions decay toward a constant after a few steps.Source: §9.1.4 · pp. 382–384

9.2 Converting Raw Text into Sequence Data

  • Text preprocessing loads raw strings, tokenizes them, builds a vocabulary, and converts tokens into numerical indices for model input.The demonstrated pipeline uses The Time Machine, ignores punctuation and capitalization, and represents the corpus as character indices for later training.Source: §9.2 · p. 385

9.2.2 Tokenization

  • Token choice trades sequence length against vocabulary size: the same sentence can be represented by words or by many more characters.The section uses character tokenization after preprocessing to simplify later training.Source: §9.2.2 · pp. 386–387, §9.2.4 · p. 388

9.2.3 Vocabulary

  • A vocabulary assigns unique indices to tokens, can discard rare entries, and maps unseen or discarded tokens to <unk> while preserving reversibility for retained data.The resulting numerical sequence can be converted back to its original string representation.Source: §9.2.3 · pp. 387–388

9.2.5 Exploratory Language Statistics

  • Word frequencies follow Zipf’s law, and this pattern extends to bigrams and trigrams with smaller exponents as sequence length increases.Many n-grams are rare, while the number of distinct n-grams remains limited, indicating substantial structure but making frequency-based language modeling difficult.Source: §9.2.5 · pp. 390–391, §9.2.6 · p. 391

9.3 Language Models

  • Language models estimate the joint probability P(x1, x2, . . ., xT) and can generate or disambiguate text by conditioning each token on preceding context.Examples include resolving speech-recognition ambiguity and distinguishing plausible from implausible word sequences.Source: §9.3 · p. 392

Markov Models and n-grams

  • Markov and n-gram models approximate sequence probabilities by conditioning each token on only a limited number of preceding tokens.First-order models use the immediately preceding token, while higher-order unigram, bigram, and trigram formulations use progressively longer contexts.Source: Markov Models and n-grams · p. 393

Word Frequency

  • Estimating longer word combinations becomes unreliable because rare bigrams and unseen or sparsely observed trigrams receive insufficient counts.Without assigning nonzero probability to unseen combinations, such n-gram language models cannot use them, especially with small datasets or rare words.Source: Word Frequency · pp. 393–394

Laplace Smoothing

  • Laplace smoothing addresses zero-count problems by adding a small constant to every word count.The method is introduced using the total word count n and the number of unique words m.Source: Laplace Smoothing · p. 394
  • Neural language models are favored because Laplace-smoothed n-grams handle rare combinations poorly, require storing all counts, ignore word meaning, and struggle with novel long sequences.The discussion therefore focuses on neural networks for language modeling.Source: Laplace Smoothing · p. 394, §9.3.4 · p. 397

9.3.2 Perplexity

  • Perplexity evaluates how accurately a language model predicts the next token by averaging cross-entropy over a sequence and exponentiating it, making documents of different lengths comparable.Perplexity is 1 for perfect predictions, infinite when the target probability is always zero, and equal to vocabulary size under a uniform baseline.Source: §9.3.2 · p. 395

9.3.3 Partitioning Sequences

  • Language-model training partitions a token corpus into randomly offset subsequences, using each subsequence as input and the same sequence shifted one token as its target.For length-n subsequences, the target sequence is x_t+1 and has length n; minibatches randomly sample these input-target pairs.Source: §9.3.3 · p. 396

9.3.4 Summary and Discussion

  • The chapter’s workflow combines neural language modeling, random minibatch sampling of shifted sequences, and perplexity-based evaluation.Language models estimate joint sequence probabilities, while neural models address the sparsity and structural limitations of n-gram smoothing.Source: §9.3.4 · p. 397
  • Large language models can scale with data, model size, and training compute, enabling desired tasks by predicting output text from input instructions.The passage describes them as the basis of state-of-the-art systems across diverse tasks.Source: §9.3.4 · p. 397

9.3.5 Exercises

  • An exercise asks how much word-frequency and adjacent multi-word frequency information a four-gram must store for a 100,000-word training dataset.The question highlights the storage burden associated with higher-order n-gram models.Source: §9.3.5 · p. 397

2. How would you model a dialogue? … 4. What does the above result mean for gradients in RNNs?

This section develops recurrent neural network methods for sequence modeling, covering language modeling, gradient behavior, truncation, gated and deep architectures, and bidirectional models. It then applies encoder–decoder RNNs to English–French sequence-to-sequence translation, including data preparation, teacher forcing, masked training, autoregressive prediction, and evaluation.

9.4.3 RNN-Based Character-Level Language Models

  • Character-level RNN language models predict each next character from current and previous characters, training with per-step softmax and cross-entropy loss.For example, the output at time step 3 depends on “m”, “a”, and “c”, with “h” as the target next character.Source: §9.4.3 · pp. 401–402

9.4.2 Recurrent Neural Networks with Hidden States

  • RNN hidden states retain sequence history through recurrent computation, while shared parameters keep parametrization cost independent of sequence length.At each time step, the hidden state combines the current input with the previous hidden state and feeds both recurrence and output computation.Source: §9.4.2 · p. 400, §9.4.4 · p. 402

9.5.1 RNN Model

  • The from-scratch RNN updates a tanh hidden state one time step at a time from the current input and previous state, with tunable hidden-unit count.Its forward pass initializes a zero state when needed, loops over time steps, and returns outputs plus the final state.Source: §9.5.1 · p. 403

9.5.2 RNN-Based Language Model

  • The RNN language model uses equal input-output vocabulary dimensions and perplexity, making sequences of different lengths comparable during evaluation.The model wraps the RNN with a vocabulary-sized output transformation for language-model training and validation.Source: §9.5.2 · p. 404, Transforming RNN Outputs · p. 405

One-Hot Encoding

  • Categorical tokens are represented by vocabulary-length one-hot vectors, reshaped as time-step, batch, and vocabulary tensors for sequential hidden-state updates.A fully connected output layer then transforms each RNN output into token predictions.Source: One-Hot Encoding · p. 405, Transforming RNN Outputs · p. 405

9.5.3 Gradient Clipping

  • Long sequences create an additional depth of T recurrent steps, so backpropagation involves O(T) matrix products that can make gradients explode or vanish.The instability depends on the properties of the weight matrices.Source: §9.5.3 · p. 406
  • Gradient clipping limits exploding-gradient damage by projecting gradients onto a radius-θ ball, preserving direction while improving robustness, although it no longer follows the true gradient exactly.Exploding gradients can destabilize or diverge training; clipping is widely used in RNN implementations and limits any minibatch’s influence.Source: §9.5.3 · p. 407

9.5.4 Training

  • A character-level RNN language model trained on The Time Machine uses clipped gradients, and prefix conditioning generates likely continuations for applications such as autocomplete.The model predicts one character at a time after warming up on the supplied prefix, then feeds each prediction back as the next input.Source: §9.5.4 · p. 407, §9.5.5 · p. 408, §9.5.6 · p. 409

2. Which hyperparameter controls the length of history used for prediction?

  • The section’s exercises extend the model by tuning history-related hyperparameters, replacing one-hot encoding with embeddings, testing cross-book generalization, sampling outputs, and examining whether ReLU still requires gradient clipping.These exercises probe perplexity, transfer across authors and works, alternative decoding, and the effect of activation functions on exploding gradients.Source: §2 · pp. 409–410, §9 · p. 410

9.6.2 Training and Predicting

  • The high-level RNN implementation achieves comparable perplexity to the from-scratch model while running faster through optimized library implementations.High-level APIs provide standard RNN components and avoid reimplementing common models, reducing implementation and computation time.Source: §9.6 · p. 410, §9.6.2 · p. 411, §9.6.3 · p. 412

9.7 Backpropagation Through Time

  • Backpropagation through time (Werbos, 1990) unrolls the recurrent network, reuses tied parameters at every time step, and sums their gradient contributions across occurrences.Sequences exceeding 1,000 tokens create memory and numerical-stability problems because forward and backward propagation require many matrix products.Source: §9.7 · p. 413

Full Computation

  • Full computation of the recurrent gradient is rarely used because it is slow, can amplify small initial-condition changes, and becomes computationally infeasible for long sequences.The full sum may produce exploding gradients and undesirable instability, despite representing the untruncated calculation.Source: Full Computation · p. 414, Comparing Strategies · p. 415

Truncating Time Steps

  • Truncated backpropagation through time terminates the gradient after τ steps, approximating the true gradient while emphasizing short-term influence and simpler, more stable models.This approximation works well in practice and is the regular truncation used in the RNN experiments.Source: Truncating Time Steps · pp. 414–415, Comparing Strategies · p. 415

Randomized Truncation

Comparing Strategies

  • Regular truncation is often preferable to randomized truncation because it has a slight desirable regularizing effect, whereas randomized truncation adds variance without improving practice much.The passages attribute randomized truncation’s limited benefit to sufficient short-range dependencies, increased variance, and a preference for short interactions.Source: Comparing Strategies · pp. 415–416

9.7.2 Backpropagation Through Time in Detail

  • Backpropagation through time applies the chain rule backward across stored recurrent computations to obtain gradients for output and hidden-layer parameters.Intermediate gradients are cached and reused, including ∂L/∂h_t for computing gradients with respect to both W_hx and W_hh.Source: §9.7.2 · pp. 416–418
  • High powers of W_hh^⊤ make RNN gradients numerically unstable, with eigenvalues below 1 vanishing and those above 1 diverging.Truncating backpropagation through time limits these effects, while gradient clipping addresses exploding gradients but not vanishing gradients.Source: §9.7.2 · p. 417, §4 · p. 419

10.1 Long Short-Term Memory (LSTM)

  • LSTMs replace ordinary recurrent nodes with memory cells whose fixed-weight self-connected state can carry gradients across many time steps without vanishing or exploding.Multiplicative gates determine which inputs enter the memory, when stored content is forgotten, and when the memory affects the output.Source: §4 · p. 419, §10.1 · p. 420

Input Gate, Forget Gate, and Output Gate

  • Learned input, forget, and output gates regulate whether new information updates the cell, old information is retained or flushed, and stored information reaches subsequent layers.The gates use sigmoid activations in (0, 1), while the candidate input uses tanh; the output gate can hide accumulated memory until a later time step.Source: Input Gate, Forget Gate, and Output Gate · p. 421, Hidden State · p. 423

Memory Cell Internal State

  • If the forget gate stays at 1 and the input gate at 0, the cell state remains unchanged, and this learned retention mechanism alleviates vanishing gradients on long sequences.The hidden state is formed by applying tanh to the cell state and modulating it elementwise with the output gate.Source: Memory Cell Internal State · p. 422, Hidden State · p. 423

10.1.2 Implementation from Scratch

  • The chapter implements LSTMs from scratch and through a high-level API, then trains them on The Time Machine dataset and generates the prediction “it has the the the the the.”The concise implementation uses compiled operators and substantially reduces the configuration details made explicit in the scratch version.Source: §10.1.2 · p. 423, Training and Prediction · p. 425, §10.1.3 · pp. 425–426

10.2.1 Reset Gate and Update Gate

  • GRUs replace LSTM’s three gates with reset and update gates: reset controls retained history, while update controls copying the old state versus adopting new information.Both gates use sigmoid activations, constraining their values to the interval (0, 1).Source: §10.2.1 · p. 427, Figure 10.2.1 · p. 428

10.2.2 Candidate Hidden State

  • The reset gate modulates the candidate hidden state: values near 1 recover a vanilla RNN, whereas values near 0 reset prior state and use an MLP of the current input.The candidate uses a tanh activation and elementwise multiplication with the previous hidden state.Source: §10.2.2 · p. 428, Figure 10.2.2 · p. 429

10.2.3 Hidden State

  • The update gate forms an elementwise convex combination of the previous and candidate states, retaining the old state near 1 and approaching the candidate near 0.Retaining the old state effectively skips the current time step in the dependency chain.Source: §10.2.3 · p. 429, Figure 10.2.3 · p. 429

10.2.4 Implementation from Scratch

10.2.6 Summary

  • GRUs achieve similar performance to LSTMs with lower computational cost, capture long-distance dependencies better than simple RNNs, and can skip subsequences through their gates.GRUs reduce to basic RNNs in an extreme gate setting and use update gates to skip subsequences.Source: §10.2.6 · p. 432

10.3 Deep Recurrent Neural Networks

  • Deep RNNs stack recurrent layers so each layer receives the preceding layer’s sequence, with cells depending on same-layer previous states and previous-layer current outputs.The number of layers L and hidden units h are tunable, and LSTM or GRU cells can replace the hidden-state computation.Source: §10.3 · pp. 433–434, Figure 10.3.1 · p. 433

10.3.3 Summary

  • Deep RNNs pass hidden information across time within layers and across layers, but require careful initialization, learning-rate choices, and gradient clipping for convergence.The section notes that LSTMs, GRUs, and vanilla RNNs are available through high-level deep-learning APIs.Source: §10.3.3 · p. 436

10.4 Bidirectional Recurrent Neural Networks

  • Bidirectional RNNs combine forward and backward recurrent states, allowing each time step to use context from both before and after it for sequence encoding and observation estimation.They concatenate the two directional outputs, doubling the hidden representation to 2h; this design is costly to train because of long gradient chains.Source: §10.4 · pp. 437–438, §10.4.3 · p. 439
  • Bidirectional RNNs are inappropriate when predictions must condition only on prior tokens, but are useful for part-of-speech detection and masked-token prediction with two-sided context.The likely completion of a blank can change substantially depending on words that follow it, motivating access to rightward context.Source: §10.4 · p. 437

10.4.1 Implementation from Scratch

  • The bidirectional architecture runs two unidirectional RNNs on the same input in opposite directions and concatenates corresponding outputs before the output layer.In the scratch implementation, forward and backward states use separate learnable RNNs, are updated independently, and their outputs are concatenated after reversing the backward sequence.Source: §10.4 · p. 437, §10.4.1 · pp. 438–439

10.4.2 Concise Implementation

  • High-level APIs provide a concise bidirectional GRU implementation by setting bidirectional=True and doubling the model’s hidden-state output dimension.The text presents this API-based construction as an alternative to implementing two recurrent networks manually.Source: §10.4.2 · p. 439

10.5 Machine Translation and the Dataset

  • Sequence-to-sequence problems map unaligned source and target sequences, including machine translation, dialogue prompts and replies, and questions and answers.Unlike language modeling, the source and target sequences can have different lengths and corresponding words need not appear in the same order.Source: §10.5 · p. 440

10.5.1 Downloading and Preprocessing the Dataset

  • The example dataset contains English–French bilingual sentence pairs from the Tatoeba Project, with preprocessing that normalizes spaces and case and separates punctuation from words.Each tab-delimited line contains an English source sequence and its French target sequence, which may span one or multiple sentences.Source: §10.5.1 · pp. 440–441

10.5.2 Tokenization

  • Word-level tokenization appends <eos> markers, builds separate source and target token lists, and reveals that most sequences in this dataset contain fewer than 20 tokens.Infrequent tokens are later mapped to <unk>, while target inputs use <bos> and shifted target labels during training.Source: §10.5.2 · pp. 441–442, §10.5.3 · p. 443

10.5.4 Reading the Dataset

  • The English–French dataset represents source sequences with tokens followed by <eos> and padding, while decoder inputs begin with <bos> and target labels end with <eos>.Word-level tokenization shortens sequences but enlarges vocabularies; rare tokens can be mapped to an unknown token, and padding or length bucketing supports minibatches.Source: §10.5.4 · p. 445, §10.5.5 · p. 445

10.6 The Encoder−Decoder Architecture

  • Encoder–decoder architectures handle unaligned variable-length inputs and outputs by encoding the input into a fixed-shape state and decoding a variable-length target sequence.The decoder conditions on the encoded input and leftward target context to predict subsequent tokens.Source: §10.6 · p. 446, §10.6.4 · p. 447

10.6.1 Encoder

  • The reusable architecture interface separates an encoder that accepts variable-length sequences from a decoder that initializes state from encoder outputs and generates tokens sequentially.The combined forward pass encodes the input, initializes decoder state, and returns decoder outputs.Source: §10.6.1 · p. 446, §10.6.2 · pp. 446–447, §10.6.3 · p. 447

10.7 Sequence-to-Sequence Learning for Machine Translation

  • RNN sequence-to-sequence translation uses a separate encoder RNN and decoder RNN, with training conditioned on ground-truth prefixes and testing conditioned on previously predicted tokens.The decoder stops when it generates <eos>; designs differ in whether the encoder’s final hidden state is supplied at every decoding step or only initially.Source: §10.7 · pp. 448–449

10.7.1 Teacher Forcing

  • Teacher forcing feeds the decoder <bos> plus the target sequence without its final token, while training labels are the target sequence shifted one token to the left.An alternative feeds the decoder’s previous predicted token as the current input.Source: §10.7.1 · p. 449

10.7.2 Encoder

  • The RNN encoder embeds tokens and transforms their hidden states into a context variable, using the final hidden state in the basic unidirectional design or information from both directions in a bidirectional design.The implementation uses a multilayer GRU; for a two-layer example with 16 hidden units, encoder outputs have shape (number of time steps, batch size, number of hidden units).Source: §10.7.2 · pp. 449–451

10.7.3 Decoder

  • The RNN decoder predicts each next-token distribution from the previous target token, its prior hidden state, and the context variable, then applies an output layer and softmax.In the implementation, the encoder’s final hidden state initializes the decoder, and the context is concatenated with decoder inputs at every time step.Source: §10.7.3 · p. 451
  • The decoder outputs a tensor shaped (batch size, number of time steps, vocabulary size), representing predicted token distributions at every time step.Its recurrent state has shape (num_layers, batch size, num_hiddens).Source: §10.7.3 · p. 452

10.7.4 Encoder–Decoder for Sequence-to-Sequence Learning

  • The complete sequence-to-sequence model combines an RNN encoder and decoder, with code defining initialization, validation, and optimization components.The implementation is organized as a Seq2Seq encoder–decoder module.Source: §10.7.4 · p. 453

10.7.5 Loss Function with Masking

  • The loss applies cross-entropy to decoder token distributions while masking padding positions so irrelevant predictions do not affect optimization.Padding tokens are appended to equalize minibatch shapes for sequences of varying lengths.Source: §10.7.5 · p. 453

10.7.6 Training

  • The encoder–decoder is trained on a machine-translation dataset using an RNN encoder and decoder with embedding size 256, hidden size 256, two layers, and dropout 0.2.The implementation uses the Adam optimizer.Source: §10.7.6 · p. 453, §10.7.4 · p. 453

10.7.7 Prediction

  • At each prediction step, the decoder feeds its highest-probability previous token back as input, beginning with <bos> and stopping at <eos>.The section notes that more sophisticated beam-search strategies are introduced next.Source: §10.7.7 · p. 454, Figure 10.7.3 · p. 454

10.7.8 Evaluation of Predicted Sequences

  • BLEU evaluates matched n-gram precision while penalizing shorter predicted sequences, assigning greater weight to accurate longer n-grams.BLEU equals 1 when prediction and target match exactly; for k = 2, a shortened prediction receives a penalty factor of approximately 0.14.Source: §10.7.8 · p. 455
  • BLEU was 1.000 for “go .”, “i lost .”, and “i'm home .”, but 0.000 for “he's calm .” in the French translations.The trained RNN encoder–decoder was used to translate four English sentences and compute BLEU scores.Source: §10.7.8 · p. 456

10.7.9 Summary

  • Overall, the section presents teacher-forced RNN encoder–decoder training, masked loss computation, autoregressive decoding, and BLEU evaluation for output sequences.Multilayer RNNs can be used for the encoder and decoder, while masks filter irrelevant loss computations.Source: §10.7.9 · p. 456

1. Can you adjust the hyperparameters to improve the translation results? … 2. Given a powerful language model, what applications can you think of?

This section develops attention-based sequence modeling from decoding strategies and kernel regression through Bahdanau attention, multi-head self-attention, and Transformers for translation and vision. It then considers Transformer pretraining, scaling, applications, and the optimization challenges involved in training models that generalize.

1. Can you adjust the hyperparameters to improve the translation results?

  • The first subsection asks readers to investigate how masks, hidden-state initialization, teacher forcing, and replacing GRU with LSTM affect translation results.These exercises vary loss calculation, encoder–decoder dimensional mismatches, training inputs, and recurrent architecture.Source: §1 · p. 457

10.8.1 Greedy Search

  • A sequence with probability 0.054 surpasses greedy search’s 0.048, demonstrating that locally highest-probability tokens need not form the most likely output.Greedy search selects each token independently, whereas the globally most likely sequence maximizes the product of conditional probabilities.Source: §10.8.1 · pp. 458–459

10.8.2 Exhaustive Search

  • Exhaustive search finds the highest-probability sequence but costs O(|Y|T′), whereas greedy search costs O(|Y| T ′) and can be far from optimal.With |Y| = 10000 and T ′ = 10, exhaustive search evaluates 10^40 sequences while greedy search evaluates 10^5.Source: §10.8.2 · p. 459

10.8.3 Beam Search

  • Beam search trades accuracy for efficiency through beam size k, costing O(k |Y| T ′) and reducing to greedy search when k = 1.At each step, it retains the k highest-scoring candidate sequences and ultimately scores completed candidates with a length penalty, usually α = 0.75.Source: §10.8.3 · pp. 459–460, §10.8.4 · p. 460

10.8.4 Summary

  • Transformers replace recurrent connections with attention and became dominant across natural language processing, especially when combined with large-scale pretraining and downstream fine-tuning.The section presents Transformer-based models as default approaches for many language tasks and notes their competitiveness in vision, speech, reinforcement learning, and graph neural networks.Source: §10.8.4 · pp. 462–463
  • Attention was introduced to let decoders revisit and selectively focus on different input positions at each decoding step instead of relying on one fixed-length representation.Differentiable attention can improve translation performance, while attention-weight patterns may offer qualitative clues whose interpretation remains uncertain.Source: §10.8.4 · p. 463

11.1 Queries, Keys, and Values

  • Attention computes a query-dependent linear combination of values, typically using nonnegative weights that form a convex combination, without requiring database compression.Its differentiable softmax weighting supports learned, content-dependent retrieval over arbitrarily large key–value sets.Source: §11.1 · pp. 464–465

11.1.1 Visualization

  • The visualization utility represents attention weights across multiple queries and keys as heatmaps, helping inspect how queries affect attention over keys.The function accepts a four-axis tensor organized by display rows, display columns, queries, and keys.Source: §11.1.1 · p. 466, Figure 11.1.1 · p. 466

11.1.2 Summary

  • Attention provides differentiable control for selecting elements from key–value sets and constructing weighted sums over their representations.Queries, keys, and values can be interpreted through regression, where queries are evaluation locations, keys are observed locations, and values are observed outputs.Source: §11.1.2 · p. 467

11.2.2 Attention Pooling via Nadaraya–Watson Regression

  • Nadaraya–Watson regression implements attention by normalizing query–key kernel similarities into weights and taking their weighted sum of training labels.The method requires little or no training and can support regression or classification; its weights depend on query–key similarity and the availability of similar observations.Source: §11.2.2 · p. 470, §11.2.4 · pp. 472–473
  • Gaussian, Boxcar, and Epanechikov kernels produce workable regression estimates, whereas the constant kernel yields the unrealistic estimate f(x) = 1.The nontrivial kernels are similar because they induce similar attention weights despite differing functional forms.Source: §11.2.2 · p. 471

11.2.3 Adapting Attention Pooling

  • Narrower kernels produce less-smooth estimates that better follow local variation, while concentrating large attention weights over a narrower range.Using one fixed width may be suboptimal because suitable widths can depend on local density, illustrating the limits of hand-crafted attention mechanisms.Source: §11.2.3 · p. 472

11.3.1 Dot Product Attention

  • Scaled dot-product attention replaces more expensive distance functions by rescaling query–key dot products by 1/sqrt(d), preserving unit variance as vector dimensionality grows.Softmax normalizes the resulting scores, while query-only and approximately constant key-norm terms can be omitted under normalization and normalization layers.Source: §11.3 · p. 473, §11.3.1 · p. 474

Masked Softmax Operation

  • Masked softmax prevents padded tokens in variable-length sequences from contributing to attention outputs or gradients.Implementations mask invalid positions with large negative values and zero their associated values, favoring GPU-efficient linear algebra over conditional branches.Source: Masked Softmax Operation · p. 475
  • Masked softmax sets positions beyond each supplied valid length to zero, supporting either one valid length per example or separate lengths for each query vector.The operation replaces masked logits with a very large negative value before softmax, and the section also introduces batch matrix multiplication for batched queries, keys, and values.Source: Masked Softmax Operation · p. 476, Batch Matrix Multiplication · p. 476

11.3.3 Scaled Dot Product Attention

  • The scaled dot product attention computes minibatch query-key scores divided by sqrt(d), masks invalid positions, applies dropout, and aggregates the values.It supports queries Q ∈Rn×d, keys K ∈Rm×d, and values V ∈Rm×v, with valid lengths controlling which attention weights remain active.Source: §11.3.3 · p. 477

11.3.4 Additive Attention

  • Additive attention projects queries and keys into a shared hidden space, combines them with tanh, and normalizes scores before value aggregation, enabling different query and key dimensions.In the toy example, only terms within valid lengths (2, 6) are nonzero, qualitatively matching dot product attention.Source: §11.3.4 · pp. 478–479

11.4 The Bahdanau Attention Mechanism

  • Bahdanau attention dynamically aligns relevant input tokens with each prediction, overcoming the fixed-dimensional bottleneck that makes long and complex sequence translation infeasible.The mechanism updates the decoder state using source information selected for the current prediction and helped give rise to Transformers (Vaswani et al., 2017).Source: §11.4 · pp. 480–481

11.4.1 Model

  • The attention decoder uses the previous decoder hidden state as a query and encoder hidden states as keys and values, dynamically updating the context at every decoding step.The resulting context generates the next decoder state and token; additive attention computes the weights, while later variants can include already generated tokens as further context (Chan et al. (2015)).Source: §11.4.1 · p. 481

11.4.2 Defining the Decoder with Attention

  • The implemented decoder initializes attention from encoder hidden states, excludes padding through valid lengths, and concatenates each attended context with the input embedding before the RNN.At each step, the final decoder-layer hidden state queries the attention mechanism, whose output and the embedding are passed together to the RNN decoder.Source: §11.4.2 · p. 482

11.4.4 Summary

  • Bahdanau attention selectively aggregates encoder hidden states, using the previous decoder hidden state as query and encoder states as keys and values.The attention weights are non-uniform, so different decoding steps selectively aggregate different parts of the input sequence.Source: §11.4.4 · p. 485, §11.4.3 · p. 485

11.5 Multi-Head Attention

  • Multi-head attention projects queries, keys, and values into h representation subspaces, applies attention in parallel, then concatenates and linearly transforms the heads.The implementation uses scaled dot-product attention with p_q = p_k = p_v = p_o/h to limit computational and parameter growth.Source: §11.5 · p. 486, §11.5.2 · p. 487

11.6.1 Self-Attention

  • Self-attention maps each token to a representation attending to every token, with O(n^2d) complexity, O(1) sequential operations, and O(1) maximum path length.Unlike RNNs, self-attention is parallelizable, but its quadratic sequence-length cost can make it prohibitively slow for very long sequences.Source: §11.6.1 · p. 489, §11.6.2 · p. 491

11.6.3 Positional Encoding

  • Sinusoidal positional encodings add sequence-order information while preserving self-attention’s parallelism, and fixed offsets can be represented by position-independent linear projections.The encodings use alternating sine and cosine functions whose frequencies decrease across dimensions, supporting absolute and relative positional information.Source: §11.6.3 · p. 491, Relative Positional Information · p. 493, §11.6.4 · p. 494

11.7 The Transformer Architecture

  • The Transformer replaces convolutional and recurrent layers with stacked self-attention-based encoder and decoder modules, using positional encodings for sequence-to-sequence learning.Self-attention offers parallel computation and short maximum path length, motivating deep attention-based architectures.Source: §11.7 · p. 494, §11.7.1 · p. 495

11.7.1 Model

  • The encoder stacks multi-head self-attention and positionwise feed-forward sublayers with residual connections and layer normalization, while the decoder adds encoder–decoder attention and masked self-attention.Decoder masking ensures each prediction depends only on previously generated output tokens.Source: §11.7.1 · p. 496, §11.7.5 · pp. 499–500

11.7.2 Positionwise Feed-Forward Networks

  • The positionwise feed-forward network applies the same two-layer MLP independently at every sequence position, transforming the feature dimension while preserving identical outputs for identical inputs.Its input shape is (batch size, sequence length, hidden units), and its output uses ffn_num_outputs as the final feature dimension.Source: §11.7.2 · pp. 496–497

11.7.3 Residual Connection and Layer Normalization

  • Residual connections followed by layer normalization stabilize the Transformer, with layer normalization normalizing across features and remaining independent of batch size and sequence length.The AddNorm implementation also applies dropout, and residual addition requires matching input shapes.Source: §11.7.3 · pp. 497–498

11.7.4 Encoder

  • Each encoder block preserves its input shape while combining multi-head self-attention with a positionwise feed-forward network surrounded by residual connections and layer normalization.Learnable embeddings are scaled by the square root of the embedding dimension before positional encoding is added.Source: §11.7.4 · pp. 498–499

11.7.5 Decoder

  • Each decoder block combines masked self-attention, encoder–decoder attention, and a positionwise feed-forward network, then produces vocabulary predictions through a fully connected layer.The decoder and encoder use the same feature dimension for attention and residual additions, while attention weights are retained for visualization.Source: §11.7.5 · pp. 499, 501

11.7.6 Training

  • The implementation trains a two-layer, 4-head Transformer on English–French translation with 256 hidden units, 64 feed-forward units, 0.2 dropout, learning rate 0.001, and 30 epochs.Training uses batches of 128, gradient clipping at 1, and one GPU.Source: §11.7.6 · pp. 501–502
  • Transformer self-attention visualizations enforce padding masks and autoregressive decoding: encoder queries ignore padding, while decoder queries cannot attend to future positions or padded inputs.Each attention head independently attends through its own representation subspace.Source: §11.7.6 · pp. 503–504

11.7.8 Exercises

  • The exercises ask readers to study deeper Transformers, compare additive with scaled dot-product attention, choose encoder or decoder designs for language modeling, and improve efficiency for long sequences.They also prompt testing how image size affects training time and how averaged patch representations affect classification accuracy.Source: §11.7.8 · p. 505, §4 · p. 505

11.8.1 Model

  • The vision Transformer splits images into patches, linearly projects them with positional embeddings, prepends a <cls> token, and classifies using its encoded representation.Patch embedding can be implemented as a convolution whose kernel and stride equal the patch size; the encoder then applies multi-head attention and feed-forward blocks.Source: §11.8.1 · pp. 506–507, §11.8.2 · pp. 507–508, §11.8.4 · p. 509

11.8.3 Vision Transformer Encoder

  • Vision Transformer encoder blocks use GELU-based dropout MLPs, pre-normalization before attention and the MLP, residual connections, and unchanged sequence shapes.Pre-normalization is described as enabling more effective or efficient Transformer training (Baevski and Auli, 2018, Wang et al., 2019, Xiong et al., 2020).Source: §11.8.3 · pp. 508–509

11.8.6 Summary and Discussion

  • Vision Transformers underperform ResNets on small datasets such as Fashion-MNIST and even ImageNet, but outperform them substantially when trained at larger scale.This limitation reflects the absence of convolutional translation invariance and locality, while larger models and datasets reveal superior scalability.Source: §11.8.6 · pp. 510–511
  • Swin Transformers reduce self-attention’s quadratic image-size complexity, restore convolution-like priors, and extend Transformers beyond classification with state-of-the-art computer-vision results (Liu et al., 2021).Vision Transformers also became effective on ImageNet with data-efficient DeiT training strategies (Touvron et al., 2021).Source: §11.8.6 · p. 511

11.9 Large-Scale Pretraining with Transformers

  • Transformers scale better with larger pretraining data, model size, and compute, with language-model performance following a power law in parameters, tokens, and training compute (Kaplan et al., 2020).Pretraining addresses the task-specificity and distribution sensitivity of models trained from scratch.Source: §11.9 · p. 511

11.9.6 Summary and Discussion

  • Transformers extend across modalities: Gato serializes text, images, joint torques, and button presses as tokens, while larger Parti models show higher-fidelity image generation and richer text understanding.GPT-4 accepts text and images as input, and models such as Flamingo, CLIP, and DALL-E 2 apply Transformer representations to multimodal tasks.Source: §11.9 · p. 512, §11.9.6 · p. 522, Figure 11.9.12 · p. 522

Pretraining BERT

  • BERT pretrains bidirectional encoder representations by predicting randomly masked tokens using unrestricted self-attention, then fine-tunes them with task-specific layers for classification, tagging, regression, and question answering.Its 350-million-parameter model was pretrained on 250 billion tokens without manual labeling, and downstream fine-tuning updates both pretrained and newly added parameters.Source: Pretraining BERT · p. 512, Fine-Tuning BERT · p. 513, Figure 11.9.1 · p. 513

11.9.2 Encoder–Decoder

  • Encoder–decoder Transformers generate arbitrary-length sequences autoregressively by combining encoder–decoder cross-attention with causal decoder self-attention, supporting machine translation and text-to-text tasks.BART and T5 reconstruct original text from corrupted inputs, while T5 unifies tasks by conditioning generation on a task description and input text.Source: §11.9.2 · p. 514, Pretraining T5 · pp. 514–515

Fine-Tuning T5

  • T5-11B achieved state-of-the-art results on multiple encoding and generation benchmarks, while its frozen T5-XXL encoder supported Imagen’s photorealistic text-to-image generation.T5 is pretrained by reconstructing corrupted text from 1000 billion C4 tokens and fine-tuned with task descriptions without requiring additional layers.Source: Pretraining T5 · p. 515, Fine-Tuning T5 · p. 516, Figure 11.9.5 · p. 516

GPT and GPT-2

  • GPT-2 achieved state-of-the-art language-modeling results and promising results on other tasks without updating parameters or architecture, enabling more efficient reuse than fine-tuning.GPT-2 used 1.5 billion parameters and 40 GB of text, with pre-normalization, improved initialization, and weight scaling.Source: GPT and GPT-2 · p. 517, GPT-3 and Beyond · p. 518

GPT-3 and Beyond

  • GPT-3’s few-shot performance increased most rapidly with model size after pretraining 300 billion tokens, demonstrating scalable in-context learning across 42 accuracy-denominated benchmarks.In-context learning conditions generation on task descriptions, examples, and prompts without parameter updates, spanning zero-shot, one-shot, and few-shot settings.Source: GPT-3 and Beyond · p. 518, Figure 11.9.8 · p. 519

11.9.4 Scalability

  • Transformer performance improves smoothly as model size, dataset size, and training compute increase together, following power-law trends when the other factors are not bottlenecks.Larger models also achieve a given performance level with fewer training tokens, although the optimal balance among scaling factors remains debated.Source: Figure 11.9.9 · p. 519, §11.9.4 · pp. 519–520

11.9.5 Large Language Models

  • Large language models support prompting-based applications including instruction following, code debugging, creative writing, question answering, and chain-of-thought mathematical, commonsense, and symbolic reasoning.Instruction tuning and reinforcement learning from human feedback improve alignment, while sampling multiple reasoning paths, diversifying demonstrations, and multimodal inputs can improve reasoning accuracy.Source: §11.9.5 · pp. 521–522

12.1.1 Goal of Optimization

  • Training-error minimization can differ from generalization-error minimization because empirical-risk and population-risk minima may occur at different locations.Deep learning therefore must address overfitting in addition to optimizing the training objective.Source: §12.1.1 · p. 525

12.1.2 Optimization Challenges in Deep Learning

  • Deep learning optimization relies on numerical algorithms and faces local minima, saddle points, and vanishing gradients.The chapter focuses on minimizing objective functions rather than directly minimizing model generalization error.Source: §12.1.2 · pp. 525–526

Local Minima

  • Local minima can trap numerical optimization near solutions that minimize the objective only locally, although minibatch gradient noise may help dislodge parameters.Deep learning objectives typically contain many local optima, making global minimization difficult.Source: Local Minima · p. 526

Saddle Points

  • High-dimensional deep learning problems make saddle points more likely than local minima because Hessians commonly contain negative eigenvalues.At saddle points, gradients vanish even though the point is neither a local nor global minimum; most deep learning problems are nonconvex.Source: Saddle Points · pp. 527–528

Vanishing Gradients

  • At x = 4, minimizing tanh(x) gives f′(4) = 0.0013, so optimization can stall for a long time; ReLU helped address this difficulty.Vanishing gradients are especially problematic when derivatives become nearly zero.Source: Vanishing Gradients · pp. 528–529

12.1.3 Summary

  • Optimization may have many local minima, even more saddle points in nonconvex problems, and vanishing gradients that stall progress.Reparameterization and good initialization can help, while approximate local solutions often remain useful and robust algorithms are accessible to beginners.Source: §12.1.3 · p. 529, Vanishing Gradients · p. 529

12.1.4 Exercises

  • The exercises ask readers to analyze equivalent MLP minima and prove eigenvalue symmetry for symmetric random matrices.The first exercise concerns at least d! equivalent solutions in an MLP with hidden dimension d.Source: §12.1.4 · pp. 529–530

3. What other challenges involved in deep learning optimization can you think of? … 2. How rapid is the rate of convergence for the algorithm?

This section develops and analyzes deep-learning optimization, from convexity, gradient descent, and minibatch computation to momentum and adaptive methods. It examines convergence, efficiency, learning-rate control, practical implementation, and the trade-offs among speed, cost, noise, and stability.

Convex Sets

  • A convex set contains every line segment joining two of its points; intersections preserve convexity, whereas unions generally do not.The text applies these definitions to typical deep-learning domains such as R^d and bounded-norm balls.Source: Convex Sets · pp. 530–531, §12.2.4 · p. 538

Convex Functions

  • A convex function lies below every chord between points in its domain, and Jensen’s inequality extends this principle by bounding the expectation of a convex function below by the function of an expectation.The section illustrates convexity with parabolic and exponential functions, contrasts them with cosine, and notes applications to variational methods.Source: Convex Functions · p. 532, Jensen’s Inequality · p. 533

Local Minima Are Global Minima

  • A local minimum of a convex function on a convex set is also a global minimum, although multiple global minima may exist or a minimum may not be attained.The examples include f(x) = max(|x| −1, 0), which minimizes across an interval, and exp(x), which approaches but never reaches 0 on R.Source: Local Minima Are Global Minima · pp. 533–534

Convexity and Second Derivatives

  • For twice-differentiable functions, convexity is equivalent to a nonnegative second derivative in one dimension and a positive-semidefinite Hessian in multiple dimensions.The multidimensional proof reduces convexity along line segments to the condition (x −y)⊤H(x −y) ≥0.Source: Convexity and Second Derivatives · pp. 535–536

12.2.3 Constraints

  • Constrained optimization can be formulated with Lagrange multipliers, approximated with penalty terms, or enforced through projections onto convex sets.Penalties are described as more robust than exact satisfaction in practice, while projections select the closest point in the constraint set and can produce sparse weights via an ℓ1 ball.Source: Lagrangian · pp. 536–537, Penalties · p. 537, Projections · pp. 537–538

12.2 Convexity

  • Convex functions motivate optimization algorithms because they are easier to analyze, while deep-learning objectives may exhibit convex-like behavior near local minima.The section explicitly positions convexity as a way to understand gradient descent and stochastic gradient descent in detail.Source: §12.2 · p. 530, §12.2.4 · p. 538

Learning Rate

  • A learning rate that is too small slows updates, whereas η = 1.1 overshoots x = 0 and gradually diverges.The section introduces gradient descent as foundational for understanding stochastic gradient descent, where learning-rate choice can also cause divergence.Source: §12.3 · p. 539, Learning Rate · p. 542

12.3.2 Multivariate Gradient Descent

  • With η = 0.1, multivariate gradient descent approaches the minimum [0, 0] after 20 steps, but progress remains fairly well-behaved and slow.The example uses f(x) = x1^2 + 2x2^2 from initial position [−5, −2].Source: §12.3.2 · p. 544

12.3.3 Adaptive Methods

  • Adaptive methods use curvature to avoid manual learning-rate selection, while line search chooses η by binary search to minimize f(x − η∇f(x)).Full second-order methods are computationally expensive because deep neural networks can require storing O(d^2) Hessian entries; nonconvex objectives may also contain many local minima.Source: Local Minima · p. 542, §12.3.3 · p. 545, Newton’s Method · p. 545

Newton’s Method

  • Newton’s method can converge in one step for a quadratic, but on a nonconvex function it can fail catastrophically, reaching x = 26.8341 after 10 iterations.Using η = 0.5 improves the nonconvex example, reaching x = 7.2699 after 10 iterations; second-order information adapts steps to curvature but remains costly for deep networks.Source: Newton’s Method · pp. 545, 547

Convergence Analysis

  • Under convexity, nonzero second derivative, and sufficient smoothness, Newton’s method has quadratically decreasing error once iterates enter a suitable bounded region.The analysis does not guarantee when that rapid-convergence region will be reached and requires well-behaved higher-order derivatives.Source: Convergence Analysis · pp. 547–548

Preconditioning

  • Preconditioning approximates the Hessian with diagonal entries, effectively assigning each coordinate a different learning rate to correct mismatched parameter scales.It is less powerful than full Newton’s method but better than unmodified gradient descent and motivates stochastic-gradient optimization algorithms.Source: Preconditioning · pp. 548–549

12.4.1 Stochastic Gradient Updates

  • SGD lowers each iteration’s cost from O(n) to O(1) by sampling one example, whose stochastic gradient is an unbiased estimate of the full gradient.The estimate is accurate on average, but its randomness makes optimization trajectories noisier than full gradient descent.Source: §12.4.1 · pp. 550–551
  • With a constant learning rate, stochastic noise prevents high-quality convergence; the rate must decrease dynamically because rates that are too small or large respectively stall progress or yield poor solutions.After 50 steps, quality remains poor and does not improve with more steps under a constant rate.Source: §12.4.1 · p. 552, §12.4.2 · p. 552, §12.4.5 · p. 556

12.4.2 Dynamic Learning Rate

  • Exponential decay reduces parameter variance but fails to converge to x = (0, 0) after 1000 steps, whereas inverse-square-root polynomial decay improves convergence after 50 steps.Decay that is too fast can stop optimization prematurely, while decay that is too slow wastes optimization time.Source: §12.4.2 · pp. 552–553

12.4.3 Convergence Analysis for Convex Objectives

  • For convex objectives, stochastic gradient descent converges to the optimum at rate O(1/√T), with speed governed by the gradient bound L and initial distance r.Choosing η = r/(L√T) yields an upper bound proportional to rL/√T for the averaged iterate.Source: §12.4.3 · p. 555, §12.4.5 · p. 556

12.4.4 Stochastic Gradients and Finite Samples

  • Sampling without replacement is preferred because sampling with replacement increases variance and decreases data efficiency; repeated passes still traverse the dataset in a different random order.Sampling an example at least once with replacement has probability approximately 0.63.Source: §12.4.4 · p. 556

12.5 Minibatch Stochastic Gradient Descent

  • Full gradient descent becomes more expensive as datasets grow, while SGD is computationally cheaper but less data-efficient and noisier, motivating intermediate minibatch methods.Full-data line-search steps are especially costly because each candidate requires evaluating the objective on the entire dataset.Source: Gradient Descent with Line Search · p. 549, §12.4.5 · p. 556, §12.5 · p. 557

12.3.4 Summary

  • For nonconvex deep-learning problems, optimality guarantees are generally unavailable, and Newton’s method requires adjustment despite being faster once properly started on convex problems.Gradient descent can also become trapped in local minima, while learning-rate selection becomes complicated in high dimensions.Source: §12.3.4 · p. 549, §12.4.5 · p. 556

12.5.1 Vectorization and Caches

  • Performance in Gigaflops rises from 0.013 for elementwise multiplication to 0.159 column-wise and 3.200 for full matrix multiplication.Vectorization reduces framework overhead and improves memory locality and caching; blocked operations provide a practical alternative when matrices do not fit in cache.Source: §12.5.1 · pp. 558, 560

12.5.2 Minibatches

  • Minibatch multiplication reaches 1.258 Gigaflops, essentially matching full-matrix efficiency, while averaging preserves expected gradients and reduces standard deviation by b^-1/2.Larger batches eventually provide diminishing variance reduction and must remain small enough to fit GPU memory; reduced variance can also weaken batch-normalization noise benefits.Source: §12.5.2 · p. 561

12.5.3 Reading the Dataset

  • The dataset example uses the first 1,500 whitened NASA aircraft-wing-noise observations to compare optimization algorithms.Whitening removes the mean and rescales each coordinate's variance to 1.Source: §12.5.3 · pp. 561–562

12.5.5 Concise Implementation

  • Using Gluon to repeat the batch-size-10 experiment shows identical behavior, yielding a loss of 0.242 at 0.128 sec/epoch.The concise implementation uses a generic Trainer-based training function for the optimization algorithms in the chapter.Source: §12.5.5 · pp. 565–566

12.5.6 Summary

  • The section summarizes vectorization's efficiency benefits and the trade-off between stochastic-gradient statistical efficiency and large-batch computational efficiency.These conclusions reflect reduced framework overhead, improved memory locality and caching, and the competing effects of gradient noise and processing cost.Source: §12.5.6 · p. 566
  • Minibatch stochastic gradient descent generally reaches lower risk faster in clock time than stochastic gradient descent and gradient descent.It processes randomly permuted data in batches and benefits from both computational and statistical efficiency.Source: §12.5.6 · pp. 566–567

12.5.7 Exercises

  • The exercises ask readers to vary minibatch size, learning-rate schedules, sampling procedures, momentum parameters, and quadratic eigenvalues to analyze convergence behavior.They include reducing the learning rate tenfold each epoch and comparing minibatch sampling with sampling with replacement.Source: §12.5.7 · p. 567, §12.6.5 · p. 578

An Ill-conditioned Problem

  • In ill-conditioned objectives, gradient descent must trade off stability in steep directions against slow progress in flat directions, whereas momentum aggregates aligned and oscillating gradients.With learning rate 0.6, progress in x1 improves but the overall solution quality worsens; momentum with β = 0.25 barely converges yet outperforms divergence without momentum.Source: An Ill-conditioned Problem · p. 570, The Momentum Method · pp. 570–571

Implementation from Scratch

  • With learning rate 0.005 and momentum 0.9, the scratch implementation reaches loss 0.247 at 0.152 sec/epoch, while the concise implementation reaches loss 0.247 at 0.142 sec/epoch.Momentum 0.9 corresponds to an effective sample size of 10; reducing the learning rate from 0.01 to 0.005 improves convergence behavior.Source: Implementation from Scratch · pp. 573–574, Concise Implementation · p. 575

Quadratic Convex Functions

  • Gradient descent and momentum on convex quadratic objectives decompose into independent coordinate-wise optimization along the quadratic matrix’s eigenvector directions.Diagonalizing the positive-definite matrix explains why optimization does not mix different eigenspaces.Source: Quadratic Convex Functions · p. 576

Scalar Functions

  • For scalar quadratic optimization, gradient descent converges exponentially when |1 −ηλ| < 1, improves until ηλ = 1, and diverges when ηλ > 2.Momentum enlarges the convergence range to 0 < ηλ < 2 + 2β, compared with 0 < ηλ < 2 for gradient descent.Source: Scalar Functions · pp. 576–577

12.6.4 Summary

  • Momentum replaces instantaneous gradients with leaky averages, accelerating convergence and preventing optimization stalling, especially for noisy stochastic gradients.The effective gradient direction averages past updates; implementation requires storing an auxiliary velocity state.Source: Leaky Averages · p. 569, §12.6.4 · p. 577

Effective Sample Weight

  • Momentum’s effective number of gradients is 1/(1−β), because past gradients receive exponentially decreasing weights; larger β therefore produces longer-range averaging.The method uses the same learning-rate update structure while replacing the descent direction with a weighted history of gradients.Source: Leaky Averages · p. 568, Effective Sample Weight · p. 572, §12.6.4 · p. 577

12.7.2 Preconditioning

  • Adagrad uses gradient magnitudes as a cheap proxy for unavailable Hessian diagonals, mitigating distortion from uneven curvature without computing the infeasible O(d2) second derivative.Diagonal preconditioning can reduce condition-number distortion, while exact eigenvalue and Hessian calculations are often impractical in deep learning.Source: §12.7.2 · p. 580, §12.7.6 · p. 584

12.7.4 Implementation from Scratch

  • Adagrad maintains one state variable per parameter coordinate, adding little cost relative to SGD because loss and gradient computation dominate optimization overhead.Implementations update each parameter using its accumulated squared gradient and an additive ϵ for numerical stability.Source: §12.7.3 · p. 580, §12.7.4 · p. 582

12.7.3 The Algorithm

  • Adagrad’s accumulated squared gradients produce an O(t−1/2) per-coordinate learning-rate schedule, causing progressively smaller updates and sometimes overly aggressive decay in deep learning.In a quadratic example, increasing the learning rate from 0.4 to 2 produced better behavior, indicating that the default decrease can be too aggressive.Source: §12.7.3 · pp. 581–582, §12.7.6 · p. 584

12.7 Adagrad

  • Adagrad is particularly effective for sparse features because infrequently occurring coordinates receive slower learning-rate decay, while large-gradient coordinates are scaled down more strongly.It replaces feature counts with accumulated squared gradients, automatically adapting to gradient magnitude. [Duchi et al. (2011)]Source: §12.7.1 · pp. 578–579, §12.7.6 · p. 584

12.8 RMSProp

  • The passages identify coordinate-wise adaptivity as highly desirable as a preconditioner and state that, with parameter γ > 0, keeping all other parts unchanged yields RMSProp.Source: §12.8 · pp. 584–585

12.8.2 Implementation from Scratch

  • RMSProp avoids Adagrad’s late-stage stagnation because its independently controlled learning rate does not continuously decay as the cumulative state grows.In the quadratic experiment, variables continued moving during later iterations rather than becoming very slow under Adagrad’s learning rate of 0.4.Source: §12.8.2 · pp. 585–586

12.8 RMSProp

  • RMSProp uses exponentially weighted squared gradients to adapt per-coordinate scaling, with γ controlling history length and the learning rate still scheduled manually.The example sets learning rate 0.01 and γ = 0.9, averaging approximately the previous 10 squared gradients.Source: §12.8.2 · p. 587, §12.8.4 · p. 588

12.9 Adadelta

  • Adadelta replaces a fixed learning-rate parameter with parameter-change calibration, maintaining leaky averages of gradient and parameter-change second moments.With ρ = 0.9, the implementation reports loss 0.244 and 0.221 sec/epoch.Source: §12.9.1 · p. 589, §12.9.2 · p. 590, §12.9.3 · p. 591

12.10 Adam

  • The section frames adaptive optimization as combining gradient history, minibatch vectorization, per-coordinate preconditioning, and learning-rate adjustment for efficient deep-learning optimization.These techniques are introduced as complementary mechanisms that progressively improve optimization efficiency and convergence behavior.Source: §12.10 · p. 592
  • Adam combines momentum, per-coordinate scaling, bias correction, and an explicit learning rate into a robust optimization algorithm, but it can diverge when variance control is poor.Adam commonly uses β1 = 0.9, β2 = 0.999, and ε = 10^-6; Reddi et al. (2019) identify divergence cases, motivating Yogi.Source: §12.10 · p. 592, §12.10.1 · pp. 592–593

12.10.2 Implementation

  • The from-scratch Adam implementation uses bias-corrected exponential averages with η = 0.01 and reaches loss 0.246 at 0.224 sec/epoch.The concise implementation reaches loss 0.243 at 0.182 sec/epoch under the same learning-rate setting.Source: §12.10.2 · p. 594

12.10.3 Yogi

  • Yogi modifies Adam’s second-moment update to address blow-up, high variance, and sparse updates, while larger minibatches can also improve convergence.The chapter reports Adam’s failure even in convex settings and presents Yogi as an alternative estimate for the second moment.Source: §12.10.3 · pp. 595–596

4. Try to construct a case for which Adam diverges and Yogi converges? · 4. How long should warmup last? · 3. How could you measure the cache sizes on a CPU?

This section develops practical methods for optimizing deep-learning training and execution, spanning learning-rate scheduling, programming models, hardware-aware computation, memory and accelerator use, distributed and multi-GPU training, image augmentation, and transfer learning. It also introduces core object-detection representations and procedures, including bounding boxes, anchor boxes, matching, decoding, and non-maximum suppression.

12.11.1 Toy Problem

  • With default SGD at learning rate 0.3 for 30 iterations, training reached 0.926 accuracy while test accuracy reached 0.895, with the widening gap indicating overfitting.Training accuracy continued increasing while test accuracy stalled.Source: §12.11.1 · pp. 598–599

12.11.2 Schedulers

  • A square-root scheduler produced train loss 0.272, train acc 0.901, and test acc 0.883, with smoother curves and less overfitting than the default schedule.The text notes that the theoretical reason for reduced overfitting remains unresolved.Source: §12.11.2 · p. 600

12.11.3 Policies

Cosine Scheduler

  • Cosine scheduling produced train loss 0.186, train acc 0.932, and test acc 0.901, although improvements are not guaranteed across problems.The schedule decreases toward a target rate and then remains fixed after its maximum update step.Source: Cosine Scheduler · pp. 603–604

Warmup

  • Warmup increases the learning rate gradually before cooling it, preventing early divergence caused by large initial steps and improving early convergence.Warmup can be applied with any scheduler and may limit parameter divergence in very deep networks.Source: Warmup · pp. 604–605, §12.11.4 · p. 605

12.11.4 Summary

  • Decreasing the learning rate during training can improve accuracy and reduce overfitting, while piecewise reductions after progress plateaus help converge efficiently before lowering parameter variance.The summary identifies these effects as practical conclusions of the chapter.Source: §12.11.4 · p. 605

13.1.1 Symbolic Programming

  • Symbolic programming defines operations, compiles them, and then executes the compiled program, enabling interpreter bypass, code rewriting, and memory optimization.It can rewrite print((1 + 2) + (3 + 4)) into print(10) and avoid allocating no-longer-needed variables.Source: §13.1.1 · p. 608

Serialization

  • Compiling an MLP with torch.jit.script optimizes its computation while preserving the model’s computation result, and compiled models can be serialized independently of the front-end language.Serialization supports deployment to other devices and use from other front-end programming languages.Source: §13.1.3 · p. 610, Serialization · pp. 611–612

Acceleration by Hybridization

  • Without torchscript: 18.1045 sec; with torchscript: 20.5523 sec for 1,000 evaluations, while the text describes symbolic programming as improving computing performance.The benchmark compares an nn.Sequential model before and after torch.jit.script.Source: Acceleration by Hybridization · p. 611

13.1.4 Summary

  • Imperative programming eases model design and debugging, whereas symbolic programming improves efficiency and portability by optimizing code during compilation and enabling non-Python deployment.PyTorch’s TorchScript combines imperative development and debugging with conversion to symbolic programs for deployment.Source: §13.1.1 · p. 609, §13.1.2 · p. 610, §13.1.4 · p. 612

13.2.1 Asynchrony via Backend

  • PyTorch GPU operations are asynchronous: the Python frontend queues work for a C++ backend, which tracks dependencies and executes independent tasks while the frontend continues.Users should synchronize regularly, such as once per minibatch, because overfilling the queue can increase memory consumption.Source: §13.2 · p. 612, §13.2.1 · pp. 613–615, §13.2.4 · p. 615
  • NumPy took 1.4568 sec and PyTorch 0.0009 sec for the toy workload, but the apparent gap reflects CPU-versus-GPU execution and asynchronous backend scheduling.Forcing torch.cuda.synchronize reveals the actual completion behavior before results are returned.Source: §13.2.1 · p. 613

13.3.1 Parallel Computation on GPUs

  • Automatic scheduling reduced two-GPU execution to 0.4630 sec, below the separate GPU times of 0.4628 sec and 0.4522 sec.Removing synchronization allowed the framework to parallelize computation across both devices automatically.Source: §13.3.1 · p. 617

13.3.2 Parallel Computation and Communication

  • Overlapping GPU computation with CPU transfers reduced total time to 2.1902 sec versus 0.4649 sec for GPU computation and 2.5135 sec for copying alone.Nonblocking transfers exploit PCI-Express bandwidth while later GPU results are still being computed.Source: §13.3.2 · p. 618

13.3.3 Summary

  • The backend improves performance through automatic parallel computation and communication across devices and resources such as PCI Express, storage, and networks.Modern systems can use multiple CPUs and GPUs asynchronously, while communication resources operate in parallel.Source: §13.3.3 · p. 618

13.3.4 Exercises

  • The exercises ask readers to test automatic operator parallelism, CPU/GPU computation and communication, debugger-based efficiency, and correctness under complex dependencies.They also examine when parallelization helps for small operators and how to verify performance experimentally.Source: §13.3.4 · p. 619

13.4 Hardware

  • Hardware-aware design matters because bottlenecks in data movement or computation can change training time by an order of magnitude, from about one week to three months.The section surveys computers, CPUs, GPUs, and interconnects to explain why algorithms differ in efficiency.Source: §13.4 · p. 620, §13.4.1 · p. 621

13.4.1 Computers

  • Computer performance depends on coordinating CPUs, RAM, networks, PCIe-connected GPUs, and durable storage so processors do not become starved by data movement.Most peripherals connect through PCIe, while CPU-attached memory can provide up to 100 GB/s bandwidth.Source: §13.4.1 · pp. 620–621

13.4.2 Memory

  • RAM access favors burst operations: setup takes approximately 100 ns, subsequent transfers 0.2 ns, and the first read is 500 times more expensive.Multiple banks can increase random-read capacity up to 4 times, but burst reads remain 4 times faster and data should align to 64-bit boundaries.Source: §13.4.2 · p. 622

13.4.3 Storage · Hard Disk Drives · Solid State Drives · Cloud Storage · 13.4.4 CPUs · Microarchitecture · Vectorization

  • Storage tradeoffs span HDD latency, SSD throughput and write limitations, cloud-configurable IOPs, and CPU execution mechanisms including caches, prediction, and SIMD vector units.HDDs provide approximately 100 IOPs, SSDs 100,000 to 500,000 IOPs, and vector units enable multiple operations per cycle, although GPUs remain much faster.Source: Hard Disk Drives · p. 623, Solid State Drives · p. 623, Cloud Storage · p. 624

Cache

  • Caches prevent processor starvation caused by CPU compute demand exceeding memory bandwidth, but larger caches consume chip area and cache misses such as false sharing can stall multiple processors.A four-core 2 GHz CPU with 256-bit AVX2 can consume 256×10^9 bytes per second, versus 20–40 GB/s from memory.Source: Cache · pp. 626–627

13.4.5 GPUs and other Accelerators

  • GPU and accelerator designs combine many cores, vector or tensor units, and specialized memory systems; training needs more memory, precision, and processing power than inference.Turing T4 GPUs target inference, whereas V100 GPUs are preferable for training; tensor cores accelerate small 4 × 4 to 16 × 16 matrix operations.Source: §13.4.5 · pp. 628–629

13.4.6 Networks and Buses

  • Interconnect choice balances bandwidth, latency, cost, distance, and flexibility: PCIe and NVLink provide high bandwidth, while Ethernet offers cheaper, longer-distance connectivity with greater overhead.PCIe 4.0 can provide up to 32 GB/s in a 16-lane slot, while NVLink offers up to 300 Gbit/s per link.Source: §13.4.6 · p. 630

13.4.8 Summary

  • The section recommends matching algorithms to hardware through vectorization, cache-fitting memory footprints, large transfers, suitable numerical precision, profiling, and awareness of training–inference trade-offs.Small data types can cause numerical overflow during training, while sparse data and interrupts remain weaknesses of GPUs optimized for high-bandwidth burst access.Source: §13.4.5 · p. 629, §13.4.8 · p. 632

13.5.1 Splitting the Problem

  • Data parallelism is generally the most convenient multi-GPU strategy when GPUs have sufficiently large memory, whereas network and layerwise partitioning incur difficult synchronization, bandwidth, or complexity costs.Data parallelism synchronizes after each minibatch and does not support larger models, but current GPU memory resolves this limitation in most cases.Source: §13.5.1 · pp. 634–635

13.5.2 Data Parallelism

  • Data-parallel training splits each minibatch across GPUs, computes local losses and gradients, aggregates and redistributes gradients, then updates synchronized complete model copies.Each GPU maintains the full model parameters independently while parameter values remain identical across GPUs.Source: §13.5.2 · p. 636

13.5.3 A Toy Network

  • The worked examples use LeNet to expose synchronization details and a modified ResNet-18 to demonstrate concise, higher-performance multi-GPU training.The manual example initializes a toy network, while the concise example modifies the initial convolution and removes max pooling for tiny images.Source: §13.5.3 · p. 637, §13.6.1 · p. 642

13.5.4 Data Synchronization

  • Manual multi-GPU training requires distributing parameters, splitting data and labels, aggregating gradients with allreduce, and updating replicated parameters.The computational graph executes devices in parallel when minibatch shards have no cross-device dependencies.Source: §13.5.4 · pp. 637–638, §13.5.5 · p. 638, §13.5.6 · p. 639

13.5.7 Summary

  • Data-parallel training is straightforward, but larger GPU counts increase minibatch size and can require learning-rate scaling and per-device batch-normalization adjustments.The section identifies data parallelism as simpler than splitting networks across layers, which requires tightly coordinated transfers.Source: §13.5.2 · p. 636, §13.5.7 · p. 641
  • Data parallelism splits minibatches across GPUs, runs forward and backward passes independently, aggregates gradients, and broadcasts results back to all devices.Efficient use generally increases the effective minibatch size with the number of GPUs and may require a larger learning rate.Source: §13.5.2 · p. 636, §13.5.7 · p. 641

13.6 Concise Implementation for Multiple GPUs

  • The concise implementation uses a modified ResNet-18 with DataParallel to automate device placement, gradient aggregation, parameter updates, and parallel accuracy evaluation.Networks must be initialized on each device before accessing parameters there.Source: §13.6.1 · p. 642, §13.6.3 · p. 643, §13.6.4 · p. 645

13.6.3 Training

  • ResNet-18 training benefits meaningfully from two GPUs because larger computation makes parameter-synchronization overhead less relevant.The section contrasts this with LeNet, where two GPUs preserve test accuracy but provide no meaningful speedup because the model and dataset are small and Python overhead is significant.Source: §13.5.6 · pp. 640–641, §13.6.3 · p. 644

13.7 Parameter Servers

  • Parameter-server systems become necessary as distributed training spans GPUs, servers, racks, and heterogeneous interconnects with substantially different bandwidths.The section motivates abstraction because statistical modelers should not need to master networking and systems details.Source: §13.7 · pp. 645–646

13.7.1 Data-Parallel Training

  • 15–80 ms: synchronizing the same gradients can vary substantially with the communication strategy, from split aggregation across GPUs to CPU aggregation.For 160 MB gradients, split aggregation takes 15 ms, GPU aggregation 60 ms, and CPU aggregation 80 ms.Source: §13.7.1 · p. 647, Figure 13.7.3 · p. 648

13.7.2 Ring Synchronization

  • Two rings are optimal for the stated p3 and DGX-2 connectivity, exploiting heterogeneous NVLink bandwidth through a nontrivial synchronization protocol.One ring uses double NVLink bandwidth while the other uses regular bandwidth.Source: §13.7.2 · pp. 648–649, Figure 13.7.5 · p. 649
  • ≈6 ms: ring synchronization across eight V100 GPUs can outperform PCIe-based synchronization for 160 MB of gradients.The ring strategy chunks gradients so communication proceeds concurrently; practical times can be worse when frameworks fail to assemble large burst transfers.Source: §13.7.2 · p. 649, Figure 13.7.6 · p. 650

13.7.3 Multi-Machine Training

  • O(m/n): distributing parameters across n servers reduces the parameter-server update cost from a single-server O(m) bottleneck toward constant scaling as workers increase.In practice, the same machines serve as workers and servers, but coordinating them without unreasonable delays remains nontrivial.Source: §13.7.3 · p. 651, Figure 13.7.8 · p. 652

13.7.4 Key–Value Stores

  • Key–value stores expose distributed optimization through push and pull operations, decoupling statistical modeling from synchronization engineering.Push accumulates keyed gradients in common storage, while pull retrieves the aggregated values.Source: §13.7.4 · pp. 652–653

13.7.5 Summary

  • Synchronization must adapt to server connectivity; ring synchronization may suit p3 and DGX-2 systems, while hierarchical synchronization helps scale parameter-server bandwidth.The summary emphasizes that topology-specific choices can materially change synchronization time.Source: §13.7.5 · p. 653

14.1 Image Augmentation

  • Image augmentation creates varied training examples and can improve generalization by reducing reliance on attributes such as object position, color, and scale.The section motivates augmentation as especially useful when images contain meaningful variation in object appearance and placement.Source: §14.1 · pp. 654–655, Flipping and Cropping · p. 656

14.1.1 Common Image Augmentation Methods

Multi-GPU Training

  • Multi-GPU ResNet-18 training with random left-right image augmentation on CIFAR-10 reached test acc 0.840 and processed 3322.3 examples/sec.The training used Adam and evaluated augmented training data through the multi-GPU training function.Source: Multi-GPU Training · p. 661

14.2 Fine-Tuning

  • Transfer learning addresses small-data overfitting and labeling costs by transferring general image features such as edges, textures, shapes, and object composition from a source dataset.Image augmentation generates random training images to improve generalization, while random augmentation is normally omitted during prediction.Source: §14.1.3 · p. 661, §14.2 · p. 663

14.2.1 Steps

  • Fine-tuning copies pretrained source-model parameters except the output layer, replaces that layer for target classes, and trains it from scratch while adapting the remaining layers.When the target dataset is much smaller than the source dataset, this approach improves generalization.Source: §14.2.1 · p. 663

14.2.2 Hot Dog Recognition

  • The hot-dog case uses 1400 positive and 1400 negative images, with 1000 images per class for training and the remainder for testing.Training uses random resized crops and horizontal flips, whereas testing uses resizing, center cropping, and channel standardization.Source: Reading the Dataset · pp. 664–665

Defining and Initializing the Model

  • The target model replaces ResNet-18’s 1000-class ImageNet output with a two-class layer, using pretrained features and a learning rate of 10η for the new layer.Pretrained layers use a small base learning rate because their parameters are already effective, while the randomly initialized output layer needs a larger rate.Source: Defining and Initializing the Model · p. 666, Fine-Tuning the Model · p. 667

Fine-Tuning the Model

  • test acc 0.926 versus 0.871, while fine-tuning also achieved train acc 0.918 versus 0.819 against training ResNet-18 from scratch.The fine-tuned model performed better for the same epoch because its pretrained parameter values were more effective.Source: Fine-Tuning the Model · pp. 667–668

14.3 Object Detection and Bounding Boxes

  • Object detection identifies multiple objects and their positions, typically representing each position with a rectangular bounding box.Applications include self-driving, robotics, and security systems that must detect and localize objects.Source: §14.3 · p. 669, §14.3.2 · p. 672

14.3.1 Bounding Boxes

  • Bounding boxes use either upper-left and lower-right corners or center coordinates with width and height, with conversions verified computationally.The conversion functions operate on tensors shaped (n, 4), and round-trip verification returns True values.Source: §14.3.1 · pp. 670–671, §14.3.2 · p. 672

14.4 Anchor Boxes

  • Anchor boxes generate candidate regions at every pixel using multiple scales and aspect ratios, while restricting combinations to produce wh(n + m −1) boxes instead of whnm.For the demonstrated input, the returned tensor has shape torch.Size([1, 2042040, 4]).Source: §14.4 · p. 672, §14.4.1 · pp. 672–674

14.4.2 Intersection over Union (IoU)

  • Intersection over union measures bounding-box similarity as intersection area divided by union area, ranging from 0 for no overlap to 1 for equality.The box_iou procedure computes pairwise IoU across two lists of anchor or bounding boxes.Source: §14.4.2 · pp. 675–676, Figure 14.4.1 · p. 676

14.4.3 Labeling Anchor Boxes in Training Data

  • Training assigns each anchor a class and offset label, then prediction adjusts anchors using offsets and outputs boxes satisfying selected criteria.Anchor labels refer to the class and location of an assigned ground-truth bounding box.Source: §14.4.3 · pp. 676–677

Assigning Ground-Truth Bounding Boxes to Anchor Boxes

  • Ground-truth assignment repeatedly selects the largest remaining IoU, discards its anchor row and ground-truth column, then threshold-matches remaining anchors.This procedure assigns every ground-truth box to an anchor before processing the remaining na −nb anchors.Source: Assigning Ground-Truth Bounding Boxes… · p. 677
  • Anchor boxes are assigned to ground-truth boxes using IoU thresholds, while a greedy matching step ensures every ground-truth box receives an anchor.Anchors whose maximum IoU reaches 0.5 are assigned directly; remaining ground-truth boxes are matched iteratively to the largest available IoU.Source: Assigning Ground-Truth Bounding Boxes… · p. 678

Labeling Classes and Offsets

  • Assigned anchors receive object classes and transformed center-and-size offsets, whereas unassigned anchors receive the background class.The offset transformation uses default constants µx = µy = µw = µh = 0, σx = σy = 0.1, and σw = σh = 0.2; background offsets are masked from the objective.Source: Labeling Classes and Offsets · pp. 678–679, An Example · pp. 681–682

An Example

  • In the worked example, the five anchors receive class labels [0, 1, 2, 0, 2], with anchors below the 0.5 IoU threshold labeled background.A4 matches the cat, A1 matches the dog, A2 matches the cat, and A0 and A3 remain background.Source: An Example · p. 681

14.4.4 Predicting Bounding Boxes with Non-Maximum Suppression

  • Non-maximum suppression sorts predictions by confidence, keeps the highest-scoring box, and removes remaining boxes whose IoU exceeds the suppression threshold.The procedure repeats until all retained boxes have pairwise IoU below the threshold, producing a less redundant final output.Source: §14.4.4 · pp. 682–683
  • The detection output has shape (batch size, number of anchor boxes, 6), encoding class, confidence, and four corner coordinates, with -1 indicating background or suppression.The example applies NMS with threshold 0.5 and removes class -1 boxes before displaying the final predictions.Source: §14.4.4 · pp. 684–685

14.4.5 Summary

  • The summary emphasizes generating varied anchor shapes, labeling each anchor with a class and offset, and using NMS to remove similar predictions.The exercises ask readers to vary anchor sizes and ratios, construct boxes with IoU 0.5, modify anchors, and consider Soft-NMS (Bodla et al., 2017).Source: §14.4.5 · pp. 685–686, §14.4.6 · p. 686

5. Rather than being hand-crafted, can non-maximum suppression be learned? · 2. Is it efficient to use matrix multiplications to implement convolutions? Why?

This section covers object detection and semantic segmentation, introducing multiscale and region-based detectors, non-maximum suppression, anchor boxes, and pixel-level prediction. It also develops practical pipelines and models, including TinySSD, transposed convolution, and fully convolutional networks, with attention to training, inference, and dataset handling.

14.5.1 Multiscale Anchor Boxes

  • Multiscale anchor boxes reduce excessive proposals by sampling more small-object regions and fewer large-object regions across feature-map scales.Anchor-box centers are uniformly distributed, while larger scales use reduced feature-map dimensions and larger boxes.Source: §14.5.1 · pp. 686–688

14.5.2 Multiscale Detection

  • At each scale, units sharing a receptive field predict classes and offsets for nearby anchor boxes, while different layers support different object sizes.This leverages layerwise image representations at multiple levels for multiscale detection.Source: §14.5.2 · p. 689

14.6 The Object Detection Dataset

  • The banana dataset contains 1,000 training and 100 validation images generated from bananas with varied rotations, sizes, and positions, each labeled by bounding boxes.Object-detection labels include class and upper-left and lower-right bounding-box coordinates; minibatches pad images to a common maximum box count.Source: §14.6 · p. 690, §14.6.2 · p. 692, §14.6.3 · p. 693

14.7.1 Model

  • SSD combines a base CNN with successive multiscale feature-map blocks that shrink spatial dimensions, enlarge receptive fields, and generate varying-size anchor boxes.The base network can be VGG truncated before classification or ResNet, and larger intermediate maps provide more anchors for small objects.Source: §14.7 · p. 694, §14.7.1 · p. 694
  • SSD uses multiscale feature maps with different receptive fields to detect objects of varying sizes by predicting anchor-box classes and offsets.Feature maps closer to the output have wider receptive fields and are suited to detecting fewer, larger objects.Source: §14.5.2 · p. 689, §14.7.1 · p. 694

Class Prediction Layer

  • SSD replaces costly fully connected classification with convolutional prediction layers whose channels encode classes for every anchor at each spatial position.For q object classes and a anchors, the class layer uses a(q + 1) output channels, while box prediction requires four offsets per anchor.Source: Class Prediction Layer · p. 695, Bounding Box Prediction Layer · p. 696

Concatenating Predictions for Multiple Scales

  • Multiscale prediction tensors are reshaped to (batch size, height × width × number of channels) and concatenated along their prediction dimension despite differing spatial and channel sizes.The channel dimension is moved inward before concatenation, preserving predictions from feature maps at different scales.Source: Concatenating Predictions for Multiple… · pp. 696–697

Downsampling Block

  • Each downsampling block uses two 3 × 3 convolutions followed by 2 × 2 max-pooling, halving spatial dimensions while enlarging each output unit’s receptive field to 6 × 6.The 3 × 3 convolutions preserve feature-map size, whereas max-pooling performs the downsampling.Source: Downsampling Block · p. 697

Base Network Block

  • A 256 × 256 input becomes 32 × 32 feature maps with 64 channels after three downsampling blocks that double channel count at each stage.The base network produces output shape torch.Size([2, 64, 32, 32]) for a batch of two images.Source: Base Network Block · p. 698

The Complete Model

  • 5444 anchor boxes per image are generated across five feature-map scales, with outputs shaped as anchors [1, 5444, 4], class predictions [32, 5444, 2], and box predictions [32, 21776].The model uses a base network, three downsampling blocks, and global pooling; each scale predicts anchor classes and offsets.Source: The Complete Model · pp. 698, 700

Defining Loss and Evaluation Functions

Training the Model

  • 3.31e-03 class error and 3.11e-03 bounding-box MAE are reported at 3535.8 examples/sec on cuda:0 during training.Inference applies predicted offsets to anchors, then uses non-maximum suppression and displays detections with confidence at least 0.9.Source: Training the Model · p. 702, §14.7.3 · p. 703

14.7.5 Exercises

  • The exercises propose smooth ℓ1 loss for offsets and focal loss for classification, with larger γ reducing the relative loss of well-classified examples.Smooth ℓ1 becomes similar to ℓ1 for very large σ, while smaller σ makes it smoother; focal loss emphasizes difficult examples.Source: §14.7.5 · pp. 703–705
  • The exercises suggest improving single-shot detection by resizing images for small objects, downsampling negative anchors, weighting loss terms, and using alternative evaluation methods.They also ask readers to consider framing detection as a single regression problem, as in YOLO (Redmon et al., 2016), and compare it with the methods presented.Source: §14.7.5 · pp. 705–706, §14.8.6 · p. 712

14.8 Region-based CNNs (R-CNNs)

  • The R-CNN family progresses from region-wise CNN processing to shared image computation, learned proposals, and pixel-level mask prediction.The sequence comprises R-CNN, fast R-CNN, faster R-CNN (Ren et al., 2015), and mask R-CNN.Source: §14.8 · p. 706, §14.8.5 · p. 711

14.8.1 R-CNNs

  • R-CNN uses selective search to generate many multiscale proposals, extracts CNN features independently for each, then trains class-specific SVMs and a linear bounding-box regressor.Its thousands of CNN forward propagations repeat computation across overlapping regions, making the model too slow for widespread real-world use.Source: §14.8.1 · pp. 706–707

14.8.2 Fast R-CNN

  • Fast R-CNN shares computation by applying a trainable CNN once to the entire image, then using RoI pooling to produce same-shaped features for proposals.RoI pooling divides each region into a fixed grid and takes each subwindow’s maximum, enabling class and bounding-box prediction for all proposals.Source: §14.8.2 · pp. 707–708

14.8.3 Faster R-CNN

  • The faster R-CNN replaces selective search with a jointly trained region proposal network, reducing proposals while maintaining detection accuracy.The network predicts anchor-box classes and bounding boxes, removes overlapping predictions with non-maximum suppression, and learns proposals end to end.Source: §14.8.3 · pp. 709–710, §14.8.5 · p. 711

14.8.4 Mask R-CNN

  • Mask R-CNN extends faster R-CNN with RoI alignment and an additional fully convolutional network for pixel-level object masks.RoI alignment uses bilinear interpolation to preserve spatial information and supports class, bounding-box, and pixel-level predictions from detailed labels.Source: §14.8.4 · pp. 710–711, §14.8.5 · p. 711

14.9 Semantic Segmentation and the Dataset

  • Semantic segmentation labels every pixel by semantic class, producing finer-grained region boundaries than object-detection bounding boxes.Image segmentation instead groups pixels without requiring semantic labels, whereas instance segmentation distinguishes the pixel-level regions of different object instances.Source: §14.9 · p. 712, §14.9.1 · pp. 712–713

14.9.2 The Pascal VOC2012 Semantic Segmentation Dataset

  • Pascal VOC2012 stores training and test splits, input JPEG images, and same-sized color-coded segmentation labels whose colors identify semantic classes.The dataset archive is about 2 GB, and the section provides code to read images and labels into memory.Source: §14.9.2 · pp. 713–714

Data Preprocessing

Custom Semantic Segmentation Dataset Class

  • VOCSegDataset loads indexed pixel labels, normalizes RGB inputs, filters undersized examples, and provides indexed image-label pairs for training and testing.The dataset maps VOC RGB colors to class indices and standardizes channels using the specified normalization transform.Source: Custom Semantic Segmentation Dataset… · pp. 716–717

Reading the Dataset

14.10.1 Basic Operation

  • Basic transposed convolution multiplies each input element by the kernel, places the resulting patch at its corresponding location, and sums overlapping intermediate tensors.With stride 1 and no padding, an nh × nw input and kh × kw kernel produce an (nh+kh−1) × (nw+kw−1) output.Source: §14.10.1 · pp. 719–720, Figure 14.10.1 · p. 720

14.10 Transposed Convolution

  • Transposed convolution broadcasts input elements through a kernel, producing an output larger than the input for semantic-segmentation upsampling.It reverses convolutional downsampling operations and helps restore spatial dimensions for pixel-level classification.Source: §14.10 · p. 719, §14.10.4 · p. 724

14.10.2 Padding, Strides, and Multiple Channels

  • With matching convolution hyperparameters and output channels set to the input-channel count, transposed convolution restores the original input shape.The stated shape relationship holds even though the values need not be identical.Source: §14.10.2 · p. 722, §14.10.4 · p. 724

14.10.3 Connection to Matrix Transposition

  • Convolutions can be implemented as sparse matrix multiplications, while transposed convolution uses the weight matrix transpose and exchanges convolution forward and backward operations.For convolution y = Wx, backpropagation uses W⊤, so transposed convolution uses W⊤ forward and W backward.Source: §14.10.3 · pp. 722–723, §14.10.4 · p. 724

14.11.1 The Model

  • A fully convolutional network extracts image features with a CNN, maps channels to classes with a 1 × 1 convolution, and restores input height and width with transposed convolution.Its output has the same spatial dimensions as the input, with each output channel representing predicted classes for the corresponding input pixel.Source: §14.11.1 · p. 725, §14.11.6 · p. 730
  • For 320 × 480 inputs, the feature extractor reduces spatial dimensions to 10 × 15, while a stride-32 transposed convolution restores them to the input resolution.The model uses a pretrained ResNet-18 without its final global average pooling and fully connected layers, and predicts 21 Pascal VOC2012 classes.Source: §14.11.1 · pp. 725–726

14.11.2 Initializing Transposed Convolutional Layers

  • Bilinear interpolation initializes the transposed convolution, doubling image height and width while producing an image that looks the same as the original apart from scale.The example changes the shape from [561, 728, 3] to [1122, 1456, 3].Source: §14.11.2 · p. 728

14.11.3 Reading the Dataset

  • The semantic-segmentation dataset is randomly cropped to 320 × 480 outputs so both dimensions are divisible by 32; 1114 training and 1078 validation examples are read.The dimensions align with the stride of the transposed convolution used for upsampling.Source: §14.11.3 · pp. 728–729

14.11.4 Training

  • The trained fully convolutional network achieved loss 0.448, train acc 0.862, and test acc 0.852 at 195.7 examples/sec on GPUs.Training computes loss and pixel accuracy from the output channel corresponding to each pixel’s predicted class.Source: §14.11.4 · p. 729

14.11.5 Prediction

  • Prediction standardizes input channels, converts pixel predictions back to dataset label colors, and handles dimensions not divisible by 32 by averaging overlapping rectangular crops.Each crop has dimensions that are integer multiples of 32, and averaged transposed-convolution outputs are passed to softmax.Source: §14.11.5 · pp. 729–730

14.11.6 Summary

  • The section summarizes fully convolutional networks as CNN feature extractors followed by 1 × 1 class-channel mapping and transposed-convolution upsampling initialized with bilinear interpolation.This design preserves one-to-one spatial correspondence between output predictions and input pixels.Source: §14.11 · p. 724, §14.11.6 · p. 730

2. Can you further improve the accuracy of the model by tuning the hyperparameters? … 1. How does the running time of code in this section changes if not using subsampling?

This section develops practical computer-vision workflows for image classification and neural style transfer, then introduces word representations through skip-gram, CBOW, negative sampling, hierarchical softmax, GloVe, and subword embeddings. It combines runnable training pipelines and exercises with techniques for data organization, augmentation, pretrained features, efficient embedding training, and handling rare or out-of-dictionary words.

14.12.1 Method

  • Style transfer updates only the synthesized image, using a frozen pretrained CNN to extract selected content and style features.VGG-19 is used, with a deeper layer for content features and the first convolutional layer of each block for style features.Source: §14.12.1 · p. 732, §14.12.4 · pp. 734–735, §14.12.6 · p. 738

14.12.3 Preprocessing and Postprocessing

  • Images are standardized for CNN input and restored afterward, with output pixels clipped to the printable range 0 to 1.The preprocessing pipeline resizes images, converts them to tensors, and normalizes RGB channels before postprocessing reverses those transformations.Source: §14.12.3 · p. 734

Loss Function

  • The loss combines content, style, and total variation terms to balance content retention, style transfer, and noise reduction.Content loss compares content-layer features, style loss compares Gram-matrix style representations, and total variation loss makes neighboring pixels more similar.Source: Style Loss · p. 736, Loss Function · p. 737, Total Variation Loss · p. 737

14.12.7 Training

  • Training initializes the synthesized image from the content image, extracts fixed target features beforehand, and optimizes the image with Adam while decaying the learning rate.The experiment resizes both images to 300 by 450 pixels and trains for 500 epochs with learning rate 0.3 and decay every 50 epochs.Source: §14.12.7 · pp. 738–739, §14.12.6 · p. 738
  • The synthesized image retains the content image’s scenery and objects while transferring the style image’s colors, blocks, and brush-stroke texture.The method demonstrates this behavior using a landscape content image and an autumn-oak oil painting as style image.Source: §14.12 · p. 732, §14.12.7 · p. 739

14.13 Image Classification (CIFAR-10) on Kaggle

  • The CIFAR-10 Kaggle exercise applies prior deep-learning knowledge to raw image files, organizing and transforming them into tensors for classification.The dataset has 50,000 training images and 300,000 test images, with 10,000 test images used for evaluation; images are 32×32 RGB PNGs across 10 categories.Source: §14.13 · p. 740, §14.13.1 · p. 741

Organizing the Dataset

  • The dataset is reorganized by reading CSV labels, splitting 10% of training examples into validation data, and grouping images by class for loading.For the complete dataset, the split yields 45,000 training and 5,000 validation images; the demo uses batch size 32, whereas the full dataset uses 128.Source: Organizing the Dataset · pp. 742–743

14.13.2 Image Augmentation

  • Image augmentation combats overfitting during training through random resized crops and horizontal flips, while testing uses standardization only for deterministic evaluation.The RGB channels are standardized using means [0.4914, 0.4822, 0.4465] and standard deviations [0.2023, 0.1994, 0.2010].Source: §14.13.2 · p. 744

14.13.3 Reading the Dataset

  • Validation and test loaders omit augmentation, and the final model is retrained on the combined training and validation sets before generating a Kaggle-compatible submission.csv.This procedure makes full use of labeled data for final prediction after hyperparameters have been selected.Source: §14.13.3 · p. 745, §14.13.7 · p. 747

14.13.4 Defining the Model

  • The CIFAR-10 workflow uses a ResNet-18 classifier, selects hyperparameters by validation performance, and applies scheduled optimization during training.The training function tracks loss, training accuracy, validation accuracy, and throughput while using data parallelism across available GPUs.Source: §14.13.4 · p. 745, §14.13.5 · pp. 745–746

14.13.6 Training and Validating the Model

  • 0.484 validation accuracy and 613.0 examples/sec were obtained with 0.727 training loss and 0.738 training accuracy in the 20-epoch demonstration.The setup used learning rate 2e-4, weight decay 5e-4, and a learning-rate schedule multiplying the rate by 0.9 every 4 epochs.Source: §14.13.6 · p. 746

14.13.8 Summary

  • The section concludes by proposing full-data CIFAR-10 experimentation with batch_size = 128, num_epochs = 100, lr = 0.1, lr_period = 50, and lr_decay = 0.1 to improve accuracy and ranking.It then introduces dog-breed identification, a 120-class ImageNet subset with 10,222 training and 10,357 test images.Source: §14.13.8 · p. 748, §14.14 · p. 748, §14.14.1 · p. 748

Organizing the Dataset

  • The dog-breed data workflow downloads a sample or full Kaggle dataset, splits training data into validation data, and organizes images into label-based folders.The validation ratio is set to 0.1, with batch size 32 for the demo and 128 for the full dataset.Source: Downloading the Dataset · p. 749, Organizing the Dataset · p. 750

14.14.2 Image Augmentation

  • Image augmentation adapts to larger, variable-size ImageNet images through random resized crops, horizontal flips, color changes, noise, and channel normalization, while prediction uses deterministic resizing and center cropping.Training crops are scaled to 224 x 224, whereas prediction resizes to 256 and center-crops a 224 x 224 region.Source: §14.14.2 · pp. 750–751

14.14.5 Defining the Training Function

  • Model selection and hyperparameter tuning use validation performance, while training updates only parameters that require gradients in the custom output network.The training function uses SGD with momentum 0.9, weight decay, and a StepLR schedule; the example multiplies the learning rate by 0.9 every 2 epochs.Source: §14.14.5 · p. 752, §14.14.6 · p. 753

14.14.7 Classifying the Testing Set and Submitting Results on Kaggle

  • After validation, all labeled images, including the validation set, train the custom output network before softmax predictions are written to a Kaggle submission.csv file.The submission contains an identifier column followed by probabilities for the 120 dog-breed classes.Source: §14.14.7 · pp. 754–755

14.14.4 Fine-Tuning a Pretrained Model

  • A pretrained ResNet-34 extracts ImageNet features while only a small custom output network is trained, reducing computational time and memory cost.The pretrained model’s output-layer input becomes the custom network’s input, and the feature layers remain frozen.Source: §14.14.4 · pp. 751–752, §14.14.8 · p. 755

15.1.1 One-Hot Vectors Are a Bad Choice

  • One-hot word vectors cannot encode similarities among different words because the cosine similarity between any two distinct one-hot vectors is 0.Word embeddings instead map words to fixed-length real vectors intended to express semantic similarity and analogy relationships.Source: §15.1.1 · p. 757, §15.1 · p. 757, §15.1.2 · p. 758

15.1.2 Self-Supervised word2vec

  • word2vec provides self-supervised skip-gram and CBOW models that learn semantically meaningful word representations by predicting words from surrounding context without labeled data.The models obtain supervision from conditional probabilities in text corpora.Source: §15.1.2 · p. 758

15.1.3 The Skip-Gram Model

  • Skip-gram models a center word as generating nearby context words, assumes conditional independence among those context words, and learns separate center and context vectors by maximizing likelihood.For a context window, the conditional probability factors into products of probabilities for individual context words, and training minimizes the corresponding negative log-likelihood.Source: §15.1.3 · p. 758, Training · p. 759

Training

  • After training, skip-gram typically uses center-word vectors as representations, while CBOW typically uses context-word vectors.Both models provide word embeddings, or real-valued feature vectors for words.Source: Training · p. 760, §15.1.5 · p. 761

15.1.4 The Continuous Bag of Words (CBOW) Model

  • CBOW predicts a center word from surrounding context words, averaging their context vectors, whereas skip-gram predicts surrounding words from a center word.CBOW training is otherwise almost the same as skip-gram training.Source: §15.1.4 · p. 760, Training · p. 761, §15.1.5 · p. 761

15.2 Approximate Training

  • The original softmax gradients are expensive because they sum over dictionaries containing hundreds of thousands or millions of words.Negative sampling and hierarchical softmax are introduced as approximate alternatives to this computation.Source: §15.2 · p. 762
  • Negative sampling makes gradient cost independent of dictionary size and linear in the number K of sampled noise words.It augments positive context events with mutually independent negative examples.Source: §15.2.1 · p. 763, §15.2.3 · p. 765

15.2.2 Hierarchical Softmax

  • Hierarchical softmax reduces each training step to O(log2|V|) path operations by representing dictionary words as leaves in a binary tree.The conditional probability follows the root-to-leaf path for the target word.Source: §15.2.2 · pp. 764–765, §15.2.3 · p. 765

15.3 The Dataset for Pretraining Word Embeddings

  • PTB pretraining converts Wall Street Journal text into tokenized minibatches, replacing words occurring fewer than 10 times with <unk>.The corpus is split into training, validation, and test sets.Source: §15.3 · p. 765, §15.3.1 · p. 766

15.3.2 Subsampling

  • Subsampling discards high-frequency words with probability increasing with frequency, shortening sentences and speeding training; the experiment sets t to 10^-4.The sampling rate of “the” is less than 1/20, while low-frequency “join” tokens are completely kept.Source: §15.3.2 · pp. 767–768

15.3.3 Extracting Center Words and Context Words

  • For each center word, the implementation randomly selects a window from 1 through the maximum and uses nearby tokens, excluding the center, as contexts.PTB uses maximum context window size 5.Source: §15.3.3 · pp. 768–769

15.3.4 Negative Sampling

  • Negative sampling draws five noise words per center-context pair with probabilities proportional to dictionary frequency raised to 0.75, excluding context words.A cached random generator supports weighted sampling, and the extracted data contains 1503184 center-context pairs.Source: §15.3.4 · pp. 769–770, §15.3.3 · p. 769

15.3.5 Loading Training Examples in Minibatches

  • Minibatch construction concatenates context and noise words, pads variable-length examples, and uses masks and labels to distinguish valid positive and negative entries.The PTB iterator returns centers, concatenated contexts and negatives, masks, and labels with shapes [512, 1] and [512, 60] for the latter three tensors.Source: §15.3.5 · pp. 770–771, §15.3.6 · p. 772

15.3.6 Putting It All Together

  • The data-loading pipeline reads PTB, builds a vocabulary, subsamples high-frequency words for speed, extracts training pairs, samples negatives, and loads minibatches.The summary identifies subsampling as a speedup and minibatching as the mechanism for handling padding and positive-versus-negative examples.Source: §15.3.6 · pp. 771–772, §15.3.7 · pp. 772–773

Defining the Training Loop

15.4 Pretraining word2vec

15.4.3 Applying Word Embeddings

  • Word embeddings can identify semantically similar words using cosine similarity, while exercises ask readers to improve retrieved neighbors by tuning hyperparameters.The exercises also ask readers to examine the benefits of resampling context and noise words across training epochs for large corpora.Source: §15.4.3 · p. 778, §15.4.5 · p. 778

15.5 Word Embedding with Global Vectors (GloVe)

  • GloVe fits precomputed global word-word co-occurrence statistics using squared loss instead of cross-entropy, with logarithms, word biases, and a weighting function h(xij).This avoids expensive vocabulary-wide normalization and reduces the influence of rare events; nonzero co-occurrence counts can be sampled for efficient training.Source: §15.5.2 · p. 779, §15.5.1 · p. 779, §15.5.4 · p. 781

15.5.3 Interpreting GloVe from the Ratio of Co-occurrence Probabilities

  • GloVe is motivated by co-occurrence-probability ratios: words associated with “ice” or “steam” produce large or small ratios, while unrelated or jointly related words produce ratios near 1.The model fits these ratios through a function of word vectors and ultimately derives its weighted squared-loss formulation.Source: §15.5.3 · pp. 780–781

15.6.1 The fastText Model

  • fastText represents each center word as the sum of character n-gram vectors, sharing parameters across words so rare and out-of-vocabulary words can receive better representations.The approach uses subwords of lengths 3 to 6 plus a special subword, but increases vocabulary size and representation cost.Source: §15.6 · p. 782, §15.6.1 · pp. 782–783

15.6.2 Byte Pair Encoding

  • Byte pair encoding learns variable-length subwords by repeatedly merging the most frequent consecutive symbol pair, producing dataset-dependent segmentations.In the example, 10 merges create symbols such as “ta”, “tal”, “tall”, “fast”, and “er_”, segmenting “faster_” as “fast er_” and “taller_” as “tall er_”.Source: §15.6.2 · pp. 783–785
  • Byte pair encoding greedily merges the most frequent consecutive symbol pairs discovered from training data, then segments new tokens using the learned subwords.For example, “tallest_” becomes “tall e s t _” and “fatter_” becomes “fa t t er_”.Source: §15.6.2 · p. 785

15.6.4 Exercises

  • Subword embeddings may improve representations of rare and out-of-dictionary words, while fastText represents a center word as the sum of its subword vectors.The exercises also consider the issue of too many possible subwords, continuous-bag-of-words subword design, and merge counts for reaching a vocabulary of size m from n initial symbols.Source: §15.6.2 · p. 786, §15.6.4 · p. 786

4. How to extend the idea of byte pair encoding to extract phrases? … 3. Can we leverage BERT in machine translation?

This section develops BERT from contextual representations and self-supervised pretraining through fine-tuning for sentiment analysis and natural language inference, alongside related models and data pipelines. It also introduces reinforcement learning through Markov decision processes, returns, discounting, and the Markov assumption.

15.7.1 Loading Pretrained Word Vectors

  • Pretrained GloVe and fastText embeddings are loaded through a TokenEmbedding class, with the demonstrated 50-dimensional GloVe vocabulary containing 400000 words and an unknown token.Available embeddings include GloVe dimensions 50, 100, and 300, plus a 300-dimensional English fastText model.Source: §15.7.1 · pp. 787–788

Word Similarity

  • Cosine-similarity search retrieves semantically related words, while analogy completion uses vec(c) + vec(b) − vec(a) to identify the missing word.Examples recover lexical, relational, syntactic, and tense patterns, including “chip,” “baby,” “beautiful,” capital-country, adjective-superlative, and present-past analogies.Source: Word Similarity · pp. 788–789, Word Analogy · p. 790

15.8.1 From Context-Independent to Context-Sensitive

  • Context-independent embeddings assign one vector per word, limiting representations for polysemous words such as “crane” across different contexts.This motivates context-sensitive representations that depend on both the token and its surrounding context.Source: §15.8.1 · p. 791

15.8.3 BERT: Combining the Best of Both Worlds

  • BERT combines bidirectional context with minimal task-specific architecture changes, fine-tuning a pretrained Transformer encoder for downstream NLP tasks.BERT represents each token using context from both directions, adds a task-dependent output layer, and fine-tunes all encoder parameters.Source: §15.8.3 · p. 792
  • BERT further improved the state of the art on eleven NLP tasks spanning single-text classification, text-pair classification, question answering, and text tagging.The section presents context-sensitive pretraining as a conceptually simple but empirically powerful approach that transformed NLP solutions.Source: §15.8.3 · pp. 792–793

15.8.4 Input Representation

  • BERT represents single texts or text pairs with <cls> and <sep> tokens, distinguishes segments using eA and eB, and sums token, segment, and learnable positional embeddings.For text pairs, the sequence concatenates the first text, a separator, the second text, and a final separator; single texts use only eA.Source: §15.8.4 · pp. 793–794
  • BERT combines token, segment, and learnable positional embeddings in a Transformer encoder that produces contextual representations for every input token and special token.The encoder sums token, segment, and positional embeddings before applying Transformer blocks.Source: §15.8.4 · p. 794, §15.8.5 · p. 795, §15.8.7 · p. 799

15.8.5 Pretraining Tasks

Masked Language Modeling

  • Masked language modeling randomly selects 15% of tokens, replacing them with “<mask>” 80% of the time, random tokens 10%, or unchanged tokens 10%.Random-token replacement introduces noise that reduces dependence on the masked token during bidirectional context encoding.Source: Masked Language Modeling · p. 796

Next Sentence Prediction

  • Next sentence prediction balances consecutive and randomly paired sentences, then classifies each pair from the “<cls>” representation encoding both sentences.The classifier returns binary predictions for each BERT input sequence.Source: Next Sentence Prediction · pp. 797–798

15.8.6 Putting It All Together

  • BERTModel combines the encoder, masked-language-model head, and next-sentence-prediction head, returning encoded representations and both task predictions.Its final loss is a linear combination of the masked language modeling and next sentence prediction losses.Source: §15.8.6 · pp. 798–799, §15.8.7 · p. 799

15.9 The Dataset for Pretraining BERT

  • Because full pretraining corpora are impractical for many readers and domain-specific applications may need customization, the book uses WikiText-2 and helper functions to generate both task datasets.WikiText-2 retains punctuation, case, and numbers; preprocessing creates next-sentence pairs and masked-token predictions while respecting special tokens and maximum sequence length.Source: §15.9 · p. 800, §15.9.1 · p. 801, Generating the Next Sentence… · p. 801

Generating the Masked Language Modeling Task

  • The implementation replaces 15% of eligible tokens for masked language modeling, using the mask token most often while sometimes retaining or randomly replacing the original token.Special <cls> and <sep> tokens are excluded from prediction, and the resulting prediction positions and labels are returned with the transformed inputs.Source: Generating the Masked Language… · pp. 802–803

15.9.2 Transforming Text into the Pretraining Dataset

  • The dataset pipeline pads token, segment, prediction-position, label, and weight arrays, filters infrequent tokens, and loads WikiText-2 through a customized dataset class.The original BERT uses WordPiece embeddings with a 30000-word vocabulary, whereas this implementation uses the d2l.tokenize function for simplicity.Source: §15.9.2 · pp. 803–805
  • The WikiText-2 dataset generates arbitrarily accessible masked-language-modeling and next-sentence-prediction examples from sentence pairs, with padded inputs and masked-token weights.With batch size 512 and maximum sequence length 64, each minibatch predicts 10 masked positions, yielding shapes [512, 64] and [512, 10].Source: §15.9.2 · pp. 803–805, §15.9.3 · p. 806

15.9.4 Exercises

  • An exercise suggests replacing period-only sentence splitting with spaCy or NLTK’s Punkt tokenizer to handle alternative sentence boundaries.For example, NLTK splits “This is great ! Why not ?” into two sentence strings.Source: §15.9.4 · p. 806

15.10.1 Pretraining BERT

  • The pretraining procedure uses a demonstration-scale BERT with 2 Transformer layers, 128 hidden units, and 2 attention heads, rather than the much larger original variants.The original BERT base and large models use 110 million and 340 million parameters, respectively; the demonstration model combines both task losses during iteration-based training.Source: §15.10.1 · p. 807
  • After 50 training steps, the small BERT reports MLM loss 5.411, NSP loss 0.764, and processes 2447.5 sentence pairs/sec on CUDA.The pretraining loss is the sum of masked language modeling and next sentence prediction losses.Source: §15.10.1 · pp. 807–808

15.10.2 Representing Text with BERT

  • BERT returns contextual representations for complete inputs and individual tokens; for a six-token “a crane is flying” input, the encoded sequence has shape [1, 6, 128].The <cls> representation summarizes the entire input sentence, while the token representation at the crane position supports examining polysemy.Source: §15.10.2 · pp. 808–809

15.10.3 Summary

  • BERT representations are context-sensitive: the same token receives different representations when its surrounding context changes.BERT can represent a single text, a text pair, or any token within them.Source: §15.10.2 · p. 809, §15.10.3 · p. 810

16.1 Sentiment Analysis and the Dataset

  • The IMDb dataset contains 25,000 training and 25,000 test reviews with balanced positive and negative labels for sentiment classification.Sentiment analysis transforms variable-length text sequences into fixed-length categories.Source: §16.1 · p. 812, §16.1.1 · p. 813

16.1.2 Preprocessing the Dataset

  • Reviews are word-tokenized with vocabulary filtering, then truncated or padded to 500 tokens before batching into tensors of shape [64, 500].The resulting iterator contains 391 batches, with labels shaped [64].Source: §16.1.2 · pp. 813–814, §16.1.3 · p. 814

16.2.1 Representing Single Text with RNNs

  • The sentiment model feeds fixed pretrained 100-dimensional GloVe token vectors into a multilayer bidirectional LSTM, concatenates endpoint hidden states, and predicts positive or negative classes.The pretrained word vectors remain fixed during downstream training.Source: §16.2 · p. 816, §16.2.1 · p. 816, §16.2.2 · pp. 817–818

16.2.3 Training and Evaluating the Model

  • 0.500 test accuracy was achieved by the two-layer bidirectional RNN using pretrained GloVe embeddings for IMDb sentiment analysis.The reported training accuracy was 0.502, with loss 0.693 and throughput of 2089.4 examples/sec on GPUs.Source: §16.2.3 · p. 818

16.3.1 One-Dimensional Convolutions

  • One-dimensional convolution slides a kernel across a sequence, computes elementwise products and sums, and extends to multiple channels by summing channel-wise cross-correlations.Multi-input-channel one-dimensional cross-correlations are equivalent to single-input-channel two-dimensional cross-correlations.Source: §16.3.1 · pp. 820–822, §16.3.4 · p. 826

16.3.2 Max-Over-Time Pooling

  • Max-over-time pooling selects the maximum value from each channel, extracting the most important feature across time while allowing channels to have different numbers of time steps.The operation corresponds to one-dimensional global max-pooling.Source: §16.3.2 · pp. 822–823, §16.3.4 · p. 826

16.3.3 The textCNN Model

  • TextCNN applies multiple one-dimensional convolutions to token representations, max-over-time pools their channels, concatenates the results, and classifies them with a fully connected layer.Different kernel widths capture local features across different numbers of adjacent tokens, while dropout can reduce overfitting.Source: §16.3.3 · p. 823

Defining the Model

  • The implementation combines trainable and fixed 100-dimensional GloVe embeddings, applies convolutional layers with kernel widths 3, 4, and 5, and uses 100 output channels per layer.The fixed embedding weights are not trained, while the other embedding weights are trained.Source: Defining the Model · p. 824, Loading Pretrained Word Vectors · p. 825

Training and Evaluating the Model

  • 0.871 test accuracy was achieved by textCNN after five epochs, with 0.979 training accuracy and loss 0.066.The model processed 2,832.0 examples/sec on the listed CUDA devices.Source: Training and Evaluating the Model · p. 825

16.4 Natural Language Inference and the Dataset

  • Natural language inference determines whether a hypothesis follows from a premise, requiring reasoning over sentence pairs rather than classifying a single text sequence.The task supports applications including information retrieval and open-domain question answering.Source: §16.4 · p. 827, §16.4.1 · p. 827

16.4.1 Natural Language Inference

  • The three natural-language-inference relationships are entailment, contradiction, and neutral, depending on whether the hypothesis, its negation, or neither can be inferred from the premise.The examples distinguish these cases using hugging and showing affection, running and sleeping, and performing and being famous.Source: §16.4.1 · p. 827

16.4.2 The Stanford Natural Language Inference (SNLI) Dataset

  • The Stanford Natural Language Inference Corpus contains over 500000 labeled English sentence pairs for studying natural language inference.The dataset is downloaded and stored under ../data/snli_1.0.Source: §16.4.2 · p. 828

Defining a Class for Loading the Dataset

Reading the Dataset

  • SNLI contains about 550000 training pairs and 10000 testing pairs, with balanced entailment, contradiction, and neutral labels in both splits.The observed counts are [183416, 183187, 182764] for training and [3368, 3237, 3219] for testing.Source: Reading the Dataset · p. 829

16.4.3 Summary

  • Natural language inference determines whether a hypothesis follows from a premise, with entailment, contradiction, and neutral as the possible relationships.SNLI is presented as a popular benchmark dataset for this task.Source: §16.4.3 · p. 831

16.5 Natural Language Inference: Using Attention

Attending

  • Attention softly aligns every token in each sequence with tokens in the other sequence using weighted averages of their representations.The alignment mechanism applies an MLP separately to tokens, requiring m + n rather than mn applications and therefore achieving linear instead of quadratic complexity.Source: §16.5.1 · p. 832, Attending · pp. 833–834

Comparing

  • The comparing step concatenates each token with its softly aligned representation and feeds the result through an MLP for both sequences.This produces comparison vectors for premise tokens and hypothesis tokens before aggregation.Source: Comparing · p. 835

Aggregating

  • The aggregating step sums comparison vectors from both sequences, concatenates the summaries, and applies an MLP and linear layer to predict the logical relationship.The complete model uses three output classes: entailment, contradiction, and neutral.Source: Aggregating · p. 835, Putting It All Together · p. 836

16.5.2 Training and Evaluating the Model

Training and Evaluating the Model

16.5.3 Summary

  • The decomposable-attention model predicts premise–hypothesis relationships through attending, comparing, and aggregating, while its decomposition trick reduces attention-weight computation from quadratic to linear complexity.Attention softly aligns tokens in each sequence with tokens in the other using weighted averages.Source: §16.5.3 · p. 838

16.6.1 Single Text Classification

  • For single-text classification, the <cls> representation encodes the whole sequence and feeds an MLP that predicts the distribution over discrete labels.The setup applies to tasks such as sentiment analysis and linguistic acceptability classification.Source: §16.6.1 · p. 840

16.6.2 Text Pair Classification or Regression

  • For text-pair classification or regression, BERT represents both sequences with <sep>, and regression tasks can output a continuous value using mean squared loss.Semantic textual similarity predicts scores from 0, indicating no meaning overlap, to 5, indicating meaning equivalence.Source: §16.6.2 · pp. 840–841

16.6.3 Text Tagging

  • For token tagging, the representation of every input token is passed through shared fully connected layers to predict that token’s label, such as a part-of-speech tag.This differs from single-text classification, which uses the representation of the special classification token.Source: §16.6.3 · p. 841

16.6.4 Question Answering

  • For question answering, BERT predicts start and end positions for an answer span using separate token-level scoring layers and selects the valid span with the highest combined score s_i + e_j.Training maximizes the log-likelihoods of the ground-truth start and end positions.Source: §16.6.4 · pp. 842–843

16.6.5 Summary

  • BERT supports single-text classification, text-pair classification or regression, text tagging, and question answering using only extra fully connected layers beyond the pretrained model.The extra layers are learned from scratch while all pretrained BERT parameters are fine-tuned for the downstream task.Source: §16.6 · p. 839, §16.6.5 · p. 843

16.7. Natural Language Inference: Fine-Tuning BERT

  • BERT fine-tuning for SNLI packs each premise–hypothesis pair into one input with segment IDs, truncates the longer sequence to max_len, and uses parallel workers to generate examples.The section loads the demonstration-scale bert.small model and feeds pretrained BERT into an additional MLP for natural language inference.Source: §16.7 · p. 844, §16.7.2 · p. 845, §16.7.1 · p. 845

16.7.3 Fine-Tuning BERT

  • BERT fine-tuning adds a two-layer MLP that maps the <cls> representation of premise-hypothesis pairs to entailment, contradiction, and neutral outputs.The pretrained encoder and hidden MLP layer are fine-tuned, while the output layer is learned from scratch; pretraining-only loss parameters are not updated.Source: §16.7.3 · p. 847
  • 0.777 test accuracy was achieved by fine-tuning BERT on SNLI, with 0.788 training accuracy after five epochs.The model processed 8176.1 examples/sec on the reported CUDA devices.Source: §16.7.3 · p. 848

16.7.5 Exercises

  • Reinforcement learning studies sequential decisions whose actions affect future states and outcomes, using MDPs, value iteration, Q-learning, imitation, and deep networks for unknown environments.The chapter presents these methods as foundations for more advanced reinforcement-learning algorithms and real-world applications.Source: §16.7.5 · pp. 849–850

17.1.1 Definition of an MDP

  • An MDP is defined by (S, A, T, r), comprising states, actions, transition probabilities, and user-designed rewards that quantify action usefulness.The transition function satisfies T(s, a, s′) = P(s′ | s, a) and defines a probability distribution over next states.Source: §17.1.1 · p. 851, §17.1.2 · p. 852, §17.1.4 · p. 853

17.1.2 Return and Discount Factor

  • The reinforcement-learning objective is to maximize trajectory return, while discount factor γ < 1 keeps infinite trajectories meaningful and controls the balance between short-term and future rewards.Small γ favors shorter goal-reaching trajectories, whereas γ = 0.99 encourages exploration before finding a good route.Source: §17.1.2 · p. 852

17.1.3 Discussion of the Markov Assumption

  • The Markov assumption requires the next state to depend only on the current state and action, but state design can restore this property by including relevant history such as velocity.For an accelerating robot, representing state as (location, velocity) makes the dynamics Markovian.Source: §17.1.3 · pp. 852–853

1. What would be the set of states? … 3. What should we have done instead?

This section develops methods for sequential decision-making, probabilistic function modeling, hyperparameter optimization, and generative modeling, combining theoretical foundations with practical implementations. It covers value iteration and Q-learning, Gaussian-process regression, asynchronous and multi-fidelity optimization, and GAN architectures and training.

17.2.2 Value Function

  • The value function decomposes into immediate expected reward plus the expected next-state value, forming the dynamic-programming foundation for reinforcement-learning algorithms.This identity holds for every state and can be expressed using Markov decision-process transition probabilities.Source: §17.2.2 · pp. 854–855

17.2.3 Action-Value Function

  • An optimal deterministic action maximizes immediate reward plus the expected discounted return over possible next states, and the same principle defines optimal value and action-value functions.The action-value function fixes the first action while averaging subsequent returns; policy values depend on the chosen policy.Source: §17.2.3 · p. 855, §17.2.4 · p. 855

17.2.7 Policy Evaluation

  • Policy evaluation uses iterative updates to compute the value function of any policy, including stochastic policies, and converges to the correct value regardless of initialization.Value Iteration instead computes the optimal deterministic policy, whereas policy evaluation evaluates a supplied policy.Source: §17.2.7 · pp. 856–857

17.2.6 Value Iteration

  • Value Iteration converges to the optimal value function from arbitrary initialization and, on FrozenLake, finds the optimal policy after 10 iterations.The FrozenLake setup uses a reliable 4 × 4 grid, reward 1 at the goal, and reward 0 elsewhere; the goal is reachable from every non-hole state.Source: §17.2.6 · p. 856, §17.2.8 · pp. 857, 859

17.3 Q-Learning

  • Q-Learning removes the need to know transition dynamics by replacing the model-based next-state expectation with states observed in the robot’s collected data.Its optimization uses trajectory samples, handles terminal states, and derives an optimal deterministic policy from the learned action-value estimate.Source: §17.3 · p. 859, §17.3.1 · p. 860, §17.3.2 · pp. 860–861

17.3.3 Exploration in Q-Learning

  • Q-Learning depends critically on exploration: insufficient state-action coverage harms estimates globally, while ε-greedy or softmax policies balance exploitation with random exploration.Its self-correcting updates reduce overestimated bad actions and reinforce good ones, enabling convergence to the optimal policy even from a random starting policy.Source: §17.3.3 · p. 861, §17.3.4 · p. 862

17.3.5 Implementation of Q-Learning

  • Q-learning finds FrozenLake’s optimal solution after roughly 250 iterations, whereas value iteration needs far fewer because it has access to the full MDP.The environment is a 4 × 4 grid with unknown holes, frozen cells, and goal location; Q-learning does not require complete transition and reward functions.Source: §17.2.8 · p. 857, §17.3.5 · pp. 863–864, §17.3.6 · p. 865

1. What is the difference between epistemic uncertainty versus observation uncertainty?

  • The chapter’s exercises probe Q-learning’s dependence on grid size, discount factor γ, and exploration parameter ϵ, while later questions examine epistemic versus observation uncertainty and RBF assumptions.These questions extend the presented methods by asking readers to compare larger grids and parameter settings and assess whether distance-based covariance is reasonable.Source: §17.3.7 · p. 865, §1 · pp. 878–879

17.3.7 Exercises

  • The Gaussian-process chapter introduces priors, kernels, posterior inference, numerical methods, and GPyTorch implementations to connect function-space reasoning with scalable deep-learning practice.It presents Gaussian processes as broadly useful for active learning, hyperparameter learning, auto-ML, and spatiotemporal regression, while emphasizing their complementarity with deep neural networks.Source: §17.3.7 · p. 866, §18.1 · p. 878

18.1 Introduction to Gaussian Processes

  • Gaussian processes model distributions over functions directly, enabling reasoning about properties such as variation, periodicity, conditional independence, and translation invariance rather than uninterpretable parameters.They represent joint Gaussian distributions over function values, with covariance functions controlling correlations and generalization properties.Source: §18.1 · pp. 867, 874, 875, §18.1.1 · p. 878
  • Conditioning a Gaussian-process prior on observed data produces a posterior whose mean gives point predictions and whose variance gives uncertainty, computable in closed form.A 95% credible interval is formed as m ± 2s, and epistemic uncertainty grows away from observed data because fewer solutions remain constrained there.Source: §18.1 · pp. 867, 868, 869, 875
  • The RBF kernel’s length-scale controls function wiggliness and locality: increasing it makes functions smoother and uncertainty smaller, while decreasing it makes uncertainty rise quickly away from data.The amplitude controls vertical scale without changing the rate of variation; reasonable hyperparameters can be selected automatically through the marginal likelihood.Source: §18.1 · pp. 869, 870, 872, 874

18.2 Gaussian Process Priors

  • Gaussian processes define priors directly over functions through mean and covariance functions, supporting reasoning about model construction, generalization, active learning, and hyperparameter tuning.Any finite collection of function values has a joint multivariate Gaussian distribution determined by m(x) and k(x, x′).Source: §18.2 · p. 879, §18.2.1 · pp. 879–880, §18.2.3 · p. 881

18.2.2 A Simple Gaussian Process

  • The linear model f(x)=w0+w1x is a Gaussian process because each function value is Gaussian and every finite collection is jointly multivariate Gaussian.For w0,w1∼N(0,1), f(x)∼N(0,1+x^2).Source: §18.2.2 · p. 880

18.2.3 From Weight Space to Function Space

  • Function-space representations specify mean and covariance directly, allowing models with infinitely many parameters to be sampled using finite computation.For f(x)=w^Tϕ(x), w∼N(u,S), the mean is u^Tϕ(x) and covariance is ϕ(x)^TSϕ(x′).Source: §18.2.3 · pp. 881–882, §18.2.6 · p. 885

18.2.4 The Radial Basis Function (RBF) Kernel

  • An RBF-kernel Gaussian process is a universal approximator, representing any continuous function to arbitrary precision while retaining strong flexibility and generalization on small datasets.The RBF kernel uses amplitude and length-scale hyperparameters, and sampled functions are generated from the corresponding multivariate Gaussian distribution.Source: §18.2.4 · pp. 882–883

18.2.5 The Neural Network Kernel

  • Infinite-width neural networks induce Gaussian processes under independent, scaled output weights, while their kernels are non-stationary unlike stationary RBF kernels.The central limit theorem gives joint Gaussian function values; the neural-network kernel can vary qualitatively across input space, especially near the origin.Source: §18.2.5 · p. 884

18.3.1 Posterior Inference for Regression

  • For regression, Gaussian-process inference is closed form by combining a GP latent function with independent Gaussian observation noise and applying Gaussian identities.The observations have distribution N(0,K(X,X)+σ^2I), with cross-covariance cov(f*,y)=K(X*,X).Source: §18.3 · p. 886, §18.3.1 · pp. 886–887

18.3.2 Equations for Making Predictions and Learning Kernel Hyperparameters in GP Regression

  • The standard procedure maximizes the marginal likelihood to learn kernel hyperparameters, then uses the predictive mean and twice the predictive standard deviation for a 95% credible set.The marginal likelihood separates model fit from a log-determinant complexity term, favoring the simplest fits consistent with the data.Source: §18.3.1 · p. 887, §18.3.2 · p. 888, §18.3.3 · p. 888

18.3.3 Interpreting Equations for Learning and Predictions

  • GP regression permits exact Bayesian inference in closed form, with the predictive mean as a kernel-weighted combination of training targets and uncertainty increasing away from observed inputs.The predictive mean depends on target values, whereas predictive variance is governed by the kernel and learned hyperparameters.Source: §18.3.3 · p. 888
  • Basic GP regression requires solving a linear system and computing a log determinant on an n × n matrix, each costing O(n^3) time and O(n^2) storage.These bottlenecks historically limited GPs to roughly fewer than 10,000 training points, although advanced methods can scale to millions.Source: §18.3.3 · pp. 888–889

18.3.4 Worked Example from Scratch

  • A learned length-scale of 0.299 and noise standard deviation of 0.24 closely match the true noise, indicating a well-specified GP for this problem.The posterior mean almost perfectly matches the true noise-free function, and the 95% credible set contains that function without being overly wide or narrow.Source: §18.3.4 · pp. 891–892

18.3.5 Making Life Easy with GPyTorch

  • GPyTorch reproduces the regression example while enabling alternative kernels, scalable inference, and approximate inference through small code changes instead of extensive numerical-linear-algebra implementations.Exact inference assumes a Gaussian likelihood; classification and other settings require approximations.Source: §18.3.5 · p. 894
  • GPyTorch produces fits virtually identical to scratch code, with noise variance 0.81 approximately equal to the scratch noise standard deviation 0.283 squared.Its credible interval is plotted in observation space and covers the observed datapoints.Source: §18.3.5 · p. 896

19.1 What Is Hyperparameter Optimization?

  • Hyperparameter optimization automates global search by minimizing validation error or another business metric, optionally subject to training-time, inference-time, or model-complexity constraints.Source: §19.1 · p. 900

The Objective Function

  • The optimization objective maps hyperparameters to validation loss, is expensive and generally nondifferentiable through training, and is observed with Gaussian noise from stochastic neural-network training.Source: The Objective Function · p. 901

The Configuration Space

  • A configuration space specifies each hyperparameter’s type, bounded range, and prior distribution, with logarithmic scales useful for positive parameters spanning several orders of magnitude.Ranges that exclude good configurations or are excessively broad can respectively prevent discovery or make the search infeasible.Source: The Configuration Space · pp. 902–903

19.1.2 Random Search

  • Random search independently samples configurations until its budget is exhausted and returns the lowest-validation-error configuration, while supporting parallel evaluations and arbitrary parameter distributions.Source: §19.1.2 · pp. 903, 905
  • Random search wastes resources because it neither adapts sampling to prior observations nor stops poorly performing configurations early, motivating model-guided and early-stopping methods.Source: §19.1.2 · p. 905

19.1.3 Summary

  • Random search is often more efficient than grid search because it mitigates dimensionality when performance depends mainly on a small subset of hyperparameters.Source: §19.1.3 · p. 906

3. What should we have done instead?

  • Gradient-based hyperparameter optimization is difficult because validation performance depends on a large computational graph, substantial activation storage, and potentially vanishing or exploding gradients.Source: §3 · p. 906

19.2 Hyperparameter Optimization API

  • The HPO API separates searching for configurations from scheduling trial evaluations, with HPOTuner executing the optimization process.Searcher and scheduler abstractions also appear in Syne Tune, Ray Tune, and Optuna.Source: §19.2 · p. 907

19.2.1 Searcher

  • HPOSearcher samples configurations and updates trial history, enabling methods such as Bayesian optimization to propose increasingly promising candidates; RandomSearcher optionally evaluates an initial configuration first.RandomSearcher then draws subsequent configurations independently from the specified configuration-space distributions.Source: §19.2.1 · pp. 907–908

19.2.2 Scheduler

  • HPOScheduler requests configurations and decides evaluation parameters such as max_epochs, while BasicScheduler schedules a new configuration whenever resources become available.Scheduler updates receive observations returned by completed trials.Source: §19.2.2 · p. 908

19.2.3 Tuner

  • HPOTuner sequentially runs training jobs, passes configurations to the objective, updates the scheduler, and records errors and runtimes for each trial.The tuner maintains incumbent information, incumbent trajectories, cumulative runtime, and trial records.Source: §19.2.3 · p. 909

19.2.4 Bookkeeping the Performance of HPO Algorithms

  • HPO performance is tracked through incumbent validation error versus cumulative wall-clock time, measuring both the quality of the best configuration and how quickly it is found.Each iteration’s runtime includes objective evaluation and the scheduler’s decision time.Source: §19.2.4 · pp. 909–910

19.2.5 Example: Optimizing the Hyperparameters of a Convolutional Neural Network

  • The LeNet example tunes learning rate and batch size with validation error as its objective, evaluates the default configuration first, and plots random search’s incumbent trajectory over time.The configuration space uses learning_rate sampled log-uniformly from 1e-2 to 1 and batch_size sampled as an integer from 32 to 256; the default is learning_rate 0.1 and batch_size 128.Source: §19.2.5 · pp. 910–911

19.2.6 Comparing HPO Algorithms

  • After roughly 1000 seconds, Bayesian optimization quickly outperforms random search, although both perform similarly earlier across 50 random-seed repetitions.The comparison reports average incumbent performance with standard-deviation bands and motivates repeated experiments because training and HPO introduce separate randomness sources.Source: §19.2.6 · p. 913

19.3 Asynchronous Random Search

  • Asynchronous random search schedules a new trial whenever resources free up, avoiding synchronization overhead and reaching a given performance K times faster with K parallel trials.Because configurations are sampled independently without using prior observations, random search is straightforward to parallelize asynchronously, though it requires more resources.Source: §19.3 · p. 915, Figure 19.3.1 · p. 915

19.3.1 Objective Function

  • The objective function trains the model and reports validation error after each epoch for Syne Tune.Training is invoked through trainer.fit or trainer.fit_epoch, followed by validation-error extraction and reporting.Source: §19.3.1 · p. 916

19.3.2 Asynchronous Scheduler

  • A best validation_error of 0.15581202507019043 was obtained for trial-id 4 in the asynchronous tuning experiment.The experiment used a wallclock stopping criterion of 720 seconds and reported 12 finished trials.Source: §19.3.2 · pp. 921–922

19.3.3 Visualize the Asynchronous Optimization Process

  • Asynchronous scheduling starts a new trial whenever a worker becomes available, keeping workers busy instead of waiting for slower trials at synchronization points.The method can distribute random-search trials across parallel resources without changing the underlying algorithm, and the tuner centralizes experiment bookkeeping.Source: §19.3.2 · pp. 916–918, §19.3.3 · p. 923, §19.3.4 · p. 923

19.3.4 Summary

  • The summary exercises extend the setup by comparing random search with Bayesian optimization, testing worker scaling, and implementing and benchmarking a new LocalSearcher.The worker-scaling exercise varies n_workers=1, n_workers=2, and n_workers=4, with linear scaling expected for random search.Source: §19.3.4 · p. 924

19.4 Multi-Fidelity Hyperparameter Optimization

  • Multi-fidelity hyperparameter optimization speeds evaluation by allocating more resources to promising configurations and stopping poorly performing ones early.Because learning curves can reveal promising configurations before full training, this approach can test more configurations with the same total resources.Source: §19.4 · pp. 924–925
  • Multi-fidelity HPO evaluates configurations with a resource budget r, assuming error decreases and computational cost increases as r grows.The resource can represent training epochs, training-subset size, or the number of cross-validation folds.Source: §19.4 · p. 925, §19.4.2 · p. 933

19.4.1 Successive Halving

  • Successive halving starts with N configurations at rmin, retains the top 1/η fraction after each rung, and increases surviving trials’ budgets by η until rmax.With N = η^K and rmax = rminη^K, one trial reaches the full budget per round.Source: §19.4.1 · p. 926
  • The implementation maintains rung queues, sorts configurations by observed validation error, and promotes the best configurations when each rung has collected enough observations.If a worker becomes free before a rung is complete, a new round starts temporarily, while completed-rung promotions receive queue priority.Source: §19.4.1 · pp. 927, 928, 932

19.4.2 Summary

  • Most configurations are stopped early, so only better-performing trials reach rmax, unlike vanilla random search, which allocates rmax to every configuration.This reduces overall HPO computation rather than merely wall-clock time.Source: §19.4.1 · pp. 929–930, §19.4.2 · p. 933

19.5 Asynchronous Successive Halving

  • Synchronous successive halving can leave distributed workers idle because promotions require all current-rung observations and training times and rung capacities vary.These synchronization costs arise from stragglers and from rung sizes that are not multiples of the worker count.Source: §19.5 · pp. 935–937
  • ASHA promotes configurations after at least η observations at a rung, eliminating synchronization points while allowing occasional suboptimal promotions; the example reuses the prior objective and configuration space with a two-worker scheduler.The method starts a new rmin trial when no promotion is available, and the implementation specifies epoch as the resource attribute and validation_error as the minimized metric.Source: §19.5 · pp. 938–939, §19.5.1 · p. 941, §19.5.2 · pp. 941–942

19.5.2 Asynchronous Scheduler

  • The asynchronous scheduler maps metric, resource, maximum-resource, grace-period, and reduction-factor settings to the tuner configuration.The run uses max_epochs as the maximum resource, min_number_of_epochs as the grace period, and η as the reduction factor.Source: §19.5.2 · p. 942
  • Syne Tune detects 8 GPUs and schedules trials with varied learning rates and batch sizes, using max_epochs=10.The logs show trial configurations being launched through separate subprocesses while results are saved to the Syne Tune run directory.Source: §19.5.2 · pp. 942–943
  • After 437.06 seconds, 16 trials had finished, 2 were running, and 3 remained until the end of the tuning run.The status output reports ongoing asynchronous execution rather than final tuning results.Source: §19.5.2 · p. 946

19.5.3 Visualize the Optimization Process

  • The asynchronous scheduler visualizes trial learning curves over wall-clock time, showing that trials stop at different rung levels because their epoch durations differ.Standard successive halving would require worker synchronization before promoting configurations to the next rung.Source: §19.5.3 · p. 951

19.5.2 Asynchronous Scheduler

  • Asynchronous successive halving usually gains more from avoiding synchronization than it loses through occasional suboptimal promotions.Configurations are promoted as quickly as possible, even when some wrong ones advance, and most trials stop after 1 or 2 epochs at different times.Source: §19.5.4 · p. 951, §19.5.3 · p. 951

20.1 Generative Adversarial Networks

  • GANs train a generator and discriminator adversarially, using the discriminator’s real-versus-fake judgment as a constructive two-sample training signal.The generator seeks D(G(z)) ≈ 1, while the discriminator minimizes binary cross-entropy for real and generated data.Source: §20.1 · pp. 952–953, §20.1.5 · p. 958

20.1.1 Generate Some “Real” Data

  • The introductory GAN example transforms Gaussian noise into samples matching a shifted Gaussian distribution, using a linear generator and a three-layer MLP discriminator.The real data are generated from a Gaussian transformation with mean b and covariance A^T A; training alternates discriminator and generator updates with Adam.Source: §20.1.1 · p. 954, §20.1.2 · p. 955, §20.1.3 · p. 955

20.2 Deep Convolutional Generative Adversarial Networks

  • The DCGAN demonstration extends GANs from simple distributions toward photorealistic image generation by adapting convolutional architectures for generation.It uses the Pokemon sprites dataset, resizing images to 64×64 and normalizing pixels to match the generator’s tanh output range.Source: §20.2 · p. 958, §20.2.1 · pp. 958–959

20.2.2 The Generator

  • The DCGAN generator maps a latent vector z ∈ R^d to a 64×64 RGB image through transposed-convolution blocks with batch normalization and ReLU activation.These blocks progressively enlarge the input spatial dimensions.Source: §20.2.2 · pp. 959–960
  • The generator uses four transposed-convolution blocks to expand a latent variable from 1 × 1 to 32 × 32, then outputs 3-channel 64 × 64 values in (−1, 1) via tanh.A 100-dimensional latent input produces an output shape of torch.Size([1, 3, 64, 64]).Source: §20.2.2 · p. 961

20.2.3 Discriminator

  • The discriminator mirrors the generator with strided convolutional blocks using batch normalization and leaky ReLU, ending in a single output-channel prediction.Leaky ReLU provides nonzero outputs for negative inputs and is intended to address the dying-ReLU problem.Source: §20.2.3 · pp. 961–963

20.2.4 Training

  • Training uses the same learning rate for generator and discriminator, Adam with β1 = 0.5, four-dimensional random noise, and GPU acceleration.The lower β1 reduces momentum smoothness to accommodate rapidly changing gradients as the networks compete.Source: §20.2.4 · p. 963
  • loss_D 0.085 and loss_G 7.452 were reported after a 20-epoch demonstration run processing 1912.6 examples/sec on cuda:0.The authors note that increasing num_epochs can improve performance.Source: §20.2.4 · p. 964

1. What will happen if we use standard ReLU activation rather than leaky ReLU?

The supplied passage describes applying DCGAN to Fashion-MNIST to identify which categories work well and which do not.

1. What will happen if we use standard ReLU activation rather than leaky ReLU?

  • DCGAN is applied to Fashion-MNIST to compare which categories work well and which do not.Source: §1 · p. 965

A Mathematics for Deep … A.4.6 Hessians

This section provides mathematical foundations for deep learning, spanning geometric linear algebra, matrix analysis, differential calculus, and multivariable optimization. It connects these concepts to practical modeling and computation through eigendecompositions, Taylor approximations, gradients, backpropagation, and Hessian-based local analysis.

A Mathematics for Deep

  • The appendix develops core mathematics for deep learning without aiming to be exhaustive, covering linear algebra, calculus, probability, statistics, and information theory.It targets the understanding of architectural effects, gradient flow, loss-function assumptions, entropy, and bits-per-character.Source: A Mathematics for Deep · pp. 966–967

A.1 Geometry and Linear Algebraic Operations

  • Linear algebra is a foundational pillar of machine learning, and this section extends earlier operational treatment with geometric interpretations and eigenvalue-related concepts.The discussion builds intuition for transformations between potentially different high-dimensional spaces, while acknowledging that complete internalization requires substantial practice.Source: §A.1 · p. 967, §A.1.4 · p. 974

A.1.1 Geometry of Vectors

  • Vectors can represent either points or directions, enabling geometric views of data, clustering, vector addition, and subtraction.In this view, classification becomes the problem of separating distinct clusters of points in space.Source: §A.1.1 · pp. 967–969

A.1.2 Dot Products and Angles

  • The dot product and vector norms determine the angle between vectors in any dimension, while orthogonality occurs exactly when their dot product is zero.Angles preserve directional similarity under scaling, making them useful when brightness or document length changes without changing content.Source: §A.1.2 · pp. 970–971

Cosine Similarity

  • Cosine similarity measures vector closeness by direction, ranging from 1 for aligned vectors to −1 for opposite vectors and 0 for orthogonal vectors.Random high-dimensional vectors with mean-zero components typically have cosine similarity near 0.Source: Cosine Similarity · p. 971

A.1.3 Hyperplanes

  • Hyperplanes have d −1 dimensions in d-dimensional space and divide it into two half-spaces, providing the geometric basis for linear decision boundaries.Deep classifiers can be viewed as learning nonlinear embeddings that make target classes separable by hyperplanes.Source: §A.1.3 · pp. 971–973

A.1.4 Geometry of Linear Transformations

  • A matrix transforms an entire coordinate grid through global skewing, rotation, and scaling, rather than distorting different regions independently.Its action can be understood by tracking the basis vectors, which reduce the behavior of every vector to their weighted combinations.Source: §A.1.4 · pp. 975–976, Figure A.8 · p. 975

A.1.5 Linear Dependence

  • Linear dependence identifies compression: a matrix’s rank is the largest number of linearly independent columns, while full rank permits invertibility.Dependent columns make transformations irreversible; computing rank by examining every column subset can be exponentially inefficient.Source: §A.1.6 · p. 977, §A.1.5 · p. 977, §A.1.7 · p. 978

Numerical Issues

  • Matrix inversion theoretically undoes full-rank transformations, but numerical stability, near-low-rank matrices, and dense inverses make direct inversion undesirable in practice.The text recommends more numerically stable linear-equation solvers and generally avoiding inversion.Source: §A.1.7 · p. 978, Numerical Issues · pp. 978–979

A.1.8 Determinant

  • The determinant measures signed area or higher-dimensional volume scaling: for a 2×2 matrix it is ad −bc, and a nonzero determinant is equivalent to invertibility.A zero determinant corresponds to compression into a lower-dimensional space, while larger n × n determinants scale n-dimensional volumes.Source: §A.1.8 · pp. 979–980

A.1.9 Tensors and Common Linear Algebra Operations

  • Tensor contractions generalize matrix multiplication by specifying summed indices, with Einstein notation compressing repeated-index sums into concise expressions.The notation represents a more flexible family of transformations and replaces many specialized linear-algebra notations with short tensor expressions.Source: §A.1.9 · p. 981, Common Examples from Linear Algebra · p. 981

Expressing in Code

  • In code, PyTorch tensors expose shapes and einsum reproduces matrix multiplication, yielding identical results to A@v while supporting concise tensor contractions.For the supplied example, both implementations return tensor([ 5, 11]); PyTorch does not support the alternative integer-index notation.Source: Expressing in Code · p. 982

A.2 Eigendecompositions

  • Eigenvectors preserve their direction under a matrix while eigenvalues give the corresponding stretch factors; they can be found by solving det(A −λI) = 0 and then Av = λv.For the example matrix, the eigenvalues are 4 and 1.Source: §A.2 · p. 984, §A.2.1 · p. 985, An Example · p. 985

A.2.2 Decomposing Matrices

  • Eigendecomposition reduces matrix powers and inverses to operations on eigenvalues, showing that invertibility requires no zero eigenvalues, the determinant is their product, and rank counts the nonzero eigenvalues.These simplifications require a full set of linearly independent eigenvectors; symmetric matrices guarantee this with an orthogonal eigenvector matrix and real eigenvalues.Source: §A.2.3 · pp. 986–987, §A.2.2 · p. 986, §A.2.4 · p. 987

A.2.5 Gershgorin Circle Theorem

  • The Gershgorin Circle Theorem places every eigenvalue within a disc centered at a diagonal entry, yielding accurate approximations when diagonal entries dominate; an example gives approximately 0.99, 2.97, 4.95, and 9.08.The computed values are reported as approximately 0.99, 2.97, 4.95, and 9.08, and lie within the theorem’s ranges.Source: §A.2.5 · pp. 987–988

Behavior on Random Data

  • Repeatedly applying an unnormalized random Gaussian matrix can make norms grow uncontrollably, with consecutive norms stabilizing near a stretching factor of 1.974459321485[…].If the matrix instead shrinks vectors, deep iterates lose dependence on their inputs; useful initialization must balance growth and decay.Source: Behavior on Random Data · pp. 989–990

An Observation

  • Iterated matrix powers are governed primarily by the largest eigenvalue: a random vector’s long-term stretching factor matches that eigenvalue, while repeated application aligns the vector with its principal eigenvector.This behavior underlies power iteration for finding the largest eigenvalue and eigenvector (Golub and Van Loan, 1996).Source: An Observation · p. 991, §A.2.8 · p. 993

A.2.7 Discussion

  • Normalizing a random matrix by its principal eigenvalue prevents iterated outputs from exploding, causing random-data norms to equilibrate; large Gaussian matrices have largest eigenvalue about √n on average.For n = 5, √5 ≈2.2, consistent with the circular law; eigenvalue and singular-value behavior connects to neural-network initialization (Pennington et al., 2017).Source: §A.2.7 · p. 993

A.3.1 Differential Calculus

  • Differential calculus studies responses to small changes: zooming into a tiny interval makes a complex function behave like a straight line, motivating local analysis of neural-network loss with respect to weights.Training is framed as repeatedly taking small steps in directions that decrease the loss most rapidly.Source: §A.3 · p. 994, §A.3.1 · pp. 994–995
  • Derivatives measure how a small input change scales the corresponding output change, with finite differences approaching 8 for L(x) at x = 4.The derivative is introduced as a function of x and as the scaling factor relating input and output changes.Source: §A.3.1 · pp. 996–997

A.3.2 Rules of Calculus

  • The calculus treatment emphasizes common rules for computing derivatives of explicit functions rather than deriving everything from first principles.These rules reduce derivative computation to a manageable set of recurring operations.Source: §A.3.2 · p. 998, Common Derivatives · p. 998

Common Derivatives

  • Core derivative rules include constants mapping to 0, linear functions mapping to a, the power rule, and the exponential identity d/dx e^x = e^x.These derivatives serve as reference cases for more complex expressions.Source: Common Derivatives · p. 998

Derivative Rules

  • Sums, products, and compositions generalize derivative computation, allowing complex expressions to be differentiated mechanically while ignoring vanishing higher-order terms.The text illustrates these rules through an expression involving logarithms, powers, sums, constants, and linear functions.Source: Derivative Rules · pp. 998–1000

2. Having a human follow these rules can be tedious and error prone!

  • Because manually applying derivative rules is tedious and error prone, backpropagation provides a route to mechanize the process.The section identifies backpropagation as the mechanism that revisits this problem later.Source: §2 · p. 1000

Linear Approximation

  • Geometrically, the derivative gives a line through (x, f(x)) whose slope is df/dx(x), providing a local linear approximation.The section demonstrates this idea with linear approximations to sin(x).Source: Linear Approximation · p. 1000

Higher Order Derivatives

  • The second derivative describes curvature: positive values produce upward curves and minima, negative values produce downward curves and maxima, and zero values yield straight lines.Quadratic approximations use the first two derivatives to approximate functions such as sin(x).Source: Higher Order Derivatives · pp. 1001–1003

Taylor Series

  • Taylor series construct the best n-th degree polynomial approximation from a function’s first n derivatives at x0.The quadratic case is developed first, and the cubic coefficient includes 3! = 6.Source: Taylor Series · pp. 1003–1004
  • Taylor series approximate well-behaved functions with polynomials from derivative information, supporting theoretical simplification and numerical computation.For real analytic functions such as cos(x) and e^x, increasing the approximation degree approaches the target function; Taylor series also address machine-computation precision questions.Source: Taylor Series · pp. 1004–1005

A.3.3 Summary

  • Higher-order derivatives provide increasingly fine-grained information, while derivatives can be combined through rules to analyze functions with many variables.The preceding derivative concepts motivate extending single-variable calculus to losses involving potentially billions of weights.Source: §A.3.3 · p. 1005, §A.4 · p. 1006

A.3.4 Exercises

  • The section’s exercises reinforce derivative calculation, critical-point reasoning, and minimization for elementary functions.They include derivatives of x^3 − 4x + 1 and log(1/x), testing whether zero derivatives imply extrema, and finding the minimum of x log(x) for x ≥ 0.Source: §A.3.4 · p. 1005

A.4.1 Higher-Dimensional Differentiation

  • Multivariable differentiation approximates changes in a function under arbitrary input perturbations using the gradient and a vector dot product.Changing one variable at a time yields partial derivatives, which combine into the gradient; for f(x,y)=log(e^x+e^y), the approximation gives 1.0819 versus the true value 1.0821.Source: §A.4.1 · pp. 1006–1008

A.4.2 Geometry of Gradients and Gradient Descent

  • Gradient descent minimizes loss by stepping opposite the gradient, the direction of steepest descent, using w ← w − ϵ∇_wL(w).The gradient identifies the direction that decreases the loss most rapidly under the local approximation.Source: §A.4.2 · pp. 1008–1009

A.4.3 A Note on Mathematical Optimization

  • A minimum must be a critical point with zero gradient, but not every zero-gradient point is a global optimum.For the example, candidate points x = −1, 0, 2 have values −5, 0, and −32, so the minimum occurs at x = 2.Source: §A.4.3 · p. 1009

A.4.4 Multivariate Chain Rule

  • The multivariate chain rule sums derivative contributions across computational paths, avoiding repeated symbolic work and explaining gradient flow through neural-network architectures.This perspective connects gradient propagation to architectural choices such as LSTMs and residual layers.Source: §A.4.4 · pp. 1011–1012

A.4.5 The Backpropagation Algorithm

  • Backpropagation applies the multivariate chain rule using local partial derivatives, turning derivatives of composed functions into a manageable computational procedure.The example computes derivatives from intermediate quantities a and b using only single-step partials.Source: §A.4.5 · p. 1012
  • Backpropagation computes the function and single-step partials forward, then propagates the gradient from the output back to every input or weight in one backward pass.This decomposition underlies deep learning algorithms and can be implemented automatically with f.backward().Source: §A.4.5 · pp. 1014–1015

A.4.6 Hessians

  • For a function of n variables, the n^2 second derivatives are assembled into an n × n Hessian, whose mixed entries agree when the mixed partials exist and are continuous.The Hessian captures higher-order behavior beyond the gradient and supports a best-fitting quadratic near a point.Source: §A.4.6 · pp. 1015–1016
  • At [−1, 0]⊤, the example’s approximating quadratic is f(x, y) ≈ e−1 (−1 − (x + 1) + (x + 1)^2 + y^2).The quadratic is obtained from the function value, gradient, and Hessian, and illustrates the general expansion valid for any dimensional input.Source: §A.4.6 · p. 1016
  • The Hessian-based quadratic approximation forms the basis for Newton’s Algorithm, which iteratively finds and exactly minimizes the best-fitting quadratic.The section connects local second-order approximation to numerical optimization discussed in Section 12.3.Source: §A.4.6 · p. 1017

A.4.7 A Little Matrix Calculus … A.8.2 Discrete Uniform

This section develops matrix calculus, integration, and probability, connecting multivariable differentiation and backpropagation with geometric and numerical integration, probability densities, distributions, and random variables. It also introduces joint and marginal distributions, covariance and correlation, maximum likelihood and negative log-likelihood, and common discrete distributions.

A.4.7 A Little Matrix Calculus

  • Matrix derivatives often require involved calculations but yield concise results resembling single-variable derivatives, with transposes arising from denominator-shape conventions.The text recommends index-based derivations or automatic differentiation when matrix operations become numerous or complex.Source: §A.4.7 · pp. 1017, 1018, 1021
  • The matrix-calculus examples show that matching derivative shapes can require transposes, such as using U⊤ to obtain a derivative with the shape of V.A direct expression can fail when matrix dimensions do not match; the detailed derivation confirms the shape-corrected result.Source: §A.4.7 · pp. 1020–1021

A.4.8 Summary

  • In higher dimensions, gradients describe how multivariable functions change, while backpropagation organizes the multivariable chain rule to compute many partial derivatives efficiently.Matrix calculus expresses derivatives of matrix expressions in concise forms.Source: §A.4.8 · pp. 1021–1022

A.4.9 Exercises

  • The exercises ask readers to apply vector derivatives, gradients, critical-point analysis, and geometric interpretations of ∇f = 0.They include dot products, squared norms, log-sum-exp, polynomial critical points, and decompositions f(x) = g(x) + h(x).Source: §A.4.9 · p. 1022

A.5.1 Geometric Interpretation

  • Integrals represent area under a curve, and partitioning an interval into many rectangles provides a numerical approximation whose accuracy improves with finer slices.For the illustrated example, the approximation is 0.7944855690002441 versus a truth of tensor([0.8047]).Source: §A.5.1 · pp. 1022, 1024

A.5.2 The Fundamental Theorem of Calculus

  • The fundamental theorem of calculus reduces definite integration to evaluating an antiderivative at interval endpoints, with additive constants canceling.It reframes area computation as finding a function whose derivative is the integrand.Source: §A.5.2 · pp. 1025–1026
  • The fundamental theorem of calculus makes integration rules follow from the relationship between derivatives and accumulated area.Every integration rule is developed from this connection.Source: §A.5.2 · p. 1026

A.5.3 Change of Variables

  • Change of variables transforms integrals by accounting for how the substitution stretches infinitesimal intervals, enabling difficult integrals to become tractable.The section gives an intuitive rectangle-based derivation and notes that suitable choices such as u(x) = e−x2 can simplify complex integrals.Source: §A.5.3 · pp. 1027–1028

A.5.4 A Comment on Sign Conventions

  • Definite integrals represent signed areas: negative function values and reversed integration limits each introduce a negative sign, while two reversals cancel.This convention parallels the determinant’s interpretation as signed area.Source: §A.5.4 · p. 1028

A.5.5 Multiple Integrals

  • Multiple integrals can be computed by iterating one-dimensional integrals because discretized sums permit rearranging the order of addition, although Fubini’s theorem can fail for some functions.For continuous functions encountered in machine learning, the text states that this concern does not arise; either integration order is otherwise presented as equivalent.Source: §A.5.5 · pp. 1029–1030

A.5.6 Change of Variables in Multiple Integrals

  • In higher dimensions, change of variables uses an injective reparameterization and the determinant of its Jacobian to measure local area, volume, or hyper-volume scaling.The Jacobian generalizes the one-dimensional derivative, and polar coordinates provide an application of the formula.Source: §A.5.6 · p. 1031

From Discrete to Continuous

  • Continuous random variables require probability densities because exact-point probabilities arise as interval probabilities shrink, with each additional decimal digit reducing the probability by roughly a factor of 10.A density p(x) describes relative likelihood across points; it is nonnegative, and integrating it over the real line gives total probability 1.Source: From Discrete to Continuous · pp. 1033–1034, Probability Density Functions · p. 1035

Probability Density Functions

Cumulative Distribution Functions

  • Cumulative distribution functions represent probabilities directly and provide one framework for continuous, discrete, and mixed random variables.This avoids confusing density values with probabilities and accommodates distributions with jumps.Source: Cumulative Distribution Functions · pp. 1036–1037

Means

  • The mean is a weighted average that captures a random variable’s typical location; in the symmetric example taking a −2, a, and a + 2, it equals a.The example assigns probabilities p, 1 −2p, and p to those three values.Source: Means · p. 1037

Variances

  • Variance measures squared deviation from the mean, is nonnegative, and is additive for independent variables; in the running example, it ranges from 0 when p = 0 to 4 when p = 1/2.Variance is measured in squared units, which motivates standard deviation for interpretability in the original units.Source: Variances · pp. 1038–1039

Standard Deviations

  • Standard deviation returns variability to the original units; Chebyshev’s inequality gives a typical range, with 99% of samples within 10 standard deviations of the mean.For the running example, the bound is sharp at p = 1/8 and remains valid when p < 1/8, when the interval contains only the center.Source: Standard Deviations · pp. 1039–1040

Means and Variances in the Continuum

  • Continuous means and variances follow from discretizing the real line and taking limits, but the Cauchy distribution has neither a finite variance nor a well-defined average value.Its variance diverges because the relevant integrand has infinite area, while its mean behaves like log(∞) = ∞.Source: Means and Variances in the Continuum · pp. 1041–1043

Joint Density Functions

  • Joint density functions are needed for correlated random variables, such as neighboring image pixels or nearby stock prices, by describing simultaneous interval probabilities for X and Y.The section introduces the probability that X and Y lie in epsilon-sized intervals around x and y.Source: Joint Density Functions · p. 1044
  • Joint densities extend density functions to multiple potentially correlated variables, remaining nonnegative with total integral one.For two variables, small rectangular probabilities are approximately ϵ^2p(x,y), and the construction extends to p(x1,…,xn).Source: Joint Density Functions · p. 1044

Marginal Distributions

Covariance

  • Covariance measures linear co-fluctuation: in the discrete example it is 2 when variables align, −2 when they flip, and 0 when outcomes are unrelated.Covariance can miss nonlinear dependence, such as the deterministic relation X = Y^2, which has covariance zero.Source: Covariance · p. 1046

Correlation

  • Correlation standardizes covariance into a unit-less coefficient between −1 and 1, preserving relationship direction and strength across changes in scale.For Y = aX + b, correlation is +1 when a > 0 and −1 when a < 0.Source: Correlation · pp. 1048–1049

A.6.2 Summary

  • The chapter summary connects continuous variables to densities and cumulative distributions, and identifies means, variances, standard deviations, Chebyshev’s inequality, marginalization, covariance, and correlation as core tools.These tools support reasoning about distributions, concentration, and linear relationships between correlated variables.Source: §A.6.2 · p. 1050

A.6.3 Exercises; A.7.2 Numerical Optimization and the Negative Log-Likelihood

  • The exercises apply the material to probability tails, Laplace moments, Chebyshev-style plausibility, and covariance from a specified joint density, while numerical likelihood products become impractical for billions of examples.For independent data, multiplying probabilities such as (1/2)^1000000000 falls below machine precision, motivating the negative log-likelihood.Source: §A.6.3 · pp. 1050–1051, §A.7.2 · p. 1053

A.7.1 The Maximum Likelihood Principle

  • Maximum likelihood chooses model parameters that make the observed data most probable, with an uninformative prior yielding the likelihood maximizer.The likelihood is P(X | θ), the probability of the data given the parameters.Source: §A.7 · p. 1051, §A.7.1 · pp. 1051–1052

A Concrete Example

  • The final maximum-likelihood estimate for 9 heads in 13 flips is θ̂ = 9/13, obtained by selecting the nonzero-probability stationary point.The example illustrates how maximum likelihood recovers an intuitive estimate from first principles and can generalize to more complex models.Source: A Concrete Example · pp. 1052–1053

A.7.2 Numerical Optimization and the Negative Log-Likelihood

  • Negative log-likelihood converts products into sums, gives linear-time computation instead of quadratic-time direct differentiation, and connects average loss to cross-entropy.It also supports numerical convenience and retains the same maximizer because logarithms are increasing.Source: §A.7.2 · pp. 1053–1055

A.7.3 Maximum Likelihood for Continuous Variables

  • Maximum likelihood extends to continuous variables by replacing probabilities with probability densities, without changing the optimal parameter choice.For observations matched within range ϵ, the resulting additive term −N log(ϵ) is independent of θ and can be dropped.Source: §A.7.3 · p. 1056

A.7.4 Summary

  • The chapter summarizes maximum likelihood as selecting the model that assigns the observed dataset the highest probability, or probability density in continuous settings.It presents negative log-likelihood as an alternative motivated by numerical stability, simpler gradients, and information-theoretic connections.Source: §A.7.4 · p. 1056

A.7.5 Exercises

  • The exercises apply maximum likelihood to estimate an exponential distribution’s rate and a unit-variance Gaussian’s unknown mean.Both exercises ask for the parameter value maximizing likelihood from the stated observation or dataset.Source: §A.7.5 · p. 1056

A.8 Distributions

  • The distributions section introduces common probability distributions as a basic reference, supported by mathematical explanations and executable plotting code.The text notes that the needed distribution repertoire depends on the machine-learning area.Source: §A.8 · p. 1057

A.8.1 Bernoulli

  • A Bernoulli random variable models a coin flip that equals 1 with probability p and 0 with probability 1 − p, with plotted mass and cumulative functions and array sampling.The implementation samples arbitrary-shaped arrays by thresholding uniform random values against p.Source: §A.8.1 · pp. 1057–1058

A.8.2 Discrete Uniform

  • A discrete uniform random variable on {1, 2, . . ., n} assigns every supported integer probability 1/n and includes cumulative-function plotting and sampling procedures.Its cumulative distribution is 0 below 1, floor(x)/n between the endpoints, and 1 at or above n.Source: §A.8.2 · pp. 1059–1060

A.8.3 Continuous Uniform … Markdown Files in Jupyter

This section develops foundational probability, statistical inference, hypothesis testing, and information-theoretic concepts, including common distributions, estimators, uncertainty, entropy, mutual information, KL divergence, and cross-entropy. It also connects these ideas to practical machine learning through naive Bayes classification and introduces Jupyter workflows for editing and contributing to the book.

A.8.3 Continuous Uniform

  • The continuous uniform distribution assigns equal probability across [a, b], defines probability density and cumulative distribution functions, and can be sampled by scaling U(0, 1).Arrays of arbitrary shape are sampled with (b - a) * torch.rand(...) + a.Source: §A.8.3 · pp. 1060–1061

A.8.4 Binomial

  • The binomial random variable counts successes across n independent Bernoulli(p) experiments, with its distribution determined by the number and probability of success configurations.Its cumulative distribution is obtained by summing the probabilities of exactly k successes over the relevant values.Source: §A.8.4 · pp. 1062–1063

A.8.5 Poisson

  • The Poisson distribution models event counts such as bus arrivals as the infinite-subdivision limit of binomial trials, with λ representing the expected arrivals per unit time.The construction keeps the mean and variance finite as the number of subintervals grows while the per-interval success probability decreases.Source: §A.8.5 · p. 1065

A.8.6 Gaussian

  • The central limit theorem shows that standardized sums of many independent identically distributed variables converge toward a Gaussian, making Gaussian models fundamental for aggregated measurements.The discussion notes additional conditions such as E[X4] < ∞ and presents the Gaussian as the maximum-entropy choice for fixed mean and variance.Source: §A.8.6 · pp. 1067–1069

A.8.7 Exponential Family

  • The distributions discussed include Bernoulli, discrete and continuous uniform, binomial, Poisson, and Gaussian variables, all of which belong to the exponential family.The exponential-family form uses a base measure h(x), natural parameters η, sufficient statistics T(x), and a cumulant function A(η) that normalizes the density.Source: §A.8.7 · pp. 1070–1071, §A.8.8 · p. 1072

A.9.1 Optical Character Recognition

  • The classifier trains on binarized 28×28 MNIST images by counting class and pixel occurrences, using Laplace smoothing for pixel probabilities.The dataset contains 60,000 training and 10,000 validation images, and smoothing replaces counts niy and ny with niy + 1 and ny + 2.Source: §A.9.1 · p. 1073, §A.9.4 · p. 1076

A.9.3 The Naive Bayes Classifier

  • Naive Bayes reduces model complexity from O(2^dn) to O(dn) by assuming features are conditionally independent given the label.Bayes’ rule then factors p(x|y) into a product of per-feature probabilities, enabling generalization from comparatively modest training data.Source: §A.9.3 · p. 1075

A.9.4 Training

  • Logarithmic computation prevents numerical underflow when multiplying 784 small per-pixel probabilities.For example, 0.1^784 underflows to 0.0, whereas its logarithmic form is -1805.2267129073316.Source: §A.9.4 · p. 1077
  • The Bayes classifier works pretty well on validation examples, but performs relatively poorly because its assumption that pixels are independently generated given the digit is incorrect.The text contrasts this limitation with modern deep networks achieving error rates of less than 0.01, without reporting a numerical accuracy for the Bayes classifier.Source: §A.9.4 · pp. 1078–1079

A.10 Statistics

  • Statistical inference is organized around evaluating and comparing estimators, conducting hypothesis tests, and constructing confidence intervals for a population parameter θ.The discussion assumes θ is scalar but notes that the framework extends straightforwardly to vectors and tensors.Source: §A.10 · p. 1080

A.10.1 Evaluating and Comparing Estimators

Statistical Bias

  • A biased estimator can be practically useful when unbiased estimation is unavailable or intractable, and many practical estimators become unbiased asymptotically as sample size grows.The text characterizes this condition as limn→∞bias(ˆθn) = 0.Source: Statistical Bias · p. 1082

The Bias-Variance Trade-off

  • The mean squared error decomposes into squared bias, variance, and irreducible error, linking underfitting to high bias and overfitting to high variance.Bias reflects systematic estimation error, variance reflects sensitivity to dataset fluctuations, and irreducible error arises from noise in θ itself.Source: The Bias-Variance Trade-off · p. 1083

Evaluating Estimators in Code

  • The code example validates the bias–variance trade-off numerically by showing that an estimator’s MSE agrees with its variance plus squared bias.The simulation uses 10,000 samples from N(θ, σ2) with θ = 1 and σ = 4, and the two quantities agree to numerical precision.Source: Evaluating Estimators in Code · p. 1084

A.10.2 Conducting Hypothesis Tests

  • Hypothesis testing evaluates evidence against a null hypothesis H0 using an alternative hypothesis HA, while experimental design controls confounding through randomized treatment and control groups.Testing requires decisions about metrics, sample size, duration, and whether to compare averages or variability.Source: §A.10.2 · pp. 1084–1085

Statistical Significance

  • Statistical significance controls false-positive risk through α, whereas statistical power 1 −β measures the probability of detecting a real discrepancy when H0 is false.A commonly used significance level is 5%, and 80% is a commonly used power threshold; smaller effects require substantially larger samples.Source: Statistical Significance · p. 1085, Statistical Power · p. 1086

Test Statistic

General Steps of Hypothesis Testing

  • Hypothesis testing proceeds by defining a null hypothesis and risk level, collecting samples with sufficient statistical power, computing a test statistic and p-value, and rejecting the null when the result falls in the reject region.The decision is based on the p-value and significance level α, with applications including clinical trials and A/B testing.Source: General Steps of Hypothesis Testing · p. 1088

Interpretation

  • Confidence intervals quantify uncertainty about a fixed parameter through random intervals, but a 95% interval means 95% of repeatedly constructed intervals would contain the parameter—not that one observed interval has 95% probability of doing so.Narrow intervals do not necessarily imply precise estimation, and values inside an interval are not more likely to be true than values outside it.Source: Interpretation · p. 1089

A Gaussian Example

  • For a Gaussian with unknown mean and variance, the standardized sample mean follows Student’s t with n −1 degrees of freedom, yielding a 95% confidence interval for the mean after rearrangement.For large n, the t statistic is approximately standard Gaussian, while finite samples require wider precomputed t-based intervals.Source: A Gaussian Example · pp. 1089–1090

A.11 Information Theory

  • Information theory provides a language for measuring and comparing information in signals and for discussing information processing in machine-learned systems, including cross-entropy loss.The section frames information theory as concerned with encoding, decoding, transmitting, and manipulating information.Source: §A.11 · p. 1092

A.11.1 Information

  • Information corresponds to surprise: the card examples assign 0 bits, 2 bits, 5.7 bits, and 225.6 bits to increasingly specific outcomes, while an n-digit binary sequence contains n bits.Self-information generalizes this idea by mapping an event probability p to its information using base-2 logarithms.Source: §A.11.1 · p. 1092, Self-information · p. 1093

A.11.2 Entropy

  • Entropy generalizes self-information to random variables by measuring expected information, with a uniquely determined form whose units are normalized so a fair coin flip provides one bit.The logarithm makes information additive for independent variables, the negative sign makes rarer events more informative, and the expectation gives average surprise.Source: §A.11.2 · p. 1094, Motivating Entropy · p. 1094, Definition · pp. 1094–1095

Properties of Entropy

  • Entropy is nonnegative for discrete variables, lower-bounds average coding length, and is minimized by the true distribution because H(X) = −E[log p(x)] ≤ −E[log q(x)], with equality only when P = Q.For continuous variables, entropy can be negative; among distributions supported on a finite interval, the uniform distribution has highest entropy.Source: Properties of Entropy · p. 1096

Joint Entropy

  • Joint entropy captures total information in a pair: it equals one variable’s entropy when X = Y, sums entropies for independent variables, and lies between either entropy and their sum.The section defines discrete and differential joint entropy and implements it from the joint distribution.Source: Joint Entropy · pp. 1096–1097

Conditional Entropy

  • Conditional entropy measures information in Y remaining after observing X; it equals joint entropy minus the information already contained in X, motivating image-label uncertainty analysis.For an image X and class label Y, the label should contain little additional information once a legible image is shown.Source: Conditional Entropy · pp. 1097–1098

Mutual Information

Applications of Mutual Information

  • Pointwise mutual information compares the likelihood of an outcome pair with independent occurrence, while applications use mutual information between context words to resolve whether “Amazon” denotes the company or rainforest.Large positive values indicate pairs occurring more often than chance, and large negative values indicate less frequent co-occurrence.Source: Pointwise Mutual Information · p. 1100, Applications of Mutual Information · pp. 1100–1101

Example

  • KL divergence is non-symmetric: symmetric candidate distributions q1 and q2 yield DKL(p∥q1) and DKL(p∥q2) within 3%, whereas reversing q2 and p differs by about 40%.In the example, DKL(p∥q1) = 8582.0341796875 and DKL(p∥q2) = 8828.3095703125, a 2.8290698237936858% difference; the reversed comparison is 46.18621024399691%.Source: Example · p. 1103

Formal Definition

  • Cross-entropy equals the true entropy plus the KL divergence between the true distribution P and estimated distribution Q.This relationship connects cross-entropy with information-theoretic divergence measures.Source: Formal Definition · p. 1104
  • Cross-entropy is implemented by selecting each example’s predicted probability for its true class, taking its negative logarithm, and averaging across examples.The book also demonstrates an equivalent PyTorch implementation using LogSoftmax and NLLLoss.Source: Formal Definition · p. 1104, Cross-Entropy as An Objective Function… · p. 1106

Properties

  • Maximizing predictive log-probability, minimizing cross-entropy, and minimizing KL divergence are equivalent when the true-data entropy is constant.The equivalence links likelihood optimization with the cross-entropy loss used in classification.Source: Properties · pp. 1104–1105

Cross-Entropy as An Objective Function of Multi-class Classification

  • For multi-class classification, minimizing cross-entropy is equivalent to maximizing the log-likelihood under a multinoulli model.The labels use one-hot encoding, and the predicted class probabilities parameterize the multinoulli distribution.Source: Cross-Entropy as An Objective Function… · pp. 1105–1106

A.11.6 Summary

  • Dive into Deep Learning presents information theory, entropy, KL divergence, and cross-entropy as foundations for understanding classification objectives.Its summary emphasizes that minimizing cross-entropy loss is equivalent to maximizing the log-likelihood function.Source: §A.11.6 · p. 1106

A.11.7 Exercises

  • The book uses exercises to reinforce entropy, KL-divergence nonnegativity, information measures, perplexity, mutual information, and Gaussian KL divergence.The exercises include entropy calculations for random text sources and proofs using Jensen’s inequality.Source: §A.11.7 · pp. 1106–1107

B Tools for Deep Learning

Markdown Files in Jupyter

  • Because Jupyter’s ipynb format stores auxiliary execution data that complicates Git review, contributors can edit the book’s Markdown source directly through the notedown plugin.The setup installs d2l-notedown, loads it when launching Jupyter, or enables it by default through the generated configuration file.Source: §B.1.2 · p. 1111, Markdown Files in Jupyter · pp. 1111–1112

Running Jupyter Notebooks on a Remote Server … B.8.2 Functions

This section explains how to run the book’s notebooks and code locally, remotely, or in the cloud, including hardware selection, GPU execution, updates, and contribution workflows. It also documents reusable utilities, classes, and functions for data handling, visualization, evaluation, training, optimization, and sequence processing.

Running Jupyter Notebooks on a Remote Server

  • SSH port forwarding lets users access Jupyter notebooks running on a remote server through a local browser at http://localhost:8888.Use `ssh myserver -L 8888:localhost:8888`, where `myserver` identifies the remote server.Source: Running Jupyter Notebooks on a Remote… · p. 1112

Timing

  • The ExecuteTime plugin records the execution time of each Jupyter code cell after installation through Jupyter’s contrib-nbextensions commands.The provided installation sequence uses `pip install jupyter_contrib_nbextensions`, installs the user extension, and enables `execute_time/ExecuteTime`.Source: Timing · p. 1112

B.2 Using Amazon SageMaker

  • Amazon SageMaker provides notebook instances for running the book’s GPU-intensive code, with `ml.p3.2xlarge` offering one Tesla V100 GPU and an 8-core CPU.Users can clone the book’s GitHub repository during instance creation, open Jupyter after startup, stop the instance afterward, and update notebooks from its terminal.Source: §B.2 · p. 1113, §B.2.2 · p. 1114, §B.2.3 · p. 1115

B.3 Using AWS EC2 Instances

  • AWS EC2 setup involves requesting a GPU Linux instance, installing CUDA, and installing the deep learning framework and other libraries, with minor adaptations for other clouds.The walkthrough also advises selecting a nearby region, checking instance limits, and using a GPU-enabled framework with suitable drivers.Source: §B.3 · p. 1116, Presetting Location · p. 1117, Increasing Limits · p. 1117

B.3.4 Running the Jupyter Notebook remotely

  • On EC2, users activate the `d2l` environment, run `jupyter notebook`, and open the forwarded local port using the URL printed by Jupyter.When forwarding to port 8889, replace `8888` with `8889` in the displayed URL before opening it locally.Source: §B.3.4 · p. 1123, Figure B.12 · p. 1123

B.3.5 Closing Unused Instances

  • Cloud instances should be closed when unused: stopping preserves restartability but retains disk charges, whereas terminating deletes the disk and associated data permanently.Cloud services bill by usage time, so users should close unused instances and terminate them only when future access is unnecessary.Source: §B.3.5 · p. 1124
  • AWS instances can be launched and stopped on demand, while saved images preserve installed environments such as CUDA for reuse without reinstalling them.Port forwarding enables running Jupyter Notebook on a remote server, but cloud convenience can increase costs.Source: §B.3.6 · p. 1124, §B.3.5 · p. 1124, §B.3.7 · p. 1124

B.4 Using Google Colab

  • Google Colab runs each section’s code and automatically requests a GPU instance when the section needs one.Users open a section with the Colab button, connect to an instance, and may need to click “RUN ANYWAY” on the first code-cell warning.Source: §B.4.1 · p. 1125, §B.4 · p. 1125

B.5 Selecting Servers and GPUs

  • GPU-based deep learning favors cloud computing for larger deployments because GPUs are cheaper and often over an order of magnitude faster than CPUs.A single server can support up to 8 GPUs, while engineering workstations more typically support up to 4 because heat, cooling, and power requirements escalate quickly.Source: §B.5 · p. 1126

B.5.1 Selecting Servers

  • Server construction must balance GPU power, cooling, chassis space, CPU single-thread speed, and PCIe bandwidth to avoid instability, thermal throttling, or reduced data transfer.The guidance includes budgeting up to 350W per GPU, using PCIe 3.0 x16 slots, and checking that multiple GPUs retain full bandwidth.Source: §B.5.1 · p. 1126, §B.5.3 · p. 1130

B.5.2 Selecting GPUs

  • The GTX 1000 series has about twice the performance-to-cost ratio of the 900 series, while RTX 2000 pricing reflects superior FP16, INT8, and INT4 performance.Within each series, price and performance are roughly proportional, and RTX 2000 performance in GFLOPs is an affine function of price.Source: §B.5.2 · p. 1128, Figure B.1 · p. 1129
  • GPU energy consumption scales mostly linearly with computation, while later generations are more efficient; RTX 2000 energy use reflects TensorCores’ disproportionately high power draw.The chapter recommends newer GPU generations when possible and using FP16 or lower precision for efficiency.Source: Figure B.2 · p. 1129, §B.5.2 · p. 1130, §B.5.3 · p. 1130

B.6 Contributing to This Book

  • Readers can contribute fixes through GitHub pull requests, with version control and continuous integration enabling improvements to be incorporated within hours to days rather than years.Minor edits can be made directly in Markdown files, while code changes should be tested in Jupyter Notebook, outputs cleared, and framework-specific blocks marked with %%tab.Source: §B.6 · p. 1130, §B.6.1 · p. 1130, §B.6.2 · p. 1131

B.6.1 Submitting Minor Changes

  • Major contributions use a standard GitHub workflow: install Git, fork the repository, clone the fork locally, edit the book, and submit the changes for review.The instructions cover installing Git, forking and cloning the repository, editing files, and checking modified files with Git.Source: §B.6.3 · p. 1131, Installing Git · p. 1132, Logging in to GitHub · p. 1132

Editing and Pushing

Submitting Pull Requests

  • On GitHub, go to your fork, select “New pull request” to view changes between your edits and the main repository, then click the button to submit it.Source: Submitting Pull Requests · p. 1134

B.6.4 Summary

  • The contribution guidance recommends GitHub contributions, direct editing for minor changes, local fork-based work for major changes, and several small pull requests instead of one large request.Pull requests bundle contributions for review and incorporation.Source: §B.6.4 · p. 1134

B.6.5 Exercises

  • Exercises ask readers to star and fork d2l-ai/d2l-en, submit pull requests for improvements, and learn to create them from new branches.One example improvement is adding a missing reference.Source: §B.6.5 · p. 1135

B.7 Utility Functions and Classes

  • The utility library provides reusable support for saving hyperparameters, plotting progress, processing FrozenLake-v1, visualizing value and policy changes, loading data, computing accuracy, and training models with GPUs.The documented implementations include minibatch stochastic gradient descent, Fashion-MNIST loading, GPU-aware evaluation, and GPU-based training; some functions are marked for deprecation, and Value iteration supports only FrozenLake-v1.Source: §B.7 · pp. 1135, 1137
  • Reusable utilities cover image display, linear regression and squared loss, Fashion-MNIST labels, animated plotting, accumulation, accuracy, downloads, extraction, and tokenization.The functions support visualization, metric computation, dataset access, and basic preprocessing workflows.Source: §B.7 · pp. 1141–1143
  • Machine-translation helpers preprocess and tokenize English-French text, truncate or pad sequences, build arrays with end-of-sequence and padding tokens, and load minibatches with vocabularies.The loader uses reserved padding, beginning-of-sequence, and end-of-sequence tokens and returns iterators plus source and target vocabularies.Source: §B.7 · pp. 1145–1146
  • MaskedSoftmaxCELoss masks irrelevant sequence positions, while train_seq2seq uses Xavier initialization, Adam optimization, teacher forcing, token-normalized loss, gradient clipping, and device-aware training.Training clips gradients with threshold 1 and tracks loss and token throughput during epochs.Source: §B.7 · pp. 1146–1147
  • Sequence-to-sequence inference switches the network to evaluation mode, encodes padded source tokens with valid lengths, initializes decoder state, and can save attention weights.The prediction routine lowercases and tokenizes the source sentence, appends an end-of-sequence token, and constructs a batch axis.Source: §B.7 · p. 1147
  • B.7 Utility Functions and Classes includes autoregressive decoding that selects the highest-likelihood token, records attention weights when requested, and stops at the end-of-sequence token.The generated token sequence is converted to text and returned together with the attention-weight sequence.Source: §B.7 · p. 1148

B.8 The d2l API Document

  • B.8 The d2l API Document lists classes and functions alphabetically, identifies where they are defined in the book, and directs readers to detailed implementations, explanations, and the GitHub source.The API documentation is intended to help readers locate corresponding material in the book and repository.Source: §B.8 · p. 1148

B.8.1 Classes

  • B.8.1 Classes catalogs reusable d2l.torch components for attention, encoder–decoder models, regression, recurrent networks, datasets, and other deep-learning building blocks.The catalog includes interfaces, model descriptions, and selected implementation helpers such as scaled dot-product attention and multi-head transposition.Source: §B.8.1 · pp. 1150, 1151, 1153, 1157
  • The class APIs require subclass implementations of forward computation, while calling the Module instance preserves registered hook execution.The documentation notes that forward recipes belong in the subclass function, but users should invoke the Module instance afterward because direct calls silently ignore hooks.Source: §B.8.1 · pp. 1157, 1159
  • The classes define reusable components for RNN, sequence-to-sequence, Transformer, softmax regression, data loading, optimization, and training workflows.They include high-level and from-scratch RNN language models, sequence-to-sequence encoders and decoders, Transformer encoder components, synthetic and Time Machine datasets, minibatch SGD, gradient clipping, and a base training class.Source: §B.8.1 · pp. 1157–1159

B.8.2 Functions

  • The functions provide utilities for class extension, tensor checks and operations, device selection, CNN and sequence-to-sequence initialization, masked softmax, and visualization.Visualization helpers configure matplotlib, display heatmaps and list-length histograms, and use SVG output for Jupyter plots; device helpers fall back to CPU when GPUs are unavailable.Source: §B.8.2 · pp. 1161–1163

Related papers

Loading 2106.11342v5…