Dive into Deep Learning

Aston Zhang; Zachary C. Lipton; Mu Li; Alexander J. Smola

Source-linked AI summary

Dive into Deep Learning

Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola

arXiv:2106.11342v5Jun 21, 2021cs.LGcs.AIcs.CLcs.CV

TL;DR

Deep learning’s rapid advances across diverse practical fields create a need for an approachable resource that also provides sufficient technical depth and runnable examples. The resource discusses chain-of-thought prompting, using few-shot demonstrations to elicit complex reasoning capabilities in large language models. Vision Transformers outperform ResNets by a large margin on image classification when trained on larger models and datasets such as 300 million images.

Problem
Deep learning’s rapid advances across diverse practical fields create a need for an approachable resource that also provides sufficient technical depth and runnable examples.
Method
The resource discusses chain-of-thought prompting, using few-shot demonstrations to elicit complex reasoning capabilities in large language models.
Results
Vision Transformers outperform ResNets by a large margin on image classification when trained on larger models and datasets such as 300 million images.
Takeaways & Limitations
The resource provides an integrated path from deep-learning concepts and context to practical implementation and discussion.
Takeaways & Limitations
Linear models can fail for image classification because individual pixel significance depends on complex interactions among surrounding pixels.

Abstract

from arXiv · show

This open-source book represents our attempt to make deep learning approachable, teaching readers the concepts, the context, and the code. The entire book is drafted in Jupyter notebooks, seamlessly integrating exposition figures, math, and interactive examples with self-contained code. Our goal is to offer a resource that could (i) be freely available for everyone; (ii) offer sufficient technical depth to provide a starting point on the path to actually becoming an applied machine learning scientist; (iii) include runnable code, showing readers how to solve problems in practice; (iv) allow for rapid updates, both by us and also by the community at large; (v) be complemented by a forum for interactive discussion of technical details and to answer questions.

About This Book … 2 Preliminaries

Dive into Deep Learning is an open, hands-on resource that teaches deep learning concepts, context, mathematics, and code through runnable notebooks. It combines just-in-time explanations, from-scratch implementations, practical framework usage, online distribution, and community discussion while introducing the preliminaries needed for applied work.

About This Book: The book combines textbook-quality exposition with clean, runnable code to provide an up-to-date, technically deep, freely available starting point for applied machine learning.It was designed to support rapid updates and community contributions while integrating concepts, context, and code.
About This Book: Its notebooks interleave mathematics, explanatory text, and executable examples across book, PDF, website, and downloadable formats, supported by custom tooling and a discussion forum.The project addresses the tension between LaTeX, Python, HTML, and JavaScript by assembling its own workflow and sharing source through GitHub.
Learning by Doing: The book teaches concepts just in time through self-contained examples with real datasets, often making tools available before fully explaining them so readers can solve problems quickly.This organization prioritizes practical progress over exhaustive sequential coverage, while introducing fundamental linear algebra and probability preliminaries first.
Learning by Doing: Each concept is taught from scratch, with basic tutorials often presenting both low-level NumPy-like implementations and concise high-level deep learning framework versions.This approach exposes details that frameworks typically hide while retaining practical code for later use.
Summary: The book progresses from basics and preliminaries through modern deep learning techniques to scalability, efficiency, and real-world applications.Its coverage includes numerical prerequisites, regression and classification, CNNs, RNNs, attention mechanisms, optimization, computational performance, computer vision, and language representation models.
2 Preliminaries: The preliminaries establish practical computational foundations, including data manipulation, linear algebra, calculus, probability, notation, and executable Python workflows.Examples emphasize convergence of probability estimates at rate 1/√n, the importance of conditional independence in sequential testing, and the limitation that estimation may converge slowly.
Code: Most sections provide executable code for experimentation, but the book acknowledges that deep learning practice often lacks formal theory sufficient to explain why techniques work.The lightweight d2l package reduces repetition, and the code is primarily based on PyTorch, whose rapid evolution can make print-edition code outdated while the online version is maintained.
Summary: The intended audience includes students, engineers, and researchers with modest linear algebra, calculus, probability, and Python, but no prior machine learning or deep learning background.The resource is designed to lower access barriers through freely available notebooks, online materials, and community support.

1. How does the variance scale with the number of observations? … 5. Why is the K-fold cross-validation error estimate biased?

The supplied passages emphasize practical, modular deep-learning instruction: readers inspect APIs, formulate linear models, optimize them analytically or with minibatch SGD, and connect modeling choices to generalization and implementation tradeoffs.

2.7 Documentation: PyTorch exploration begins with official documentation, tutorials, examples, and module introspection using dir and help.The text recommends ignoring special or internal names, testing interpretations, and consulting source code when documentation is insufficient.
3.1 Linear Regression: Linear regression models numerical targets as weighted sums of features plus a bias, with noise accounting for measurement error.The formulation is expressed both componentwise and with vector or design-matrix notation.
Loss Function: Squared error penalizes large prediction deviations disproportionately, so anomalous data can make the loss excessively sensitive.Training minimizes the aggregate loss over the examples.
Analytic Solution: Linear regression admits an analytic least-squares solution when the augmented design matrix has full rank, but this convenience excludes most deep-learning models.The optimum is unique only when X^⊤X is invertible, equivalently when the design-matrix columns are linearly independent.
Minibatch Stochastic Gradient Descent: Minibatch SGD balances full-batch inefficiency against single-example SGD’s computational and statistical drawbacks by updating from randomly sampled intermediate-sized batches.A minibatch size between 32 and 256 is suggested as a starting point, while learning rate and batch size remain tunable hyperparameters.
3.1.2 Vectorization for Speed: Vectorizing the implementation produces order-of-magnitude speedups while reducing handwritten mathematics, potential errors, and portability problems.The library handles more calculations directly than the slower alternative.
3.1.3 The Normal Distribution and Squared Loss: Mean squared error is equivalent to maximum-likelihood estimation for a linear model with additive Gaussian noise when σ is fixed.The noise-scale-dependent multiplicative constant does not change the optimizer.
Book approach: The book presents deep learning through runnable, modular notebooks that combine explanations, mathematics, figures, and interactive code.Its stated goals include free access, technical depth, practical code, rapid updates, and an interactive discussion forum.

1. What is the problem if we try to design a binary code for it? … 3. How do researchers typically determine the stopping criterion?

The section develops practical deep-learning foundations: numerically stable softmax and cross-entropy, reliable dataset and evaluation practices, and insights into model capacity, optimization, distribution shift, and early stopping. It emphasizes that accessible high-level abstractions should be paired with implementation-level understanding and careful validation.

1. What is the problem if we try to design a binary code for it?: Fashion-MNIST provides a more realistic classification benchmark than MNIST, with 10 apparel categories, 60,000 training images, and 10,000 test images.The dataset uses grayscale images represented as c × h × w tensors and supports minibatch iteration and visualization.
2. What happens if we let the temperature approach 0?: Computing softmax and cross-entropy directly from logits with the LogSumExp trick avoids overflow and underflow while preserving access to output probabilities.The loss computes softmax and its logarithm together rather than receiving softmax probabilities as input.
1. How does this change between GPUs and CPUs?: High-level APIs improve accessibility and concision but can hide numerical hazards, so the book recommends studying both bare-bones and elegant implementations.The concise implementations are generally performant, with convolutions identified as a major exception.
2. What happens if we keep evaluating models on the same test repeatedly?: Test-set error estimates converge at O(1/√n), but finite-sample uncertainty makes small reported improvements difficult to interpret reliably.Approximately 10,000 samples are needed for a 95% confidence interval of ±0.01 asymptotically, while a finite-sample calculation requires roughly 15,000 examples.
3. Should you use it? When would it make sense to?: Distribution shift can make models that perform well on benchmark or synthetic data fail in deployment, as shown by medical sampling bias, unrealistic road textures, and time-dependent shadows.Restricted-assumption methods can sometimes detect and adapt to distribution shift, improving the original classifier’s accuracy.
3. What is the VC dimension of the class of fifth-order polynomials?: Linear models impose restrictive monotonicity and feature-interaction assumptions, making them inadequate for image categories whose meaning depends on surrounding pixels.A single-hidden-layer network can represent any function with enough nodes and appropriate weights, although learning those weights remains difficult.
8. Try out different activation functions. Which one works best?: ReLU improves optimization because its derivatives avoid the widespread vanishing-gradient behavior of sigmoid and tanh activations.Xavier initialization selects a variance based on fan-in and fan-out and works well in practice despite simplifying assumptions in the derivation.
2. Why might early stopping be considered a regularization technique?: Deep networks can fit arbitrary or randomly assigned labels, yet they often learn cleanly labeled examples before memorizing mislabeled ones, motivating early stopping as a practical control.Deep-learning theory still lacks a comprehensive account of why optimization succeeds and why gradient-trained models generalize so well, even though fitting the training data is usually feasible.

4. Why is dropout not typically used at test time? … 4. What is the computational cost for the backpropagation?

The supplied passages span practical regularization and Kaggle workflows, modular neural-network implementation, and convolutional architectures that exploit locality, parameter sharing, and downsampling. They emphasize preprocessing, validation, reusable modules, and reductions in computational complexity, but do not provide direct answers to the listed dropout, memory, dimensionality, or backpropagation questions.

1. Submit your predictions for this section to Kaggle: Kaggle supplies an objective platform for quantitative comparisons and code sharing, while the competition separates labeled training data from unlabeled test data and requires uploaded predictions for evaluation.The training set contains 1460 examples and 80 features, whereas the validation data contains 1459 examples and 80 features.
6.1 Layers and Modules: Neural-network modules provide a recursive abstraction for layers, multlayer components, and entire models, requiring forward propagation and parameter storage while automatic differentiation supplies backpropagation.PyTorch's nn.Sequential maintains an ordered list of constituent modules, enabling compact implementations of networks such as a 256-unit ReLU hidden layer followed by a ten-unit output layer.
3. Why is sharing parameters a good idea?: Tied parameters are represented by the same exact tensor, so modifying one layer's parameters changes the other layer's parameters as well.The passages establish parameter sharing but do not quantify its memory footprint or computational cost.
2. What happens if you specify mismatching dimensions?: Lazy initialization lets frameworks infer parameter shapes from data, simplifying architecture changes and removing a common source of errors.Parameters are initialized when data is passed through the model.
1. When might you want to impose locality and translation invariance for audio?: Convolutional networks reduce parameters by restricting interactions to local neighborhoods and sharing filters across spatial locations, turning otherwise infeasible problems into tractable models.A convolution reduces the parameter count from 10^12 to 4 × 10^6, and locality reduces it further from 4 × 10^6 to 4∆^2, with ∆ typically smaller than 10.
1. What happens if you apply the kernel K in this section to it?: Convolutional operations can detect edges, lines, blur, and sharpen images, while learned filters replace hand-designed feature-engineering heuristics.The supplied examples show cross-correlation detecting white-to-black and black-to-white edges, learning a kernel close to a predefined kernel, and equivalence between strict convolution and cross-correlation after flipping the kernel.
4. What are the computational benefits of a stride larger than 1?: Pooling layers mitigate convolutional sensitivity to location and spatially downsample representations, while padding and stride control output size as repeated convolutions otherwise shrink images.Ten successive 5 × 5 convolutions reduce a 240 × 240 image to 200 × 200 pixels, slicing off 30% of the width and height.

5. Why do you expect max-pooling and average pooling to work differently? … 3. Does causality also apply to text? To which extent?

The supplied passages trace CNN progress from LeNet’s convolutional architecture and pooling-based dimensionality reduction to modern designs that improve accuracy, efficiency, trainability, and scalability. They also introduce language-modeling challenges arising from structured but sparse n-gram statistics and compounding prediction errors.

7.6 Convolutional Neural Networks (LeNet): LeNet combines two convolutional layers with sigmoid activations and average pooling, reducing spatial dimensions while increasing channels before fully connected classification.The reproduced model replaces the original Gaussian activation layer with softmax, while preserving the rest of the LeNet-5 architecture.
8.1.2 AlexNet: AlexNet won the 2012 ImageNet challenge by a large margin, showing that learned features could surpass manually designed computer-vision features.Its eight-layer CNN became practical through GPU-parallelized convolutions and matrix multiplications.
8.3 Network in Network (NiN): Global average pooling removes giant fully connected layers, dramatically reducing parameters without harming accuracy and adding translation invariance on low-resolution representations.Network-in-Network combines 1 × 1 convolutions for channel-wise nonlinearities with global average pooling across spatial locations.
8.4 Multi-Branch Networks (GoogLeNet): GoogLeNet concatenates multi-branch convolutions to select among kernel sizes while simultaneously reducing computation and improving accuracy over predecessor networks.This marks a shift toward deliberate architectural trade-offs between evaluation cost and predictive performance.
8.5 Batch Normalization: Batch normalization accelerates convergence and regularizes optimization, whereas its original internal-covariate-shift explanation is not considered valid.Its effectiveness is especially associated with moderate minibatches, approximately 50–100 examples.
8.6 Residual Networks (ResNet): Residual connections enable much deeper networks by making identity mappings easy to represent and allowing layers to be added while initialized as the identity function.The original ResNet work supported networks with up to 152 layers, while ResNeXt used repeated transformations across independent groups to improve efficiency.
9.1.4 Prediction: Autoregressive predictions can look good for four steps but become nearly useless farther ahead because errors accumulate when each prediction perturbs the next input.Language models similarly resolve ambiguities by preferring plausible sequences, such as “dog bites man” over “man bites dog.”
9.2.5 Exploratory Language Statistics: Language exhibits Zipf-like behavior beyond unigrams, with smaller exponents for longer sequences, limited distinct n-gram inventories, and many rare n-grams.These properties reveal structure in language while making simple counting methods unsuitable for robust language modeling.

2. How would you model a dialogue? … 4. What does the above result mean for gradients in RNNs?

Recurrent neural networks model sequences by maintaining hidden states that summarize prior inputs while reusing parameters across time. Long sequences create sampling, history-length, and gradient challenges, motivating truncated backpropagation, gradient clipping, and gated architectures.

3. What other methods can you think of for reading long sequence data?: Long-sequence training requires deciding how to sample sequence examples and how uniformly to expose tokens, including whether to discard random initial tokens or preserve complete sentences.The supplied exercises identify minibatch sampling and uniform sequence coverage as practical design questions.
2. Which hyperparameter controls the length of history used for prediction?: The history available for prediction is controlled by how recurrent computation and training are organized, while token inputs are represented with one-hot vectors and mapped to vocabulary-sized predictions.The RNN processes sequences step by step, and its language-model output dimension matches the vocabulary size.
2. How would you model a dialogue?: RNNs maintain a hidden state that captures sequence history, enabling character-level language modeling without increasing parameter count as sequence length grows.At each step, the hidden state combines the current input with the previous hidden state, and the output layer predicts the next token.
3. What happens to the gradient if you backpropagate through a long sequence?: Backpropagating through many time steps creates a depth-dependent gradient problem: gradients may vanish or explode, causing difficult optimization, instability, or divergence.Exploding gradients can undo thousands of iterations of progress, whereas vanishing gradients remain a fundamental obstacle for recurrent architectures.
9. Run the code in this section without clipping the gradient. What happens?: Gradient clipping projects gradients onto a radius-θ ball, preserving their direction while bounding their norm and limiting the effect of unusually large minibatches.Clipping is a practical heuristic: it improves robustness but no longer follows the true gradient exactly.
1. Can you make the RNN model overfit using the high-level APIs?: The from-scratch RNN language model generates continuations from user prefixes, while optimized high-level implementations achieve comparable perplexity and run faster.Generation uses a warm-up period to ingest the prefix before feeding predictions back as subsequent inputs.
3. What happens to the gradient if you backpropagate through a long sequence?: Truncated backpropagation through time can act as a mild regularizer because its shorter interaction range and increased gradient variance may be desirable despite less accurate gradients.The supplied passage identifies regular truncation as a strategy whose regularizing effect can benefit models that should learn only short-range interactions.
4. What does the above result mean for gradients in RNNs?: LSTM memory cells and their input and forget gates alleviate vanishing gradients by learning when to preserve or perturb internal state, while GRUs offer similar performance with lighter computation.If the forget gate is 1 and the input gate is 0, the cell state remains constant; gated RNNs also better capture long-distance dependencies than simple RNNs.

1. Can you adjust the hyperparameters to improve the translation results? … 2. Given a powerful language model, what applications can you think of?

The section presents attention and Transformer mechanisms as scalable approaches to sequence modeling, while highlighting language-model applications enabled by prompting and larger-scale pretraining. It also frames optimization as challenging because deep-learning objectives are nonconvex and can suffer from saddle points, local minima, and vanishing gradients.

Can you adjust the hyperparameters to improve the translation results?: Beam search trades off greedy-search efficiency and exhaustive-search optimality through beam size k, with computational cost O(k |Y| T′).Greedy search has cost O(|Y| T′), whereas exhaustive search has cost O(|Y|^T′), making exhaustive decoding prohibitive for large vocabularies and sequences.
Can you design a learnable positional encoding method?: Transformers replace recurrent connections with attention mechanisms and have become dominant models for natural language processing tasks.The chapter proceeds from basic attention intuitions to Transformers, vision Transformers, and pretrained Transformer models.
What challenges can Transformers face if input sequences are very long? Why?: Attention pooling uses concise query-based computation to operate on arbitrarily large key–value databases without changing the operation.The mechanism forms a weighted combination of values, commonly using nonnegative weights that sum to 1.
Are there any other ways to design the output layer of the decoder?: Attention selectively aggregates relevant input information for each prediction instead of compressing the entire sequence into one fixed-length vector.Bahdanau et al. (2014) introduced differentiable attention for sequence-to-sequence prediction, updating the decoder state using relevant input parts.
Given a powerful language model, what applications can you think of?: GPT-2 showed that one language model can support multiple tasks without model updates, making this approach more computationally efficient than fine-tuning.Fine-tuning requires updating model parameters through gradient computation.
Given a powerful language model, what applications can you think of?: GPT-3’s few-shot performance increases most rapidly with larger model size, while Transformer performance follows power-law scaling with parameters, tokens, and compute.Large models also achieve better sample efficiency, requiring fewer training tokens to match the performance of smaller models.
Given a powerful language model, what applications can you think of?: Prompting enables large language models to perform tasks through in-context learning, including complex reasoning with chain-of-thought demonstrations.Sampling multiple reasoning paths, diversifying demonstrations, and decomposing problems can improve reasoning accuracy; simple prompts can also elicit zero-shot chain-of-thought reasoning (Kojima et al., 2022).
Given a powerful language model, what applications can you think of?: Deep-learning optimization is difficult because objectives may contain many local minima, more saddle points, nonconvexity, and vanishing gradients.Minimizing training error does not guarantee minimizing generalization error, while reparameterization and good initialization can help address optimization challenges.

3. What other challenges involved in deep learning optimization can you think of? … 2. How rapid is the rate of convergence for the algorithm?

The section develops convex optimization as a tractable foundation for understanding deep learning optimization, while showing how learning rates, stochasticity, constraints, curvature, momentum, and adaptive scaling affect convergence. It concludes that convex analyses provide useful guidance, but deep learning remains generally nonconvex and requires practical compromises such as decaying learning rates and minibatches.

Why is this hard?: For convex problems, local minima are global minima, and twice-differentiable convexity is characterized by a positive-semidefinite Hessian.These properties make convex objectives easier to analyze and motivate optimization algorithms, even though deep learning objectives are generally nonconvex.
What other challenges involved in deep learning optimization can you think of?: Convex constraints can be handled with Lagrangians, penalties, or projections, but penalties are often more robust in practice and exact optimality properties may fail for nonconvex problems.Lagrange multipliers enforce constraints through saddle-point optimization, while projections map points to the closest points in a convex set.
How rapid is the rate of convergence for the algorithm?: Learning-rate choice creates a fundamental trade-off: small rates can converge slowly, whereas large rates can overshoot or diverge, especially in ill-conditioned directions.For scalar quadratic optimization, convergence occurs when |1 −ηλ| < 1, improves initially with η, and diverges for ηλ > 2.
How rapid is the rate of convergence for the algorithm?: Stochastic gradients are unbiased on average but remain noisy near minima, while decaying learning rates reduce variance yet can still prevent convergence if decay is poorly chosen.Inverse-square-root decay improves convergence after 50 steps, whereas another schedule remains far from x = (0, 0) after 1000 iterations and fails to converge.
How rapid is the rate of convergence for the algorithm?: For convex objectives, stochastic gradient descent converges to the optimum for a wide range of learning rates, but its learning rate must eventually vanish.The convergence rate is O(1/√T), with speed depending on the stochastic-gradient bound L and initial distance r from optimality.
How rapid is the rate of convergence for the algorithm?: Minibatch stochastic gradient descent trades off convergence speed and computation efficiency, and is generally faster than stochastic gradient descent and gradient descent in clock time.Full-batch updates stalled after 6 steps, while minibatches achieved a more favorable balance between per-epoch cost and optimization progress.
Can you exploit this effect also for optimization algorithms?: Momentum accelerates convergence by replacing gradients with a leaky average and expands the feasible convergence range beyond ordinary gradient descent.For momentum, the stated convergence condition is 0 < ηλ < 2 + 2β, compared with 0 < ηλ < 2 for gradient descent.
Can you exploit this effect also for optimization algorithms?: Adaptive methods address uneven or sparse gradients by scaling coordinates using accumulated gradient information, although Adagrad’s learning rate can decay too aggressively while RMSProp avoids this later-stage slowdown.Adagrad is particularly effective for sparse features, whereas RMSProp decouples learning-rate control so variables do not move increasingly slowly in later iterations.

4. Try to construct a case for which Adam diverges and Yogi converges? · 4. How long should warmup last? · 3. How could you measure the cache sizes on a CPU?

The section explains how learning-rate schedules, warmup, compilation, and hardware-aware execution affect optimization, accuracy, generalization, and computational performance in deep learning. It emphasizes that decreasing rates can improve accuracy and reduce overfitting, while warmup can prevent early divergence and computational frameworks can optimize execution.

12.11 Learning Rate Scheduling: Learning-rate magnitude, decay rate, initialization, warmup, and cyclical adjustments are all important considerations when managing optimization.Too-large rates can cause divergence, too-small rates can slow training, and rates that remain large can prevent convergence to optimality.
12.11.2 Schedulers: On the toy Fashion-MNIST problem, scheduling reduced overfitting and produced train loss 0.272, train acc 0.901, test acc 0.883 compared with default training.The scheduler made the learning curve smoother and produced less overfitting than the earlier default setting.
Cosine Scheduler: On Fashion-MNIST, cosine scheduling achieved train loss 0.186, train acc 0.932, test acc 0.901, but improvements from cosine schedules are not guaranteed.The schedule uses a target rate and remains pinned to that rate after the maximum update step.
12.11.4 Summary: Decreasing the learning rate during training can improve accuracy and reduce overfitting, while piecewise decreases after progress plateaus efficiently refine solutions.The book notes that reduced learning rates can produce smoother curves and less overfitting, although the theoretical explanation remains unresolved.
Warmup: A warmup period gradually increases the learning rate before cooling it, preventing early divergence caused by large updates from random initialization.Warmup can be applied with any scheduler and limits parameter divergence in very deep networks.
13.1.1 Symbolic Programming: Symbolic programming can remove Python interpreter bottlenecks, enable compiler optimizations, and make models easier to port beyond Python.The compiler can inspect the full computation, rewrite operations, and release memory when intermediate values are no longer needed.
Acceleration by Hybridization: Compiling an MLP with torch.jit.script preserves its computation result, while the reported timings were 18.1045 sec without torchscript and 20.5523 sec with torchscript.The surrounding text states that scripting an nn.Sequential instance improves computing performance, despite the displayed timing values.
13.2.4 Summary: Deep-learning frameworks improve computational performance through decoupled execution backends, asynchronous command insertion, automatic parallelism, and overlapping computation with communication.Automatic scheduling can make total execution time less than the sum of separate operations, while data parallelism is convenient when GPU memory is sufficient.

5. Rather than being hand-crafted, can non-maximum suppression be learned? · 2. Is it efficient to use matrix multiplications to implement convolutions? Why?

The text presents multiscale object detection through uniformly sampled anchor boxes and layerwise receptive fields, then explains that convolutions can be implemented with matrix multiplications and transposed convolutions exchange forward and backward operations.

14.5 Multiscale Object Detection: For a 561×728 image, generating five anchor boxes per pixel requires labeling and predicting over two million anchor boxes.This motivates reducing the number of anchor boxes considered.
14.5.1 Multiscale Anchor Boxes: Multiscale anchor generation samples centers uniformly from feature-map positions while varying anchor sizes and sampling density across scales.Smaller objects receive more sampled regions and smaller anchors, whereas larger objects receive fewer regions and larger anchors.
14.5.1 Multiscale Anchor Boxes: At successive scales, 4×4 feature maps with scale 0.15 give uniformly distributed centers, scale 0.4 produces overlaps, and scale 0.8 centers an anchor box on the image.Reducing feature-map dimensions while increasing anchor scale supports detection of progressively larger objects.
14.5.2 Multiscale Detection: Deep neural networks leverage layerwise image representations at multiple levels for multiscale object detection.Feature maps with different receptive-field sizes detect objects of different sizes.
14.6 The Object Detection Dataset: The banana detection dataset contains 1000 training examples and 100 validation examples with labeled bounding boxes for varied banana rotations, sizes, and positions.Its labels include object classes and bounding-box coordinates, unlike image-classification labels.
14.7.1 Model: Single-shot multibox detection combines a CNN base network with multiscale feature-map blocks that predict classes and offsets for anchor boxes of different sizes.Higher-level feature maps are smaller, have larger receptive fields, and suit fewer but larger objects; the original model used a truncated VGG network (Liu et al., 2016).
Class Prediction Layer: Using convolutional channels to predict anchor-box classes reduces the heavy parameterization that fully connected classification would require.With hwa anchor boxes at a feature-map scale, fully connected classification can become infeasible.
14.10.4 Summary: Convolutions can be implemented with matrix multiplications, while transposed convolution exchanges the convolutional layer’s forward-propagation and backpropagation functions.Transposed convolution broadcasts input elements through the kernel, producing an output larger than the input; matching hyperparameters can recover the input shape.

2. Can you further improve the accuracy of the model by tuning the hyperparameters? … 1. How does the running time of code in this section changes if not using subsampling?

The supplied passages describe CNN-based neural style transfer as optimizing a synthesized image with pretrained feature representations and weighted losses. They also summarize approximate word-embedding training methods whose per-step cost depends on noise samples or logarithmically on dictionary size.

14.12.1 Method: Style transfer updates only the synthesized image, while a pretrained CNN remains fixed and extracts hierarchical content and style features.The synthesized image is initialized from the content image, and VGG-19 pretrained on ImageNet is used for feature extraction.
14.12.3 Preprocessing and Postprocessing: The implementation preprocesses images by resizing, tensor conversion, RGB standardization, and postprocessing that clamps reconstructed pixel values to [0, 1].The content and style images are resized to 300 by 450 pixels, and the synthesized image is initialized from the content image.
1. How does the output change when you select different content and style layers?: Different CNN layers provide different information: layers near the input capture image details, whereas layers nearer the output capture global information.The method selects content and style layers and retains the VGG computation needed to produce their intermediate outputs.
2. Can you further improve the accuracy of the model by tuning the hyperparameters?: The loss combines content, style, and total variation terms, allowing hyperparameters to balance content retention, style transfer, and noise reduction.Content loss compares content-layer features, style loss compares Gram-matrix representations, and total variation loss reduces high-frequency noise by making neighboring pixels closer.
14.12 Neural Style Transfer: The synthesized image retains the content scene while transferring the style image’s colors, block-like patterns, and brush-stroke texture.The experiment uses a landscape content image and an oil-painting style image.
1. How can we sample noise words in negative sampling?: Negative sampling makes gradient computation independent of dictionary size and linearly dependent on K, the number of noise words sampled per step.The summary characterizes negative sampling as using mutually independent positive and negative events.
1. How does the running time of code in this section changes if not using subsampling?: Hierarchical softmax reduces each training step’s cost to dependence on the logarithm of dictionary size through a binary-tree root-to-leaf path.The cited passage states that the path length is O(log2|V|), reducing cost when the dictionary is huge.

4. How to extend the idea of byte pair encoding to extract phrases? … 3. Can we leverage BERT in machine translation?

The section introduces pretrained word vectors for similarity and analogy tasks, then develops BERT as a context-sensitive, bidirectional, task-agnostic representation model pretrained with masked language modeling and next sentence prediction. BERT supports broad NLP applications with minimal architecture changes and produces different representations for tokens in different contexts.

15.8.1 From Context-Independent to Context-Sensitive: Context-sensitive pretraining emerged because context-independent embeddings assign the same vector to a word regardless of its context, limiting their handling of polysemy and complex semantics.ELMo improved the state of the art across six NLP tasks, while GPT provided a task-agnostic alternative but encoded context only left-to-right.
15.8.3 BERT: Combining the Best of Both Worlds: BERT combines bidirectional context encoding with minimal architecture changes across a wide range of natural language processing tasks.It addresses ELMo’s task-specific architectures and GPT’s left-to-right context limitation while improving the state of the art on eleven tasks.
15.8.4 Input Representation: BERT represents single texts and text pairs using special classification and separation tokens, segment embeddings, token embeddings, and learnable positional embeddings.For text pairs, segment embeddings eA and eB distinguish the two sequences; the input embeddings are the sum of token, segment, and positional embeddings.
15.8.5 Pretraining Tasks: BERT pretraining combines masked language modeling, which predicts randomly masked tokens from bidirectional context, with next sentence prediction, which models relationships between text pairs.Fifteen percent of tokens are selected for masking; selected tokens are replaced by “<mask>” 80% of the time, random tokens 10% of the time, and unchanged tokens 10% of the time.
15.8.6 Putting It All Together: The implementation assembles BERTEncoder, MaskLM, and NextSentencePred into BERTModel, whose forward pass returns token representations, masked-language-model predictions, and next-sentence predictions.The encoder uses Transformer blocks with segment embeddings and learnable positional embeddings.
15.10.2 Representing Text with BERT: BERT representations are context-sensitive: the same token receives different representations when its surrounding context changes.For the polysemous token “crane,” representations differ between the sentence pair “a crane driver came” and “he just left” and another context.

1. What would be the set of states? … 3. What should we have done instead?

The section formulates policies, value functions, and action values for Markov decision processes, then derives dynamic programming and iterative algorithms for optimal control. It also contrasts model-based Value Iteration with model-free Q-Learning, including their convergence and efficiency differences.

17.2.2 Value Function: Value functions decompose expected discounted return into immediate reward plus the next-state value, establishing the dynamic-programming foundation for reinforcement learning.The decomposition holds for every state under the Markov assumption and averages over policy actions and transition outcomes.
17.2.4 Optimal Stochastic Policy: An optimal deterministic policy selects the action maximizing immediate reward plus expected discounted value over possible successor states.The optimal policy is defined among all stochastic policies as the one achieving the largest average discounted return.
17.2.6 Value Iteration: Value Iteration converges to the optimal value function from any initialization, and its FrozenLake implementation reaches the optimum after 10 iterations under reliable transitions.The 4 × 4 environment contains holes, frozen cells, and a goal; the implementation also recovers a policy reaching the goal from every non-hole state.
17.2.7 Policy Evaluation: Policy evaluation uses analogous iterative updates to compute a given stochastic policy’s value function and converges to the correct value from any initialization.The action-value function Qπ(s, a) can be computed analogously.
17.3 Q-Learning: Q-Learning learns value functions without access to the complete MDP, using the robot’s own data and exploration to improve action estimates.Exploration favors actions with large estimated Q values, and Q-Learning can converge to the optimal policy even from a random exploratory policy.
17.3.5 Implementation of Q-Learning: Q-Learning finds the FrozenLake optimum roughly after 250 iterations, whereas Value Iteration requires far fewer because it accesses the full MDP.The comparison attributes the iteration gap to Value Iteration’s knowledge of transition and reward functions.
18.1 Introduction to Gaussian Processes: Gaussian processes specify distributions over functions in function space, combining flexible infinite-parameter models with finite computation and uncertainty that increases away from observed data.Kernel choices control function properties, while exact GP regression inference is available in closed form after learning kernel hyperparameters.
19.1 What Is Hyperparameter Optimization?: Hyperparameter optimization frames validation error or another business metric as a global optimization objective, while random search, multi-fidelity methods, and asynchronous scheduling improve resource use.Random search can outperform grid search when only a subset of hyperparameters matters, and multi-fidelity methods stop poor configurations early.

1. What will happen if we use standard ReLU activation rather than leaky ReLU?

The supplied passage frames an application of DCGAN on Fashion-MNIST, focusing on which categories work well and which do not.

1. What will happen if we use standard ReLU activation rather than leaky ReLU?: Apply DCGAN to the Fashion-MNIST dataset.
1. What will happen if we use standard ReLU activation rather than leaky ReLU?: Evaluate how well different Fashion-MNIST categories perform under DCGAN.
1. What will happen if we use standard ReLU activation rather than leaky ReLU?: Identify which Fashion-MNIST category works well and which does not.

A Mathematics for Deep … A.4.6 Hessians

This appendix develops the mathematical foundations needed to understand modern deep learning, spanning linear algebra, calculus, probability, statistics, and information theory. It emphasizes geometric intuition, eigendecompositions, gradients, and related tools while remaining non-exhaustive.

A Mathematics for Deep: Together, the appendix’s linear algebra, eigendecomposition, calculus, probability, statistics, and information-theoretic concepts form the core mathematical foundation for deep understanding of deep learning.The treatment is designed to make the theory usable in practice while acknowledging that practitioners do not always need complete mathematical foundations.
A Mathematics for Deep: The appendix provides the mathematical background for understanding modern deep learning, especially how architecture and loss choices affect gradient flow and model interpretation.It is intended as a foundation rather than an exhaustive treatment.
A Mathematics for Deep: Probability, statistics, and information theory supply languages for uncertainty, estimator evaluation, hypothesis testing, confidence intervals, and quantitative information measurement.These topics support probabilistic modeling, including the naive Bayes classifier, and help interpret quantities such as bits-per-character.
A.1 Geometry and Linear Algebraic Operations: Vectors are developed geometrically as points and directions, while dot products and angles support high-dimensional similarity reasoning and identify orthogonality.The angle-based view is invariant to rescaling, making it useful when content remains unchanged despite brightness or document-length changes.
A.1.3 Hyperplanes: Hyperplanes divide d-dimensional spaces into two half-spaces and provide the geometric basis for linear classification and decision planes.Deep classifiers can be viewed as learning nonlinear embeddings whose target classes become separable by a final linear layer.
A.1.4 Geometry of Linear Transformations: Matrices transform spaces through coordinate skewing, rotation, and scaling, whereas linear dependence causes compression and rank measures the remaining independent dimensions.Full-rank square matrices are invertible; the supplied eigendecomposition results further identify invertibility with having no zero eigenvalues and rank with the number of non-zero eigenvalues.
A.3 Single Variable Calculus: The calculus chapters develop derivatives and rules for computing changes, use Taylor polynomials as best n-th degree approximations, and extend differentiation to multivariable settings.A numerical example identifies the derivative value as 8 at x = 4, while derivative rules provide flexible tools for evaluating essentially any desired expression.

A.4.7 A Little Matrix Calculus … A.8.2 Discrete Uniform

The appendix develops matrix calculus, integration, probability distributions, and maximum likelihood as practical foundations for machine learning. It emphasizes concise matrix derivatives, the fundamental theorem and change-of-variables methods, distributional summaries, and likelihood-based estimation.

A.4.7 A Little Matrix Calculus: Matrix derivatives are often laborious to derive but yield concise results resembling single-variable calculus, with transposes introduced to match denominator shapes.The denominator-layout convention assembles derivatives in the shape of the denominator, explaining why transposes appear in matrix products.
A.5.2 The Fundamental Theorem of Calculus: The fundamental theorem of calculus reduces definite integration to finding an antiderivative and evaluating its endpoint difference, with additive constants canceling.This replaces numerical chop-and-sum intuition with differentiation-based computation and underlies subsequent integration rules.
A.5.3 Change of Variables: Change of variables transforms integrals by incorporating how the substitution stretches intervals, enabling computation of integrals that are otherwise difficult.In multiple dimensions, the analogous stretching factor is represented by the Jacobian, whose determinant measures volume scaling.
A.5.4 A Comment on Sign Conventions: The appendix presents signed-area conventions: integrals of negative functions or reversed limits are negative, while two reversals cancel.This parallels the determinant’s interpretation as signed area.
A.5.5 Multiple Integrals: Multiple integrals can be evaluated iteratively in either integration order for the continuous functions relevant to machine learning, as formalized by Fubini’s theorem.After discretization, the argument amounts to rearranging sums over small squares, although the result is not universally valid.
Cumulative Distribution Functions: The cumulative distribution function provides one framework for continuous, discrete, and mixed random variables.A mixed example combines a coin flip with either a die roll or a dart-throw distance.
Means and Variances in the Continuum: Probability summaries can expose distribution behavior: a variable may lack finite variance and even a well-defined average, while covariance quantifies dependence.For the covariance example, values are 2 when variables align maximally, −2 when they are flipped, and 0 when unrelated.
A Concrete Example: Maximum likelihood recovers intuitive parameter estimates and extends from discrete probabilities to continuous probability densities.For 9 heads in 13 coin flips, the estimate is θ̂ = 9/13; averaging negative log-likelihood yields a cross-entropy-related performance measure.

A.8.3 Continuous Uniform … Markdown Files in Jupyter

The merged material develops core probability distributions, shows how Naive Bayes makes classification tractable, and introduces statistical inference and information-theoretic tools. It also emphasizes runnable Jupyter-based computation, connecting mathematical definitions with visualization, sampling, estimation, and machine-learning applications.

A.8.3 Continuous Uniform: Continuous uniform variables select values evenly from [a, b], with piecewise density and cumulative distribution functions and scaled sampling from U(0, 1).The section visualizes both functions and samples arbitrary-shaped arrays using (b - a) * torch.rand(...) + a.
B. Tools for Deep Learning / B.1 Using Jupyter Notebooks / Markdown Files in Jupyter: Across these chapters, mathematical exposition is integrated with executable Jupyter code that plots probability functions, samples distributions, processes datasets, and demonstrates statistical identities and machine-learning objectives.The material is designed as an open, runnable, updateable resource combining concepts, context, code, figures, and interactive examples.
A.8.4 Binomial / A.8.5 Poisson: Binomial variables count successes across n independent Bernoulli(p) trials, while Poisson variables model rare-event arrivals as a limiting count process with rate λ.The binomial construction uses sums of Bernoulli variables and combinatorial probabilities; the Poisson rate denotes the expected arrivals per unit time.
A.8.6 Gaussian: The central limit theorem states that standardized sums of many independent identically distributed variables approach a Gaussian distribution, making Gaussian modeling fundamental for aggregated measurements.The Gaussian is also presented as the maximum-entropy, or most conservative, distribution with fixed mean and variance, subject to conditions such as finite fourth moment.
A.8.7 Exponential Family / A.8.8 Summary: Bernoulli, uniform, binomial, Poisson, and Gaussian distributions are unified by the exponential family, whose density uses a base measure, natural parameters, sufficient statistics, and a normalizing cumulant function.The Gaussian is used as a concrete example, and the family is described as widely used in machine learning.
A.10 Statistics / The Bias-Variance Trade-off / A.10.2 Conducting Hypothesis Tests / A.10.3 Constructing Confidence Intervals: The statistics sections introduce estimator evaluation, hypothesis testing, and confidence intervals for inferring population parameters, including the decomposition of mean squared error into squared bias, variance, and irreducible error.Code verifies the bias-variance decomposition numerically, while hypothesis testing frames experimental design and confidence in rejecting unlikely null hypotheses.
A.11 Information Theory / A.11.1 Information / A.11.2 Entropy / A.11.3 Mutual Information / A.11.4 Kullback–Leibler Divergence / A.11.5 Cross-Entropy: Information theory quantifies self-information, entropy, conditional and mutual information, KL divergence, and cross-entropy, with entropy also bounding average code length and cross-entropy aligning with multiclass maximum likelihood.For symmetric comparison distributions, the supplied example reports less than 3% difference between the relevant KL divergences.

Running Jupyter Notebooks on a Remote Server … B.8.2 Functions

The book supports local, remote, cloud, and Colab execution of its Jupyter notebooks, while documenting GPU infrastructure, contribution workflows, and API usage. It emphasizes practical setup, updating, cost control, and community-driven maintenance.

Running Jupyter Notebooks on a Remote Server; B.1.3 Summary; B.1.4 Exercises: Jupyter notebooks can be edited and run locally or remotely through SSH port forwarding, using `ssh myserver -L 8888:localhost:8888` and opening `http://localhost:8888`.Remote access connects the local browser to the server running Jupyter notebooks.
B.2 Using Amazon SageMaker; B.2.1 Signing Up; B.2.2 Creating a SageMaker Instance; B.2.3 Running and Stopping an Instance; B.2.4 Updating Notebooks; B.2.5 Summary; B.2.6 Exercises: Amazon SageMaker provides notebook instances for GPU-intensive code, including an `ml.p3.2xlarge` instance with one Tesla V100 GPU and an 8-core CPU.Users can clone the book’s GitHub repository when creating the instance, open Jupyter after startup, update notebooks through its terminal, and stop the instance to avoid further charges.
B.3 Using AWS EC2 Instances; B.3.1 Creating and Running an EC2 Instance; Presetting Location; Increasing Limits; Launching an Instance; Connecting to the Instance; B.3.2 Installing CUDA; B.3.3 Installing Libraries for Running the Code: AWS EC2 setup involves requesting a GPU Linux instance, installing CUDA, and installing the deep learning framework and other libraries.The instructions cover selecting regions, checking instance limits, choosing an Ubuntu AMI and GPU configuration, securing SSH keys, connecting, installing CUDA 12.1, and configuring the library path.
B.3.4 Running the Jupyter Notebook remotely; B.3.5 Closing Unused Instances; B.3.6 Summary; B.3.7 Exercises: EC2 notebooks run through SSH port forwarding after activating the environment and launching Jupyter, with the displayed URL adapted to the forwarded local port.Unused instances should be stopped or terminated: stopping retains data but still incurs disk charges, whereas termination deletes associated data.
B.4 Using Google Colab; B.4.1 Summary; B.4.2 Exercises: Google Colab lets users run each book section by clicking its Colab button, automatically requesting a GPU instance when a section needs one.The workflow includes dismissing the first-run warning and connecting Colab to an execution instance.
B.5 Selecting Servers and GPUs; B.5.1 Selecting Servers; B.5.3 Summary: GPU selection and server construction should account for compute power, memory, bandwidth, power supply, cooling, PCIe lanes, CPU single-thread speed, and physical compatibility.The text notes that GPUs are generally more cost-effective than CPUs for deep learning, while multi-GPU systems require progressively stronger power, cooling, memory, and interconnect support.
B.5.2 Selecting GPUs: Within GPU comparisons, newer generations improve cost effectiveness: the GTX 1000 series has about twice the performance-to-cost ratio of the 900 series, while RTX 2000 performance in GFLOPs is an affine function of price.The text also reports that energy consumption scales mostly linearly with computation and later generations are more efficient, with RTX 2000 behavior affected by TensorCores.
B.6 Contributing to This Book; B.6.1 Submitting Minor Changes; B.6.2 Proposing Major Changes; B.6.3 Submitting Major Changes; Installing Git; Logging in to GitHub; Cloning the Repository; Editing and Pushing; Submitting Pull Requests; B.6.4 Summary; B.6.5 Exercises; B.7 Utility Functions and Classes; B.8 The d2l API Document; B.8.1 Classes; B.8.2 Functions: The open-source book supports reader contributions through GitHub edits, pull requests, runnable notebook tests, and framework-specific code markers, enabling corrections and updates within hours to days.The supplied passages describe editing markdown source files, proposing changes, testing code in Jupyter, clearing outputs before submission, and using `%%tab` for framework implementations.