PaperlayerView exact revision on arXiv

Source-linked AI summary

Introduction to Computer Vision

Stan Birchfield

arXiv:2609.39627v1cs.CV

TL;DR

A prior computer-vision book was somewhat obsolete by publication because deep learning had eclipsed nearly all older methods. This book presents a code-first treatment with checked implementations; its Laplacian-pyramid example reconstructs the original image with a maximum error of 0 gray levels.

  • Problem

    A prior computer-vision book was somewhat obsolete by publication because deep learning had eclipsed nearly all older methods.

    Source: Preface · p. 9

  • Method

    The book teaches through direct image-array operations and from-scratch algorithms checked against library functions, as illustrated by its Hough-line accumulator.

    Source: Colors are typically in RGB order · p. 13, The accumulator: voting for lines · p. 57

  • Results

    0 gray levels maximum reconstruction error was reported when Laplacian-pyramid levels were summed to reconstruct the original image, up to integer rounding.

    Source: Exact reconstruction · p. 64

  • Takeaways & Limitations

    An equivalent-ellipse representation provides a compact five-number summary of arbitrarily shaped binary blobs using area, centroid, and second-moment spread.

    Source: §6.4 · p. 31

    Exact reversibility with a Laplacian pyramid requires retaining all levels, rather than only the smallest one.

    Source: Exact reconstruction · p. 65

Abstract

from arXiv · show

This book presents a code-first introduction to computer vision, spanning classical 2D image processing, classical 3D vision, and deep learning. Organized as 44 short chapters across three parts, the book builds each topic from first principles: image arithmetic and morphology; convolution, pyramids, and frequency-domain filtering; feature detection, optical flow, and stereo; projective geometry, camera calibration, and structure from motion; and the full arc of modern deep learning, from a single neuron through convolutional networks, backpropagation, classic architectures, transfer learning, object detection, and semantic and instance segmentation, concluding with engineering considerations like mixed-precision and parallel training. Every technique is implemented directly in Python and NumPy or PyTorch and checked numerically against the corresponding OpenCV or PyTorch library function, so readers see not just the mathematics but its concrete behavior on real and synthetic data. The material was distilled with AI assistance from freely available online course notes, condensing extensive working code into concise mathematical exposition while preserving verified, reproducible results throughout. It is intended as a self-contained reference for students and practitioners who want to understand computer vision algorithms and their Python implementations.

Classical 2D Image Processing … Exercises

The section introduces images as arrays and demonstrates how Python imaging operations manipulate pixels, channels, and regions. It then covers arithmetic, thresholding, and morphology, emphasizing behaviors and pitfalls that affect results.

Images as Arrays

  • An image is an array: grayscale pixels range from 0 to 255, while color images use (height, width, channels) with rows as y, columns as x, and the origin at top left.NumPy indexing follows (y, x), but many OpenCV point and size arguments use (x, y), so argument order must be checked.Source: §1.1 · p. 11, §1.2 · p. 12
  • NumPy slicing selects image rows or columns, and a range such as 0:1 preserves a one-row image’s axis, unlike integer indexing.Matplotlib expects RGB, whereas OpenCV loads color images as BGR; reverse the channel axis or use cv2.cvtColor with cv2.COLOR_BGR2RGB before display.Source: Colors are typically in RGB order · p. 13, §1.3 · pp. 13–14

Image Arithmetic

  • Naive NumPy arithmetic on uint8 images wraps out-of-range values, while OpenCV arithmetic saturates by clamping results to 0–255.For example, 220 + 80 wraps to 44 in uint8 rather than clamping to 255.Source: Image Arithmetic · p. 15, The problem: uint8 overflow · pp. 15–16, The fix: saturating arithmetic · p. 16
  • A binary mask can restrict brightening to selected pixels, either by copying only masked results or by passing the mask to cv2.add.The unmasked portion of the destination remains untouched when cv2.add writes only within the masked region.Source: §2.2 · p. 16
  • cv2.addWeighted computes a saturated weighted sum for cross-dissolving images, while absolute differences highlight where two images changed.Difference images are zero where nothing changed and brighter where content differs; they can support simple motion or change detection.Source: §2.3 · p. 17, §2.4 · p. 17, Figure 2.5 · p. 18

Thresholding, Erosion, and Dilation

  • Thresholding turns pixels above a cutoff white and all others black; Otsu selects a cutoff by minimizing within-class variance, but can fail on poorly separated or multimodal histograms.OpenCV’s cv2.threshold implements binary thresholding and also offers inverted output, capping, and automatic cutoff selection.Source: §3.1 · p. 19, Otsu’s method: choosing the threshold… · p. 20
  • Erosion shrinks white regions and removes small specks, while dilation grows them and fills small black holes.Each operation uses a structuring element: erosion retains white pixels only when the full kernel fits, whereas dilation needs any overlap.Source: §3.2 · p. 21
  • Opening applies erosion followed by dilation to remove small white specks while restoring the main region’s size.The section presents cv2.morphologyEx with cv2.MORPH_OPEN as a common way to denoise a thresholded image.Source: §3.2 · p. 21

Flood Fill and Connected Components … 6.4 It works on arbitrary shapes, too

The sections present methods for finding and filtering blobs, then use image moments and covariance eigenvectors to summarize their shape, orientation, and size. Hu moments provide a descriptor that is nearly invariant to position, scale, and rotation, while the equivalent-ellipse summary applies to arbitrary binary shapes.

4.1 Flood fill from a seed point

  • Flood fill grows from a seed pixel through connected pixels within a tolerance of the seed value, painting the region a new color.The classic algorithm keeps a frontier of pixels still to visit.Source: §4.1 · p. 22

4.2 Connected components: label every blob at once

  • Connected-component labeling scans the image to assign each blob a unique integer label, and union-find uses two image passes plus an equivalence-table traversal.It also returns each blob’s bounding box, area, and centroid with almost no extra computation.Source: §4.2 · p. 23

4.3 Filtering blobs by size

  • Thresholding connected-component area statistics can discard small, noise-like blobs while retaining blobs above a chosen cutoff.The illustrated cutoff discards the two smallest apples.Source: §4.3 · p. 23, Figure 4.4 · p. 24

Image Moments; 5.1 What is a moment?; 5.3 Area and centroid from moments; 5.4 Central moments

  • For a binary image, raw moments are weighted sums of foreground pixel coordinates; moments can summarize area and centroid, while central moments measure coordinates relative to the blob’s centroid.A shift-of-origin identity computes central moments from raw moments already in hand instead of revisiting every pixel.Source: §5.1 · p. 25, Image Moments · p. 25, §5.4 · p. 26

5.2 An elongated, rotated blob; 5.5 Orientation from central moments

  • The central moments µ20, µ02, and µ11 describe spread around the centroid, and their covariance matrix gives the blob’s major-axis orientation.The eigenvalues of the normalized covariance matrix give the equivalent ellipse’s axis lengths.Source: §5.5 · pp. 26–27

5.6 Hu moments: a shape descriptor invariant to pose

  • Translated and rotated-and-scaled dog silhouettes yield similar log-scaled Hu moments to the original, while a cat silhouette yields markedly different values.The seven Hu moments are calculated from central moments and are nearly invariant to shape position, size, and rotation, supporting shape comparison without alignment.Source: §5.6 · p. 27

Eigenvectors and Eigenvalues of the Moment Covariance Matrix; 6.1 Refresher: what are eigenvectors and eigenvalues?; From one eigenvector to both at once: diagonalization; 6.2 From central moments to a covariance matrix; 6.3 Eigen decomposition gives orientation and size

  • Diagonalizing a blob’s 2×2 moment covariance matrix gives eigenvectors along its major and minor axes and eigenvalues measuring spread; for an ellipse, λmax = a^2/4 and λmin = b^2/4.For the tested ellipse, recovered semi-axes a = 70.6 and b = 25.5 match the drawn semi-axes (70, 25); covariance eigenvectors ranked by eigenvalue also define PCA directions for general data.Source: Eigenvectors and Eigenvalues of the… · p. 29, §6.2 · p. 30, §6.3 · p. 31

6.4 It works on arbitrary shapes, too

  • Any binary shape can be summarized by an equivalent ellipse matching its area, centroid, and second-moment spread, using five values: center, two semi-axis lengths, and orientation.The shape need not itself be elliptical.Source: §6.4 · p. 31

6.5 Eccentricity: how elongated is a shape? … 9.1 The problem with forward mapping

These sections move from measuring shape elongation, pixel distances, and curve lengths to applying geometric transforms and explaining why forward mapping leaves gaps. They pair concise mathematical descriptions with concrete image-processing procedures and examples.

6.5 Eccentricity: how elongated is a shape?

  • The ratio of a shape’s principal axes gives an orientation- and size-independent elongation measure, ranging from 0 for a circle to nearly 1 for a thin shape.The glasses example has eccentricity 0.911, consistent with an elongated shape; its equivalent ellipse summarizes mass distribution rather than tracing the outline.Source: §6.5 · p. 31, Figure 6.3 · p. 32

7.1 Point-to-point distance metrics

  • Manhattan (L1), chessboard (L∞), and Euclidean (L2) distances agree along grid axes but diverge elsewhere, producing diamond-, square-, and circle-shaped distance fields.They correspond respectively to axis-only movement, king-like moves, and straight-line distance.Source: §7.1 · pp. 33–34, Figure 7.1 · p. 34

7.2 Estimating curve length from a chain code

  • For a radius-60 circle, the corrected chain-code estimate is 375.4 (−0.4% error), versus Freeman’s 395.6 (+4.9%), against a true circumference of 377.0.The correction uses tuned even- and odd-step weights to reduce the systematic overestimate that staircase-like digitized curves can produce.Source: §7.2 · pp. 34–35

7.3 Chamfer distance transform

  • The chamfer distance transform approximates each pixel’s distance to the nearest selected set using two raster-scan passes, with mask size trading accuracy for speed.The 3-4 chamfer method uses axis and diagonal neighbor weights of 3 and 4, then divides by 3; mask sizes 3 and 5 agree closely, with maximum error under half a pixel.Source: §7.3 · p. 35, Figure 7.2 · p. 35

Geometric Transformations; 8.3 The transformation hierarchy

  • Euclidean, similarity, and affine transforms form a nested hierarchy: increasing flexibility progressively gives up preservation of lengths and angles, while affine transforms preserve parallelism.Euclidean transforms preserve lengths and angles; similarity transforms preserve angles and length ratios; affine transforms can shear a square into a parallelogram.Source: Geometric Transformations · p. 37, §8.3 · p. 39

8.1 A test image with no symmetry; 8.2 Basic operations

  • Flips, crops, resizing, and rotation provide basic image operations, illustrated using an asymmetric letter F to make the effects distinguishable.Flipping uses cv2.flip, cropping uses NumPy slicing, resizing uses cv2.resize, and rotation uses cv2.getRotationMatrix2D with cv2.warpAffine.Source: §8.1 · p. 37, §8.2 · pp. 37–38

Fitting an affine transform from point correspondences

9.1 The problem with forward mapping

  • In forward mapping, a 25◦ rotation combined with a 1.6× scale leaves visible holes: only 1408 of 3600 destination pixels (39%) receive a value, because transformed source locations leave gaps.Source: §9.1 · pp. 42–43

9.2 The fix: inverse mapping … Exercises

The sections explain inverse-mapped image warping, convolution and filtering, and blur-before-downsampling Gaussian pyramids. They develop each operation from first principles and compare implementations with OpenCV or NumPy, while noting interpolation trade-offs, aliasing, and irreversible information loss.

Nearest-neighbor interpolation; Bilinear interpolation; Exercises

  • Nearest-neighbor inverse mapping avoids holes but looks blocky when enlarging, whereas bilinear interpolation blends four neighbors for smoother, slightly blurrier edges with fewer aliasing artifacts.Exercises ask readers to consider forward-mapping under shrinkage, extend bilinear sampling to bicubic interpolation, and time the implementation against OpenCV.Source: Nearest-neighbor interpolation · p. 43, Bilinear interpolation · p. 43, Figure 9.3 · p. 44

9.2 The fix: inverse mapping; Sanity check against OpenCV

  • Inverse mapping fills every destination pixel, and bilinear sampling agrees closely with cv2.warpAffine: maximum pixel difference is 1 (out of 255), and mean difference is 0.092.OpenCV’s small residual difference comes from fixed-point rounding rather than full floating-point arithmetic.Source: §9.2 · p. 43, Sanity check against OpenCV · p. 44, Figure 9.5 · p. 45

Convolution; 10.1 1D convolution; 10.2 2D convolution

  • Convolution flips a kernel before sliding it across a signal or image; correlation omits the flip, so asymmetric kernels differ, although symmetric filters produce identical results.The scratch 1D derivative example matches np.convolve exactly; OpenCV filter2D computes correlation, so convolution checks use a pre-flipped kernel.Source: §10.1 · p. 46, §10.2 · p. 47

Convolution from scratch; 10.3 Boundary handling; 10.4 Classic filters as kernels; Separable kernels: a speed trick; Exercises

  • The from-scratch 2D convolution matches cv2.filter2D with a pre-flipped kernel exactly, while padding choices affect borders and classic filters apply different kernels for blur, sharpening, and edges.A separable Gaussian reduces per-pixel work from O(n^2) to O(2n); exercises ask readers to verify separability and time implementations.Source: Convolution from scratch · p. 47, §10.3 · pp. 47–48, §10.4 · p. 48

Smoothing and Gaussian Pyramids; 11.1 Why downsampling needs blurring first; 11.2 Choosing sigma and kernel size; Kernel size has to keep up with sigma

  • Naive downsampling can turn fine repeating detail into aliasing, and a Gaussian kernel must be large enough for its sigma: a 3-tap kernel at sigma=5.0 becomes nearly a box average.That kernel’s weights are [0.331, 0.338, 0.331], while a 3×3 kernel differs from the correctly auto-sized kernel by 9.31 mean absolute gray levels.Source: §11.1 · p. 50, Kernel size has to keep up with sigma · p. 51, Figure 11.3 · p. 51

11.3 Building a Gaussian pyramid

  • A Gaussian pyramid repeatedly blurs and downsamples by two, producing anti-aliased coarser views rather than mere smaller crops.The described five-level example spans 256 × 256 to 16 × 16 pixels.Source: §11.3 · p. 52, Figure 11.4 · p. 52

Comparison to OpenCV’s built-in cv2.pyrDown

  • cv2.pyrDown uses a fixed 5-tap binomial kernel for blur-then-downsample; its output sizes match the custom pyramid, with mean absolute pixel difference rising to 0.19 at 16 × 16.Source: Comparison to OpenCV’s built-in… · p. 52

11.4 Pyramids lose information; Exercises

  • Upsampling a downsampled pyramid cannot restore discarded detail, and the resulting gap from the original is what a Laplacian pyramid captures.Exercises ask readers to inspect reconstruction error, aliasing at a smaller downsampling factor, and how pyramid levels affect recognizable image detail.Source: §11.4 · p. 52, Exercises · pp. 52–53, Figure 11.5 · p. 53

Differentiation and Edge Detection … 13.4 Laplacian pyramids: recovering what Gaussian pyramids throw away

The section develops first- and second-derivative methods for detecting image edges and shapes, then shows how Laplacian pyramids retain detail lost by Gaussian pyramids. It explains the role of smoothing and voting, and gives numerical checks of the methods against OpenCV or their underlying constructions.

12.1 From derivatives to gradients; 12.2 Prewitt, Sobel, and Scharr: derivative + smoothing; Noise robustness: why the perpendicular smoothing matters; Exercises

  • Gradient kernels combine differencing with perpendicular smoothing to reduce noise; Sobel approximates Gaussian smoothing, while Scharr optimizes rotational accuracy of gradient direction.A plain central difference is maximally noise-sensitive without perpendicular smoothing; the exercises suggest testing whether kernel ordering changes as noise increases.Source: §12.2 · pp. 54–55, Noise robustness: why the… · p. 55, Exercises · p. 58

12.3 From gradients to edges: the Canny detector; Threshold sensitivity; Exercises

  • Canny smooths before computing Sobel gradients, thins responses with non-maximum suppression, and uses double thresholds with hysteresis to keep connected weak edges; threshold choice trades noise rejection against faint-edge sensitivity.The exercises also propose comparing aperture sizes and testing when a one-pixel gap prevents hysteresis from linking edge segments.Source: §12.3 · p. 56, Threshold sensitivity · p. 56

12.4 From edges to lines: the Hough transform; The accumulator: voting for lines; cv2.HoughLinesP: the practical version; Exercises

Beyond lines: Hough circles; Exercises

  • For a circle centered at (100, 100) with radius 50, cv2.HoughCircles reports (98.5, 100.5) and radius 50.0, while a from-scratch 3D vote accumulator peaks at (100, 100) with radius 51.The exercises describe lowering cv2.HoughCircles’ vote threshold to observe false positives.Source: Beyond lines: Hough circles · p. 58, Exercises · pp. 59–60

Second Derivatives and the Laplacian; 13.1 Why zero crossings? A 1D intuition; 13.2 The Laplacian: a 2D second derivative; 13.3 The Marr-Hildreth detector: Laplacian of Gaussian (LoG)

  • A step edge is a zero crossing of the second derivative, and the 2D Laplacian sums unmixed second partials; its 4-neighbor kernel matches cv2.Laplacian with ksize=1 exactly.Because second derivatives amplify high-frequency noise, Marr-Hildreth smooths first and detects edges at zero crossings of the resulting Laplacian-of-Gaussian response.Source: §13.1 · p. 61, §13.2 · p. 62, §13.3 · p. 62

The cheaper approximation: Difference of Gaussians (DoG)

  • A scaled Difference of Gaussians correlates 0.967 with the true Laplacian of Gaussian, offering a cheaper but approximate alternative that is also used by SIFT for multiscale keypoint detection.The approximation uses two Gaussian blurs and a subtraction rather than an explicit second-derivative kernel.Source: The cheaper approximation: Difference… · p. 63

13.4 Laplacian pyramids: recovering what Gaussian pyramids throw away

  • A Laplacian pyramid stores each Gaussian level’s difference from the upsampled coarser level, preserving detail otherwise lost through blurring and downsampling; its smallest level stores image content.The difference is a discrete approximation of the Laplacian, and the illustrated five-level pyramid shows detail concentrated at edges.Source: §13.4 · pp. 63–64, Figure 13.5 · p. 64

Exact reconstruction … 16.2 From Haar to Daubechies: smoother wavelets

The sections develop nonlinear filters, frequency-domain filtering, and localized wavelet representations, explaining their mechanisms and limitations through comparisons with library implementations. They also show when image or signal reconstruction is exact and how filter choices affect noise, edges, and frequency content.

Exact reconstruction; Exercises

  • 0 gray levels maximum error and 0.0000 mean error show that the Laplacian pyramid reconstructs the original image exactly up to integer rounding.It stores the information discarded at each Gaussian-pyramid level, but exact reversibility requires retaining all levels; an exercise proposes blending Laplacian pyramids to avoid sharp seams.Source: Exact reconstruction · pp. 64–65, Figure 13.6 · p. 64, Exercises · p. 65

Nonlinear Filters; 14.1 Median filtering: robust to outliers; 14.2 Min and max filters: grayscale erosion and dilation

  • A median filter reduces mean absolute error on a noisy image from Gaussian blur’s 9.56 gray levels to 0.15, because isolated outliers barely affect the neighborhood median.Grayscale erosion and dilation instead take neighborhood minima and maxima, shifting rather than blurring edges; a from-scratch 3 × 3 min filter matches cv2.erode exactly.Source: §14.1 · p. 66, §14.2 · pp. 66–67

14.3 The problem linear blur can’t solve: smoothing without crossing edges; Edge preservation, side by side; Exercises

  • The bilateral filter weights neighbors by both spatial proximity and intensity similarity, smoothing similar pixels while down-weighting pixels across edges.On a noisy step edge it removes noise while preserving the sharp intensity jump; its OpenCV implementation differs by at most 3.40 gray levels and averages 0.56 absolute difference, and exercises probe parameter effects and runtime.Source: §14.3 · pp. 67–68, Figure 14.2 · p. 68, Exercises · p. 68

The Fourier Transform and Frequency-Domain Filtering; 15.1 1D warm-up: a signal as a sum of sinusoids; 15.2 The 2D Fourier transform of an image; A grating reveals its frequency directly as a spectrum peak

  • Fourier analysis represents signals as sums of sinusoids and images as 2D frequency components, with real-valued gratings producing symmetric spectrum peaks that reveal their frequency and orientation.The 2D DFT is computed with np.fft.fft2; log magnitude and shifting the zero-frequency term to the center aid visualization, while a square wave is approximated more closely with additional harmonics.Source: The Fourier Transform and… · p. 70, §15.1 · p. 70, §15.2 · p. 71

15.3 Filtering by editing the spectrum; The cost of a sharp cutoff: ringing; 15.4 The convolution theorem; Exercises

  • Frequency-domain masks implement low-pass blurring and high-pass edge enhancement, but sharp cutoffs cause ringing, whereas smoothly falling Gaussian masks avoid it.The convolution theorem equates spatial convolution with elementwise multiplication of Fourier transforms; an exercise extends masking to band-pass filtering.Source: §15.3 · p. 71, The cost of a sharp cutoff: ringing · p. 72, Figure 15.6 · p. 73

A catch: the DFT gives circular convolution; Exercises

  • 1.14×10−13 maximum absolute difference confirms that frequency multiplication matches spatial convolution with wraparound borders, because the DFT computes circular convolution.Ordinary convolution agrees only with matching wraparound handling; FFT filtering is most useful for large kernels, since small kernels are already fast spatially and FFT overhead can outweigh its benefit.Source: A catch: the DFT gives circular… · pp. 73–74, Exercises · p. 74

Wavelets and Gabor Filters; 16.1 The Haar wavelet: the simplest possible wavelet

  • The Haar wavelet separates local averages from pairwise differences, resembling a Laplacian-pyramid blur-and-residual representation, and a manual NumPy transform matches pywt.dwt exactly.Wavelets and Gabor filters use spatially localized oscillations to represent both where a feature occurs and its frequency, unlike global sine-wave bases.Source: Wavelets and Gabor Filters · p. 75, §16.1 · p. 75

16.2 From Haar to Daubechies: smoother wavelets

  • Daubechies wavelets use longer, smoother filters and more vanishing moments than discontinuous Haar, trading wider spatial support for better frequency behavior.The four-tap db2 decomposition matches pywt, and its inverse reconstructs the original signal perfectly.Source: §16.2 · p. 75, Figure 16.1 · p. 76

16.3 2D wavelet decomposition of an image … Why this formula is outdated (even though widely used, and still useful)

The sections explain wavelet and Gabor representations, image compression methods and tradeoffs, and color-space choices. They show where standard formulas and techniques are useful, and where their assumptions or limitations matter.

16.3 2D wavelet decomposition of an image; Exercises

  • A one-level 2D wavelet transform applies row and column passes to produce LL, LH, HL, and HH subbands; LL is a half-resolution low-pass image, while the others capture directional detail.The decomposition is exactly invertible: pywt.idwt2 reconstructs the original with maximum error 1.4×10^-13. Exercises extend this to db4 filters and a second LL decomposition level.Source: §16.3 · p. 76

16.4 Gabor filters: localized oscillations tuned to orientation; Exercises

  • Gabor filters localize sinusoidal patterns in space and tune them to frequency and orientation, making them useful for extracting oriented texture and edges at a chosen scale.Their resemblance to simple-cell receptive fields motivated their use for orientation-selective feature extraction; exercises vary wavelength to explore stripe count and texture scale.Source: §16.4 · pp. 76–77, Exercises · p. 78

Image Compression — Lossless and Lossy; Orientation selectivity, demonstrated quantitatively; Exercises

17.1 Lossless compression; Run-length encoding (RLE); Exercises

  • Run-length encoding stores consecutive identical pixels as value–length pairs, reconstructing an 80 × 80 flat-rectangle image exactly while reducing its 6400-byte raw size to roughly 369 bytes, or 17.3×.RLE suits large flat regions such as graphics, text, and masks, but does little for real photographs; an exercise tests its behavior on noisy images.Source: Run-length encoding (RLE) · p. 79, Exercises · p. 82

Huffman coding: fewer bits for common values

  • Huffman coding achieves 5.20 bits/pixel on a synthetic noisy image, near the 5.17-bit Shannon entropy floor and below naive fixed-width encoding’s 8.00 bits/pixel.It assigns shorter variable-length codes to common values and longer ones to rare values; PNG also uses predictive filtering to encode lower-entropy prediction errors.Source: Huffman coding: fewer bits for common… · pp. 79–80

17.2 Lossy compression: the DCT; Quantization: throwing away the coefficients that matter least; A closer look at JPEG artifacts; Exercises

  • JPEG transforms each 8 × 8 block with the DCT, which concentrates information in low frequencies; quantization zeros many high-frequency coefficients, while aggressive settings produce blocking and ringing.Keeping just the top-left 2×2 coefficients retains most overall structure, and exercises contrast this crude truncation with gradual coefficient quantization.Source: §17.2 · p. 80, Quantization: throwing away the… · p. 80, Figure 17.1 · p. 80

17.3 Real JPEG: the rate-distortion tradeoff; 17.4 JPEG or PNG? Choosing the right format

  • JPEG is 7.7× smaller than PNG on a photograph, while PNG is roughly 1.9× smaller than JPEG on a synthetic graphic; JPEG quality 75 reduces typical photo size by 6–8×.The rate-distortion curve flattens at high quality, so additional bytes yield diminishing quality improvements.Source: §17.4 · pp. 81–82, §17.3 · p. 81

Color Spaces; 18.1 Grayscale is already a color-space choice; Why this formula is outdated (even though widely used, and still useful)

  • Rec. 601’s 0.299, 0.587, 0.114 weights reflect 1953 CRT primaries, while Rec. 709/sRGB uses 0.2126, 0.7152, 0.0722; cv2.cvtColor retains Rec. 601 for compatibility.Because cv2.cvtColor applies weights to gamma-encoded values without decoding, blue (0, 0, 255) becomes gray 29; this is fast and often adequate for computer vision, but not physically correct luminance for calibration or photometry.Source: Why this formula is outdated (even… · pp. 84–85, Color Spaces · p. 84, §18.1 · p. 84

18.2 RGB channels are highly correlated … How many matches are actually correct?

This block moves from color representation through clustering to feature matching, showing how choices of distance, cluster shape, and feature scale affect segmentation and correspondence. Its final matching check tests SIFT matches against a known image transform.

18.2 RGB channels are highly correlated; 18.3 HSV: separating color from brightness

  • HSV hue thresholding recovers a shaded orange disk perfectly (IoU 1.000), while a fixed RGB range misses its darkened side (IoU 0.710).Real-image shading commonly scales RGB channels together, making them strongly correlated and less convenient for separating color from brightness.Source: §18.3 · p. 85, §18.2 · p. 85, Figure 18.2 · p. 86

18.4 YCbCr: luma and chroma, and why video compresses color more aggressively

  • Subsampling chroma 8× yields mean absolute error 6.44 gray levels, versus 10.15 for 8× luma subsampling, supporting lower-resolution chroma storage.The visual system resolves fine spatial detail in luma better than chroma, motivating formats such as 4:2:0.Source: §18.4 · pp. 85–86, Figure 18.3 · p. 86

18.5 L*a*b*: a perceptually uniform space

  • At the same RGB distance of 30.0, 2,000 color pairs have L*a*b* distances from 3.0 to 38.1 (median 17.3), showing RGB distance can misrepresent perceptual difference.CIELAB Euclidean distance is designed to approximate perceived color difference more consistently.Source: §18.5 · pp. 86–87, Figure 18.4 · p. 87

19.1 k-means on real color data; 19.2 Where k-means breaks: non-convex clusters; 19.3 Gaussian mixture models: soft, elliptical clusters

  • K-means nearly recovers compact, round color regions but needs k in advance; on concentric rings it reaches 54.0% accuracy, while GMM reaches 55.7%.GMM adds soft assignments and full covariances, yet its single-Gaussian-per-cluster model cannot represent ring-shaped clusters.Source: §19.1 · p. 88, §19.2 · p. 89, §19.3 · pp. 89–90

19.4 DBSCAN: clusters defined by density, not shape; 19.5 Noise robustness: a real advantage of “no forced assignment”

  • DBSCAN finds two ring clusters with 100.0% accuracy and isolates 8 of 60 clutter points as noise, unlike k-means and GMM, which force every point into a cluster.It can handle arbitrary connected dense shapes, but requires hand-chosen eps and min_samples and struggles with clusters of different densities.Source: §19.4 · p. 90, §19.5 · pp. 90–91

20.1 The structure tensor: what makes a good point to match?; 20.2 Harris corner detector; 20.3 Shi-Tomasi: “Good Features to Track”; 20.4 The scale problem

  • The gradient structure tensor distinguishes corners from edges and flat regions; from-scratch Harris and Shi-Tomasi responses each correlate perfectly (1.0) with their OpenCV counterparts.Harris uses a determinant/trace proxy, while Shi-Tomasi uses the minimum eigenvalue; both detectors use a fixed window size and are not scale-invariant.Source: §20.1 · p. 93, §20.2 · p. 93, §20.3 · p. 94

20.5 SIFT: scale-space extrema and descriptors; 20.6 Matching under rotation and scale

  • SIFT detects features across scale, assigns orientation, and builds 128-dimensional descriptors; in a 35° rotation and 0.7× shrink test, the ratio test retains 191 of 723 candidate pairs.The test detected 723 keypoints in the original and 365 in the transformed image; Lowe’s ratio test rejects ambiguous matches by requiring the best candidate to be meaningfully closer than the second-best.Source: §20.4 · p. 94, §20.5 · p. 95, §20.6 · p. 95

How many matches are actually correct?

  • 175 of 191 ratio-test survivors (92%) are geometrically correct when checked against the known transform.Each matched keypoint is mapped through the transform and compared with its observed match.Source: How many matches are actually correct? · p. 95

Exercises … Exercises

The section develops sparse and dense optical-flow methods and stereo matching, explaining their assumptions, limitations, and implementations. Synthetic and real examples show how local motion or disparity estimates behave and how they can be used to recover motion and depth.

21.1 The brightness constancy assumption / 21.2 The aperture problem / 21.3 Lucas-Kanade: solving the aperture problem with a window

  • Brightness constancy yields one flow-constraint equation for two unknowns per pixel, and an edge reveals only motion perpendicular to itself, leaving motion along the edge unconstrained.Lucas–Kanade addresses this aperture problem by assuming constant flow within a small window and combining pixel constraints in a least-squares system; the system is well-conditioned at corners because the structure tensor has two large eigenvalues.Source: §21.1 · p. 98, §21.2 · p. 98, §21.3 · pp. 98–99

21.3 Lucas-Kanade: solving the aperture problem with a window / Larger motions need iteration

  • For true motion (4.0, 3.0), manual Lucas–Kanade averages (0.846, 0.690), while cv2.calcOpticalFlowPyrLK recovers (4.000, 3.000); for (0.6, 0.4), manual flow averages (0.612, 0.393).The single-shot solve degrades for larger shifts because its Taylor approximation holds only for small motions; iterative re-warping and re-linearization on an image pyramid handles them.Source: §21.3 · p. 99, Larger motions need iteration · p. 100

21.4 Horn-Schunck: a global alternative to windowed flow

  • Horn–Schunck globally balances optical-flow constraint fit with spatial smoothness, coupling neighboring pixels so reliable estimates can propagate into textureless regions.Its global optimization is expensive and slow to converge; iterative updates spread flow outward from textured areas.Source: §21.4 · pp. 100–101, Figure 21.4 · p. 101

21.5 Farnebäck’s method for dense optical flow / A real video: the Army sequence

  • Farnebäck estimates dense flow by comparing local polynomial expansions between frames, and color-coded flow distinguishes independently moving objects in the Army video.Unlike Lucas–Kanade, it does not assume constant flow within a window; it is presented as a practical dense-flow alternative to slower-converging Horn–Schunck.Source: §21.5 · p. 101, A real video: the Army sequence · p. 101

21.5 Farnebäck’s method for dense optical flow

  • For true motion (5.0, −3.0), dense Farnebäck flow averages (2.25, −1.35) over all pixels but (5.00, −3.00) over high-gradient pixels.Textureless areas lack local motion information and bias the whole-image average.Source: §21.5 · p. 101, Figure 21.5 · p. 102

Stereo Matching / 22.1 Rectified stereo: why the search is 1D / 22.2 A synthetic stereo pair with known ground truth / 22.3 Block matching along the scanline / 22.8 From disparity to depth map

  • For rectified stereo, corresponding pixels lie on the same row, so matching searches one dimension; disparity converts to depth through Z = fB/d, and block matching selects the lowest-SSD patch.The synthetic pair uses three disparity planes and occluded pixels to provide known ground truth for testing the matcher.Source: §22.1 · p. 104, §22.2 · p. 104, §22.3 · p. 105

22.3 Block matching along the scanline

  • On synthetic stereo depth planes, block matching recovers mean disparities 5.03, 15.03, and 25.04 for true values 5, 15, and 25, with errors concentrated near depth boundaries.Each left-image patch is matched to the same-row right-image patch with the lowest sum-of-squared-differences; windows spanning depth boundaries mix disparities.Source: §22.3 · p. 105, Figure 22.2 · p. 105

22.4 Catching bad matches: the left-right consistency check / 22.5 The same thing, at full resolution, with OpenCV / 22.6 Window size: detail vs. noise tradeoff / 22.7 A real stereo pair: Tsukuba

  • Left-right consistency rejects unreliable stereo matches, larger windows trade sharper depth boundaries for noise robustness, and the same process works on real Tsukuba imagery.On the full synthetic image, enabling StereoBM’s consistency check flags 264 additional pixels invalid; a wider window smooths noise but blurs depth discontinuities.Source: §22.4 · p. 106, §22.5 · pp. 106–107, §22.6 · p. 107

Model Fitting, Robust Estimation, and RANSAC … Exercises

These sections develop robust model fitting, projective geometry, and panorama stitching as a connected computer-vision pipeline. They explain how to fit models despite noisy or incorrect correspondences, when homographies apply, and how to use them to align and blend images.

23.1 Ordinary vs. total least squares

  • With comparable noise on both axes, TLS fit the true line y = 3.00x + 1.00 as y = 2.97x + 0.98, closer than OLS at y = 2.70x + 1.67.OLS minimizes vertical error, whereas TLS minimizes perpendicular error and, for a 2D line, uses the covariance matrix’s largest-eigenvalue direction.Source: §23.1 · p. 110

23.3 RANSAC: fit from the inside out

  • On contaminated data with 40 true inliers among 55 points (27% outlier contamination), RANSAC fit y = 2.05x+4.94 and classified 42 of 55 points as inliers, essentially recovering the true line.RANSAC repeatedly samples the smallest subset needed for a candidate model, keeps the candidate with the most consensus, then refits least squares on the winning inlier set.Source: §23.3 · p. 111

How many iterations does RANSAC need?

  • At 30% inliers, a fundamental matrix with an 8-point minimal sample requires over 70,000 RANSAC iterations for the stated confidence, while a homography’s 4-point sample needs about an order of magnitude more iterations than a line’s 2-point sample at 50% inliers.The probability of an all-inlier sample per try is w^n, where w is the inlier fraction and n the minimal sample size.Source: How many iterations does RANSAC need? · pp. 111–112

23.4 A gentler alternative: robust loss functions

  • On the same contaminated line data, Huber reweighting fit y = 1.90x + 5.58—closer to the true y = 2.00x + 5.00 than OLS’s y = 1.02x + 9.81, but not as close as RANSAC’s y = 2.05x + 4.94.Huber loss grows linearly beyond a threshold, reducing the influence of large residuals; unlike RANSAC, it does not entirely exclude gross outliers.Source: §23.4 · pp. 112–113

Projective Geometry

  • A homography represents perspective effects such as parallel lines converging at a vanishing point, and applies exactly between views of a flat surface or between views taken from one camera position with rotation only.Four point correspondences determine a homography; with more correspondences, DLT fits an overdetermined model, optionally with RANSAC to reject bad matches.Source: Projective Geometry · p. 114, §24.1 · p. 114, §24.3 · p. 115

RANSAC in practice: homography estimation with bad matches

  • RANSAC found 30/45 genuine homography matches and achieved 1.49 × 10−6 pixels mean reprojection error, versus 1.06×104 pixels for least squares on contaminated data.The 15 bad matches were enough to corrupt the unweighted fit, while RANSAC rejected them as outliers.Source: RANSAC in practice: homography… · p. 116

Image Stitching and Mosaicking

How good is the fit?

Image Formation … 28.1 The epipolar constraint

These sections trace images from pinhole projection, lens and sensor effects, and color correction through camera calibration to the epipolar constraint relating corresponding points in two views.

26.1 The pinhole projection matrix, derived; Camera Calibration

  • Pinhole projection yields P = K[R|t], with K encoding focal length and principal point and [R|t] encoding camera pose; Zhang calibration estimates K from checkerboard homographies across views.Calibration also estimates lens distortion and each view’s pose.Source: §26.1 · p. 122, Camera Calibration · p. 128, §27.1 · p. 128

26.2 Real lenses: focus and defocus blur

  • With focus at 5m, objects at 2m and 9m blur with kernel sizes 19 and 25, while the focal-plane object remains sharp with size 1.Wider apertures gather more light but reduce depth of field; narrow apertures let in less light but produce sharper images.Source: §26.2 · p. 123

Demosaicking: reconstructing RGB from the mosaic

  • RGGB Bayer sensors record one color per pixel; bilinear reconstruction achieved MAE 3.37 versus OpenCV’s 3.43, with residual error attributed mostly to sensor noise in the mosaic.Each 2 × 2 Bayer tile has one red, one blue, and two green filters, leaving two-thirds of each pixel’s color information to reconstruct.Source: §26.3 · p. 123, Demosaicking: reconstructing RGB from… · p. 124

The pitfall: getting the mosaic pattern wrong

The pitfall: averaging encoded pixels is not averaging light

The gray-world assumption

  • Under the gray-world assumption, equalizing channel averages reduces mean absolute pixel error from 18.98 to 7.43, though the scene’s colored content leaves residual error.Real cameras may combine gray-world with a detected neutral reference, learned scene statistics, or user presets.Source: Figure 26.7 · p. 126, The gray-world assumption · p. 126

27.3 Running calibration; 27.5 Undoing distortion with the estimated parameters

  • With 8 noisy views, calibration reached 0.264 px RMS, fx=822.6 and fy=826.3 versus true 800 and 800, while k1=0.0054 versus true −0.3; undistortion’s maximum error was 3.4 × 10^-2 in normalized coordinates.The calibration passage notes that more views from diverse angles help constrain estimates, especially distortion.Source: §27.3 · p. 129, §27.5 · p. 130

28.1 The epipolar constraint

  • For corresponding homogeneous image points x1 and x2, the rank-deficient fundamental matrix F satisfies the epipolar constraint, and l2 = Fx1 restricts x2 to a line; rectification makes that line horizontal.Thus, row-restricted stereo matching is the special case of the general epipolar-line search.Source: §28.1 · p. 132, Epipolar Geometry · p. 132

Why a line? A geometric picture … 30.2 Matrix multiplication: many projections at once

The passages connect two-view epipolar geometry and structure-from-motion to a broader view of linear operations as projections. They describe verified implementations and synthetic or real-image examples, including the scale ambiguity in recovering camera motion and 3D structure.

28.2 A synthetic two-camera scene / 28.3 SVD: singular value decomposition / 28.4 The 8-point algorithm

  • The normalized 8-point algorithm estimates F from correspondences, solves by SVD, and enforces rank 2; its implementation matched OpenCV to 5 decimal places with residuals of 10^-7–10^-15.The method normalizes coordinates before solving and undoes the rescaling on F; the synthetic two-camera scene supplies projected correspondences.Source: §28.2 · p. 132, §28.4 · pp. 133–134, §28.3 · p. 133

Why a line? A geometric picture / 28.5 From fundamental to essential: adding calibration / Recovering camera motion

  • With known intrinsics, F yields the 5-degree-of-freedom essential matrix E = [t]×R; pose recovery matched the synthetic 15° rotation to 3 decimal places and recovered translation direction up to sign, not magnitude.For a point in camera 1, its unknown depth leaves camera 2’s projection on one epipolar line; two views cannot recover translation scale.Source: Why a line? A geometric picture · p. 132, §28.5 · p. 134, Recovering camera motion · p. 135

29.1 The classic pipeline, at a glance / Step 1: feature matching / Step 2: robust essential matrix and pose recovery

Step 3: triangulation / Cheirality: a free sanity check

  • Triangulation solves for a 3D point from two camera projection matrices by DLT/SVD, matching cv2.triangulatePoints to 10^-7; positive depth in both views reduced inliers from 48 to 47.For the 47 retained points, reprojection error was 0.169 px mean and 1.680 px maximum; cheirality selects among four valid pose sign combinations.Source: Step 3: triangulation · pp. 137–138, Cheirality: a free sanity check · p. 138

29.3 Putting it together: reconstruction using estimated pose

  • After scaling the estimated translation with the true baseline, reconstruction error was 0.058 mean and 0.096 maximum; the pipeline needed 2D correspondences, known intrinsics, and that external scale reference.Without the baseline, the reconstructed scene remains determined only up to scale.Source: §29.3 · p. 138

Structure from Motion / Visualizing the reconstruction / From sparse to dense, and from two views to many

Projection — The Central Idea / 30.1 Two projections you’ve already built

  • PCA projects points onto a direction chosen for maximum spread, whereas camera projection uses directions fixed by camera geometry; both compute y = w⊤x + b.The passages frame projection as a central idea spanning earlier vision methods and neural networks.Source: §30.1 · pp. 142–143, Projection — The Central Idea · p. 142

30.2 Matrix multiplication: many projections at once / • Convolution

  • Stacking projection directions as rows of W makes y = Wx compute them together; convolution, Fourier transforms, and camera or homography matrices are structured instances of this matrix multiplication.A hand-built Toeplitz matrix reproduced np.convolve within 5.6 × 10^-17, and a sinusoid matrix reproduced np.fft.fft within 2.9 × 10^-15; a fully connected layer learns W and b.Source: • Convolution · p. 143, §30.2 · p. 143

30.3 A projection for classification … What the hidden layer actually did

A single linear projection separates well-separated classes but cannot solve a ring-and-disk problem; adding a nonlinear hidden layer reshapes the representation so a final linear projection can separate the classes.

30.3 A projection for classification; 30.4 What a single projection cannot do

  • 74.2% was the best accuracy found by searching 2000 directions and 61 thresholds for a linear separator of a ring and its surrounded disk.A mean-difference direction achieves 100.0% on two well-separated synthetic classes, while a single projection can only make a straight-line boundary.Source: §30.4 · p. 143, §30.3 · p. 143, Figure 30.3 · p. 145

Neural Network Fundamentals; 31.3 From a raw score to a probability; 31.4 Binary cross-entropy: turning probability into a trainable loss; 31.5 Backpropagation: the chain rule, applied; 31.6 Training: the full loop

  • Gradient descent trains a neuron by repeating a forward pass, loss computation, backward pass, and update, using sigmoid probabilities and binary cross-entropy to provide a smooth trainable loss.Backpropagation applies the chain rule; the weight gradient is the prediction error weighted by the input, and the bias gradient is the averaged prediction error.Source: Neural Network Fundamentals · p. 146, §31.3 · p. 147, §31.4 · p. 147

31.1 A new library: PyTorch and tensors

  • PyTorch tensors extend NumPy arrays with gradient tracking and GPU support, and PyTorch is used for neural-network training because it provides automatic differentiation and GPU acceleration.PyTorch stores color images channel-first and typically expects a batch dimension first.Source: §31.1 · p. 146

Sanity checks: numerical gradient and PyTorch autograd; 32.2 Backprop through two layers

  • 1.01 × 10−9 is the maximum discrepancy between finite-difference and hand-derived gradients, while PyTorch autograd agrees with the hand derivation to 5.55 × 10−17.For a two-layer MLP, a from-scratch forward and backward pass also matches PyTorch autograd gradients to within 3 × 10−17 on every parameter.Source: Sanity checks: numerical gradient and… · p. 148, §32.2 · p. 152

Compare to Chapter 30’s hand-picked direction

  • 2.2° separates the learned direction from the hand-picked direction on the two-class dataset, as training from random initialization rediscovers the mean-difference heuristic.The learned parameters are w = [2.954, 2.089], b = −0.115, with 100.0% final accuracy.Source: §31.6 · p. 149, Compare to Chapter 30’s hand-picked… · p. 150

31.7 Training on the unsolvable problem

  • 50.0% accuracy after 2000 epochs on the ring-and-disk task is below brute-force search’s 74% result, as symmetric gradients nearly cancel and leave the learned weights near zero.The failure reflects the single linear projection’s limits, not merely a poor optimizer; the next chapter adds multiple projections with nonlinearities.Source: §31.7 · p. 150

32.1 Two projections, one nonlinearity; Why the nonlinearity is essential

  • An MLP applies a hidden projection, elementwise ReLU, and a final linear projection; ReLU’s kink lets the hidden layer bend and fold the input space.Without ReLU, stacked linear layers collapse into one linear projection; two linear layers matched their combined layer to within 1.78 × 10−15.Source: §32.1 · pp. 151–152, Why the nonlinearity is essential · p. 152

What the hidden layer actually did

  • 100.0% accuracy in the full 4D hidden space shows that the MLP reshapes the ring-and-disk data until a final linear projection can separate it.A PCA view of the hidden space is lossy but still shows the classes pulled apart toward linear separability, unlike the 74% ceiling for a line in the original space.Source: What the hidden layer actually did · p. 153

32.5 Model capacity: how many hidden units are enough? … 34.5 Why translation equivariance actually matters: a generalization test

The section traces how model capacity and practical optimization choices affect learning, then introduces convolutional networks and tests their ability to generalize across image positions. It explains convolution’s local, shared filters, pooling, and the role of translation equivariance.

32.5 Model capacity: how many hidden units are enough?; H mean acc min max

  • H=1 stays near Chapter 30’s ∼74% linear limit, H=2 misses 100% across seeds, H=3 reaches 100% once, and H=4 reaches 100% across all seeds.The capacity floor reflects what the model can represent; near that floor, whether training finds a solution also depends on initialization.Source: H mean acc min max · p. 154, Figure 32.5 · p. 156

Optimization; 33.1 A loss landscape where plain gradient descent struggles; 33.2 Momentum: remember where you were heading; 33.3 Adam: per-parameter adaptive step sizes

  • After 30 steps, plain gradient descent, momentum (β=0.9), and Adam (η=0.3) reach losses 0.3230, 0.2785, and 0.1244, respectively.Momentum accelerates progress along a consistent direction but does not tune its brake to each axis; Adam rescales each direction independently. The chapter also covers initialization and learning-rate schedules as practical upgrades to plain gradient descent.Source: §33.1 · p. 157, Optimization · p. 157, §33.2 · p. 158

33.4 Weight initialization: why zero doesn’t work

  • After 3000 epochs, zero-initialized four-unit MLPs reach only 50.0% accuracy, while randomly initialized weights reach 100.0%.Zero initialization keeps hidden units identical because they receive identical gradients; small random weights break this symmetry and allow units to specialize.Source: §33.4 · p. 159

33.5 Weight initialization schemes: how large should “small random” be?

  • At width 256 after 20 layers, Xavier-scale ReLU variance is 3.06 × 10−7, versus 1.324 with Kaiming scale.Fixed weight standard deviations make activation variance depend on fan-in; Xavier scales for tanh-like activations, while Kaiming compensates for ReLU zeroing roughly half its inputs.Source: §33.5 · pp. 159–160

33.6 Learning rate schedules; Why decay actually helps: noisy gradients

  • With noisy gradients, mean loss over the final 50 steps is 0.2142 at a constant learning rate and 0.0025 with step decay.Step decay preserves fast early progress, then reduces the persistent bouncing around the minimum; the chapter also describes cosine annealing and linear warmup.Source: §33.6 · p. 160, Why decay actually helps: noisy… · p. 161

Convolutional Neural Networks (CNNs); 34.1 A conv layer is a projection, restricted and shared; A conv layer kernel is 3D

34.2 Conv2d, forward and backward, from scratch; Filters you already know: Chapter 12’s edge detectors; 34.3 Pooling: downsampling with a purpose; 34.4 Putting them together: convolution and pooling, interleaved

  • From-scratch Conv2d matches F.conv2d to 8.88 × 10−16, while its input, kernel, and bias gradients match autograd to 8.88 × 10−16, 7.99 × 10−15, and 1.78 × 10−15.Pooling downsamples feature maps and adds local translation invariance; CNNs commonly alternate convolution and pooling, roughly halving spatial size while doubling channels. First-layer filters often resemble oriented edge and blob detectors.Source: §34.2 · p. 164, Filters you already know: Chapter 12’s… · p. 164, §34.3 · p. 165

34.5 Why translation equivariance actually matters: a generalization test

  • On unseen corner positions, both models fit training data perfectly, but the flatten-based MLP is near chance while the CNN generalizes with no accuracy loss.The task trains on centered shapes and tests on corner positions; the CNN’s global max pooling makes its final decision depend on which filters fired, not where.Source: §34.5 · pp. 165–166

Exercises … 37.2 A brief historical tour

Across Chapters 35–37, the text shows how limited data can cause overfitting, why per-class evaluation matters, and how CNN architectures evolved alongside ImageNet benchmarks. It illustrates these ideas with synthetic and real-image experiments, practical metrics, and a historical tour of key designs.

Exercises

  • Exercises probe unseen-position generalization, padding against PyTorch, regularization choices, balancing truck data, and per-class F1; they are proposed tests, not reported findings.The prompts also ask whether edge clipping changes shape appearance and whether augmentation plateaus when based on only 12 underlying examples.Source: Exercises · pp. 166, 170, 174, 175

Training a CNN; 35.1 A deliberately hard, small dataset; 35.2 Watching it overfit

  • With only 12 noisy training images, a CNN memorizes its training set: training accuracy reaches 100.0%, while validation accuracy is 72.7% and validation loss rises after epoch 277.The validation loss bottoms at 0.456 at epoch 277, then reaches 0.573 by the end of 400 epochs.Source: Training a CNN · p. 167, §35.1 · p. 167, §35.2 · pp. 167–168

35.3 Fix 1: data augmentation; 35.4 Fix 2: weight decay; 35.5 Fix 3: dropout

  • On the 12-image task, augmentation raises validation accuracy to 95.3% and weight decay to 84.0%, while dropout’s small-data benefit is modest and variable.Augmentation uses 20 transformed copies per base image; weight decay penalizes squared weights, while inverted dropout randomly zeros and rescales activations during training and is inactive at evaluation.Source: §35.3 · p. 168, §35.4 · p. 169, §35.5 · p. 169

35.6 Putting it together: real photos, real overfitting

  • On scarce CIFAR-10 data, augmentation, weight decay, and dropout combined reach 82.3% validation accuracy, beating each individual fix and the unregularized baseline by a modest margin.The comparison averages configurations over three seeds using 40 training images per class for cats and trucks; weight decay alone falls below the unregularized baseline, illustrating that regularization strength may not transfer across settings.Source: §35.6 · p. 170

Image Classification in Practice; 36.1 A 4-class, imbalanced dataset; 36.2 Confusion matrix; 36.3 True positives, false positives, true negatives, false negatives

  • A 50.3% overall test accuracy on the imbalanced four-class task conceals uneven errors: confusion matrices expose which classes are mistaken for others, and one-vs-rest counts define per-class evaluation.For a chosen class, true positives, false positives, false negatives, and true negatives are the four outcomes; precision and recall summarize the first three.Source: §36.1 · p. 173, §36.2 · p. 173, §36.3 · p. 174

36.4 Precision and recall

  • Truck’s high precision but very low recall reflects its 90-image training shortage, while cat–dog errors reflect visual similarity and automobile absorbs many truck mistakes.The passage warns that aggregate accuracy hides these distinct failure modes and recommends inspecting confusion matrices and per-class precision and recall, especially for imbalanced data.Source: §36.4 · p. 174

37.1 ImageNet and ILSVRC: the benchmark that drove this

  • ImageNet’s shared ILSVRC benchmark made architecture comparisons possible; winning top-5 error fell from 28.2% in 2010 to under 2.5% by 2017, and ResNet first beat the ∼5% human result.The competition’s 2017 classification track was retired after the winning error fell below 2.5%.Source: §37.1 · p. 176

Classic Architectures; 37.2 A brief historical tour

  • CNN designs addressed different limits: AlexNet scaled the LeNet template, VGG stacked small filters, GoogLeNet widened through parallel filters, and ResNet enabled greater depth.SENet refined channel weighting; the chapter then focuses on ResNet’s residual connection as its most consequential architectural idea.Source: §37.2 · p. 177

37.3 Building a ResNet block … 39.4 Do these maps actually track what the model uses?

Residual connections and batch normalization make deeper networks easier to train; pretrained backbones improve learning from scarce labels, while saliency maps and Grad-CAM can expose when a CNN relies on a spurious shortcut. The section demonstrates these ideas through numerical comparisons and controlled image-classification experiments.

The vanishing-gradient problem

  • In a 30-layer sigmoid stack, the input-to-last-layer gradient-norm ratio is 2.07 × 10−19 for a plain network but 1.05 × 100 for a residual network.A residual connection adds an identity term to each layer’s Jacobian, providing a gradient path that avoids repeated multiplication by small sigmoid derivatives.Source: The vanishing-gradient problem · p. 178, Figure 37.1 · p. 178

Batch normalization

  • A from-scratch per-channel batch-normalization implementation matches nn.BatchNorm2d to 4.77 × 10−7.Batch normalization subtracts each channel’s mini-batch mean, divides by its standard deviation, and applies learnable scale and shift; stabilizing activation distributions helps make higher learning rates and deeper networks practical.Source: Batch normalization · pp. 178–179

Putting it together: a convolutional residual block; Exercises

  • A convolutional residual block uses two convolutions with batch normalization and ReLU, then adds the input feature map before the final ReLU; it preserves tensor shape.The exercise prompts explore changing activations and shortcut scaling, and adapting shortcuts when channel counts differ.Source: Putting it together: a convolutional… · p. 179, Exercises · p. 179

Transfer Learning; 38.1 Source task and target task; 38.2 Three strategies

  • The CIFAR-10 transfer experiment pretrains on 300 airplanes and 300 automobiles, then tests cat-versus-truck classification with 15 images per class; frozen features and fine-tuning both beat training from scratch and are more consistent.The three strategies are scratch training, a frozen pretrained backbone with a new linear head, and fine-tuning with a smaller backbone learning rate; the frozen backbone’s mean test accuracy is 69.5%.Source: §38.1 · p. 180, §38.2 · p. 181

38.3 In practice: pretrained backbones in one line

  • 98.1% ± 0.2% mean test accuracy over 8 seeds: frozen ImageNet-pretrained ResNet-18 features beat the toy frozen backbone’s 69.5% on the same 30 target images.The backbone is frozen and only a new linear layer is trained; the comparison is not fully controlled because ImageNet includes cats and vehicle types.Source: §38.3 · p. 182

Visualizing and Understanding CNNs; 39.1 Setup: a cat-vs-automobile CNN, with feature maps exposed; 39.2 Saliency maps

  • For a 300-epoch cat-versus-automobile CNN, test accuracy is 86.5%; saliency maps use class-score gradients with respect to input pixels to highlight pixels where small changes affect the prediction.The chapter builds saliency and Grad-CAM from scratch to inspect model behavior and learned feature vectors.Source: §39.1 · p. 184, §39.2 · p. 184, Visualizing and Understanding CNNs · p. 184

39.3 Grad-CAM

  • Grad-CAM averages target-score gradients spatially to weight feature-map channels, sums the weighted maps, and applies ReLU to retain positive class contributions.Its coarse heatmap can be upsampled; unlike noisy pixel-level saliency, it uses lower-resolution feature maps with more class-discriminative semantic information.Source: §39.3 · p. 185

39.4 Do these maps actually track what the model uses?

  • On marked test images, model_shortcut’s saliency and Grad-CAM peaks land in the marker region 96.0% and 100% of the time; on clean images, Grad-CAM still reaches 92.0% versus saliency’s 56.0%.model_original scores 0.0% for both methods on marked images; the shortcut model’s accuracy collapses when the marker is removed, showing that Grad-CAM better exposes its hidden reliance on the absent trigger.Source: §39.4 · p. 186

39.5 Feature-space visualization: t-SNE … 42.4 Adding skip connections: U-Net

These sections move from visualizing learned image features to detecting objects and labeling pixels, explaining the methods and illustrating their behavior on synthetic and real images. They show how training separates feature neighborhoods, how detectors locate objects at different scales, and how segmentation architectures recover spatial detail.

39.5 Feature-space visualization: t-SNE

  • 78% of trained-feature t-SNE points had a same-class nearest neighbor in 2D, versus about 62% for untrained features, whose classes showed no visible separation.The embedding used 16-dimensional pooled test-image features and did not use labels during projection; the trained features formed visibly separated cat and automobile clusters.Source: §39.5 · p. 187, Figure 39.3 · p. 187

Object Detection from Sliding Windows

  • Sliding-window detection searches locations with a fixed-size patch classifier, while an image pyramid handles scale and Viola-Jones accelerates the search with a cascade of cheap Haar-like feature tests.The cascade rejects most non-face windows early; on an Apollo 11 crew photo, the supplied passage reports four detections, including all three astronauts and one false positive.Source: Object Detection from Sliding Windows · p. 189, §40.4 · p. 192, §40.5 · p. 192

40.3 Step 3: threshold, then merge duplicates

  • A synthetic window classifier achieved 100.0% test accuracy, and non-maximum suppression reduced 60 above-threshold detections to two boxes near the two true face centers.The threshold was the 97.5th percentile of scene scores, and the two retained detections landed at (29, 19) and (50, 49).Source: §40.1 · p. 190, §40.3 · p. 191, Figure 40.3 · p. 191

Object Detection as Regression

  • A one-object CNN regresses normalized (cx, cy, w, h) coordinates and reached mean test IoU 0.758, with 95.0% of boxes above IoU 0.5.For multiple objects, two-stage R-CNN methods evaluate proposed regions, whereas single-stage YOLO and SSD predict boxes and classes in one pass; both use NMS to merge overlaps.Source: §41.2 · pp. 194–195, §41.1 · p. 194, §41.3 · pp. 195–196

41.4 In practice: real detectors on real images

  • YuNet found all three astronauts with no false positives, while Viola-Jones returned four detections including a sleeve-texture false positive; pretrained Faster R-CNN returned four confident detections.The Faster R-CNN detections were two cats and two remote controls, each above 0.78 confidence at a 0.7 cutoff.Source: §40.5 · p. 192, §41.4 · pp. 196–197, Figure 41.3 · p. 196

Semantic Segmentation

  • FCN segmentation downsamples images into semantically rich features and upsamples per-pixel class predictions; on a real street photo, it labeled the cyclist and bicycle and split the tram between train and bus.The toy task labels background, circles, and squares despite pixel fractions of 0.866, 0.046, and 0.088; the real-photo tram split reflects the model’s limited class vocabulary.Source: §42.1 · p. 199, §42.2 · p. 200, §42.3 · p. 200

42.4 Adding skip connections: U-Net

  • U-Net concatenates encoder features at matching resolutions into the decoder, preserving fine details that pooling can erase; its value depends on whether a task requires precise spatial boundaries.Unlike residual connections, these skip connections cross the encoder-decoder path; they are useful for boundary-sensitive work such as medical imaging, but are not universal across segmentation networks.Source: §42.4 · p. 201
  • 98.1% pixel accuracy and 0.870 mean IoU are reported for FCNTiny; U-Net substantially improves mean IoU on small, tightly packed shapes.The supplied passages do not give U-Net’s numeric score. Its benefit is attributed to retaining high-resolution encoder features rather than relying only on the pooled bottleneck.Source: §42.2 · p. 200, §42.4 · p. 201

Exercises … Exercises

The chapters introduce instance segmentation through center-offset voting and Mask R-CNN, then examine precision and parallelism techniques used to train networks at scale. They pair toy demonstrations with a real-photo example and exercises probing the methods’ limitations.

Exercises; 44.5 Two more practical tricks

  • The exercises probe segmentation boundaries and robustness, while gradient accumulation extends shard-style averaging across time and checkpointing trades extra computation for lower activation memory.Prompts test seam confidence, heavier circle overlap, crop-center error, and when larger loss scales risk fp16 overflow; checkpointing is useful when activations rather than weights exceed memory.Source: Exercises · pp. 202, 208, 213, §44.5 · p. 213

Instance Segmentation; 43.1 Overlapping instances; 43.2 Augmenting the network with an instance head

  • Instance segmentation distinguishes individual objects of the same class, unlike semantic segmentation, and the chapter extends a U-Net with an offset-prediction head alongside its semantic head.Scenes contain two close or overlapping circles; trained for 300 epochs with semantic cross-entropy and foreground-masked offset MSE, the network achieved 100.0% semantic pixel accuracy.Source: Instance Segmentation · p. 204, Instance segmentation · p. 204, §43.1 · p. 204

43.3 Recovering instances: vote for the center, then cluster (Hough, again)

  • Offset voting recovered the correct instance count in 44/45 test scenes (97.8%).Each foreground pixel votes for its instance’s center, and clustering the votes separates instances even when their pixels touch or overlap.Source: §43.3 · p. 205

43.5 A mask head

  • Trained for 300 epochs with binary cross-entropy, the per-instance mask head achieved IoU 0.844 versus 0.687 for marking every crop foreground; every test crop contained visible pixels from a neighboring circle.Source: §43.5 · p. 207

43.4 Mask R-CNN: Faster R-CNN plus a mask branch; 43.6 In practice: Mask R-CNN on a real photo

  • On the Apollo 11 photo, Mask R-CNN detected three astronauts and one spurious clock above 80% confidence, while separating the touching astronauts into distinct instances.The person detections each had confidence ≥99.8%; the clock detection had 83.6% confidence. Its mask branch predicts each object’s silhouette within a detector-proposed box, and RoIAlign uses bilinear interpolation to keep mask features aligned with box coordinates.Source: §43.4 · p. 206, §43.6 · p. 208

44.1 Floating-point precision: fp32 vs. fp16

  • In a 12-layer sigmoid network, first-layer gradients with mean magnitude 2.65 × 10^-12 were 0.0% exactly zero in fp32 but 100.0% zero when cast to fp16.Values of 10^-8 and smaller vanished in fp16, demonstrating how gradients that remain nonzero in fp32 can disappear through underflow.Source: §44.1 · pp. 210–211

44.2 Mixed-precision training: the recipe

  • Loss scaling by 65,536 rescued most first-layer gradients, but over a third still vanished; only at 2^28 did the last gradient clear fp16’s floor.Mixed precision combines fp16 forward and backward passes with fp32 master weights and scaled losses; dynamic scaling adjusts upward when steps avoid inf/NaN and backs off when they occur. Bfloat16 avoids fp16-style underflow through fp32-like exponent range but has lower precision.Source: §44.2 · p. 211

44.3 Data parallelism: correctness proven in miniature; 44.4 Model parallelism: shipping activations across a boundary

  • Averaged shard gradients matched the single-device gradient within 1.49 × 10^-8, while the split-model gradient matched exactly with 0.0 difference.Data parallelism splits data and averages gradients, with all-reduce communication as a scaling cost; model parallelism splits layers and hands off activations and gradients, with pipeline idle time as a concern.Source: §44.3 · p. 212, §44.4 · pp. 212–213
Loading 2609.39627v1…