Source-linked AI summary
A guide to convolution arithmetic for deep learning
Vincent Dumoulin, Francesco Visin
TL;DR
CNN architecture shapes are difficult to infer because convolutional outputs depend on several interacting properties, and transposed convolutions have been explained inconsistently. This guide derives and illustrates relationships among convolutional, pooling, and transposed convolutional layers to make their arithmetic intuitive.
Problem
CNN output shapes depend on input shape, kernel shape, zero padding, and strides, while the relationship between convolutional and transposed convolutional layers has varying clarity.
Method
The guide derives relationships for convolutional, pooling, and transposed convolutional layers and illustrates them intuitively, including transposed-convolution matrix and stride interpretations.
Results
The guide establishes relationships between convolutional and transposed convolutional layers, including cases where transposed convolutions correspond to fully padded convolutions with unit strides.
Takeaways & Limitations
The resulting guide helps practitioners understand and manipulate CNN architectures across convolutional, pooling, and transposed convolutional layers.
Takeaways & Limitations
Different input sizes can produce the same convolution output size, complicating analysis for transposed convolutions; direct-convolution emulation usually has the disadvantage of added operations.
Abstract
from arXiv · showhide
We introduce a guide to help deep learning practitioners understand and manipulate convolutional neural network architectures. The guide clarifies the relationship between various properties (input shape, kernel shape, zero padding, strides and output shape) of convolutional, pooling and transposed convolutional layers, as well as the relationship between convolutional and transposed convolutional layers. Relationships are derived for various cases, and are illustrated in order to make them intuitive.
Introduction
This guide explains why CNN architecture shapes are difficult to infer and provides intuitive relationships among convolutional properties, pooling, and transposed convolutions.
- Motivation: CNN output shape depends jointly on input shape, kernel shape, zero padding, and strides, unlike fully connected layers.The guide also addresses pooling and transposed convolutional layers, whose relationships with convolutional layers may be unclear.
- Objectives: The guide explains the relationship between convolutional and transposed convolutional layers.
- Objectives: It provides an intuitive account of how input shape, kernel shape, zero padding, strides, and output shape interact in convolutional, pooling, and transposed convolutional layers.
- Scope: The guide derives relationships independently of implementation details so they apply across commonly used machine-learning frameworks.
- Discrete convolutions: Discrete convolutions preserve ordering in structured data through sparse local connections and reused parameters.A kernel slides across the input, multiplying overlapping elements and summing them to produce each output location.
- Discrete convolutions: Strides can be understood either as translating the kernel by larger hops or as retaining only periodically spaced output elements.For s = 2, translating by two is equivalent to translating by one and retaining one in every two outputs.
1.2 Pooling
Pooling layers summarize local input regions to reduce feature-map size, with average and max pooling as common examples.
- Pooling operations: Pooling reduces feature-map size by summarizing subregions with functions such as averaging or taking the maximum.
- Pooling operations: A pooling window slides across the input and applies a pooling function to each window's contents.Pooling resembles discrete convolution but replaces the kernel's linear combination with another function.
- Output size: Pooling output size depends on input size, pooling-window size, and stride along each axis.
- Examples: The guide illustrates average pooling and max pooling using 3 × 3 windows on 5 × 5 inputs with 1 × 1 strides.
Convolution arithmetic
The arithmetic analysis is simplified because convolution properties act independently across axes, while the resulting relationships generalize to N-D and non-square cases.
- Axis independence: Kernel size, stride, and zero padding along one axis affect only the output size of that same axis.
- Generalization: The simplified analysis facilitates visualization while remaining applicable to N-D and non-square cases.
2.1 No zero padding, unit strides
With unit strides and no padding, output size follows the number of valid kernel placements; padding changes the effective input size and enables same-size or larger outputs.
- No padding, unit strides: The output size equals the number of kernel placements along each axis, counting the initial position.The kernel slides one position at a time until it reaches the input boundary.
- No padding, unit strides: Relationship 1 gives the output size for arbitrary input and kernel sizes when s = 1 and p = 0.
- Arbitrary padding, unit strides: Padding p changes the effective input size from i to i + 2p, yielding Relationship 2 for unit-strided convolutions.The padded-input view extends the no-padding placement argument.
- Half padding: Same-size output is desirable and occurs for odd k with half padding p = ⌊k/2⌋.This is also called same padding; for k = 3, p = 1.
- Full padding: Full padding uses p = k − 1 and s = 1, producing outputs larger than the input by including every partial or complete kernel superimposition.For i = 5 and k = 3, full padding uses p = 2.
2.3 No zero padding, non-unit strides
Non-unit strides reduce output size by counting kernel placements separated by s steps; padding is incorporated through an effective input size, with flooring leaving some positions uncovered.
- No padding, non-unit strides: With p = 0, output size equals the number of stride-s kernel placements plus one, captured by Relationship 5.The same placement-counting logic applies independently along width and height.
- No padding, non-unit strides: The floor function accounts for cases where the final stride does not reach the input boundary, leaving some input units unused.Figure 2.7 illustrates this behavior.
- Zero padding, non-unit strides: For arbitrary padding and strides, Relationship 6 applies the no-padding formula to an effective input of size i + 2p.
- Zero padding, non-unit strides: When i + 2p − k is a multiple of s, inputs differing by up to s − 1 can produce the same output size.This ambiguity occurs only when s > 1 and is illustrated by inputs of sizes 5 and 6 with k = 3, s = 2, p = 1.
Pooling arithmetic
Pooling computes local aggregates over sliding windows and shares convolution arithmetic for output-size analysis, including for any pooling function.
- Pooling operation: Pooling slides a window across the input and applies a pooling function, such as maximum or average, to each window.Max pooling provides translation invariance to small input shifts.
- Pooling arithmetic: Relationship 7 gives the output-size relationship for pooling for arbitrary input, kernel, and stride sizes.
- Pooling arithmetic: Relationship 7 holds for any type of pooling.The shared arithmetic covers max, mean, and average pooling.
4.1 Convolution as a matrix operation
A convolution can be represented as a sparse matrix operation, making its transposed operation the matrix transpose with reversed input-output dimensionality while preserving connectivity.
- Convolution as a matrix operation: Flattening the input and output represents convolution as a sparse matrix C whose nonzero entries are kernel weights.
- Convolution as a matrix operation: For the Figure 2.1 example, C maps a 16-dimensional input vector to a 4-dimensional output vector reshaped as 2 × 2.
- Convolution as a matrix operation: The backward pass multiplies by C^T, mapping the 4-dimensional error vector to 16 dimensions with connectivity compatible with C.
- Convolution as a matrix operation: The same kernel w defines both C and C^T for the convolution's forward and backward passes.
- Transposed convolution: A transposed convolution maps from the convolution's output space to its input space while preserving the convolutional connectivity pattern.Its forward and backward passes use C^T and C, respectively.
- Transposed convolution: Direct convolution can emulate transposed convolution, but usually requires adding zero rows and columns, making the implementation less efficient.
4.3 No zero padding, unit strides, transposed
A transposed convolution recovers the spatial shape of the feature map that produced its input, while preserving the kernel and stride and implementing equivalent padding through the associated direct convolution.
- A transposed convolution maps the input feature map back to the original feature-map shape, but is not the inverse convolution.It recovers width and height rather than necessarily reconstructing the original values.
- A 3 × 3 convolution on a 4 × 4 input with unit stride and no padding produces 2 × 2, whose transpose outputs 4 × 4.
- The equivalent direct convolution keeps the kernel and stride unchanged while padding the transposed-convolution input with zeros.For the example, a 2 × 2 input receives a 2 × 2 zero border.
- Connectivity determines the padding: padding of k − 1 makes the first kernel application touch only the corresponding corner pixel.The same connectivity reasoning extends across the remaining input elements.
- For unit stride and no padding, the associated transposed convolution uses k′ = k, s′ = s, and p′ = k − 1.
4.4 Zero padding, unit strides, transposed
With unit strides, transposed convolutions adjust the equivalent direct-convolution padding according to the original padding, preserving kernel and stride; half padding is self-equivalent, while full padding becomes no padding.
- For an original convolution with padding p, the associated transpose keeps k′ = k and s′ = s but uses p′ = k − p − 1.
- 4.4.1 Half (same) padding, transposed: Half-padded convolutions with odd k = 2n + 1 and p = n are self-equivalent under transposition, so k′ = k, s′ = s, and p′ = p.Their output size equals their input size.
- 4.4.1 Half (same) padding, transposed: A 3 × 3 half-padded convolution on a 5 × 5 input provides the concrete case k = 3 and p = 1.
- The figures illustrate that zero-padding transformations depend on the original padding: p = 0 maps to p′ = k − 1, while padded cases require less equivalent padding.
- 4.4.2 Full padding, transposed: A fully padded convolution with p = k − 1 becomes an equivalent transposed convolution with no padding, p′ = 0.
4.5 No zero padding, non-unit strides, transposed
Transposed convolutions with non-unit strides can be understood as unit-stride convolutions over inputs stretched by inserting zeros. The relationships specify how kernel size, padding, output size, and edge additions correspond across several padding and input-size cases.
- Fractional strides: Non-unit-stride transposed convolutions can be interpreted as convolutions with fractional strides, where zeros are inserted between input units.The inserted zeros make the kernel move more slowly than with unit strides, although implementations avoid unnecessary multiplications by zero.
- No zero padding: For unpadded convolutions, Relationship 12 uses a stretched input, k′ = k, s′ = 1, and p′ = k − 1 for the associated transposed convolution.This relationship assumes p = 0 and that i − k is divisible by s.
- Examples: The figures illustrate these correspondences for unit-stride full padding, 2 × 2 strides with no padding, one-pixel padding, and an additional bottom-right edge border.Examples include 5 × 5 inputs with k = 3 and cases using stretched 2 × 2 or 3 × 3 inputs.
- Zero padding: For zero-padded convolutions satisfying i + 2p − k divisible by s, Relationship 13 retains k′ = k and s′ = 1 while setting p′ = k − p − 1.The stretched input is formed by inserting s − 1 zeros between adjacent input units.
- Relaxed input-size constraint: Relationship 14 removes the divisibility constraint by adding a ∈ {0, . . . , s − 1} zeros to the stretched input’s bottom and right edges, with a = (i + 2p − k) mod s.The associated transposed convolution still uses k′ = k, s′ = 1, and p′ = k − p − 1.
Miscellaneous convolutions
Dilated convolutions enlarge the kernel’s effective span by inserting spaces between kernel elements. The guide relates this dilation-controlled effective kernel size to output-size calculations and illustrates the idea with a concrete example.
- Definition: Dilated convolutions inflate kernels by inserting spaces between kernel elements, with dilation rate d controlling the spacing and d = 1 recovering regular convolution.Implementations usually insert d − 1 spaces between neighboring kernel elements.
- Motivation and use: Their practical purpose is to increase output units’ receptive fields without increasing kernel size, particularly when several dilated convolutions are stacked.The passage gives WaveNet’s autoregressive raw-audio model as an application using large past contexts.
- Output size: The dilation rate’s effect on output size can be analyzed by treating a dilated kernel as having an effective kernel size and combining that size with the standard output relationship.Relationship 15 is stated for any i, k, p, and s with dilation rate d.
- Example: Figure 5.1 illustrates a dilated convolution with i = 7, k = 3, and d = 2.The example uses a unit stride and no padding.