PaperlayerView exact revision on arXiv

Source-linked AI summary

MatConvNet - Convolutional Neural Networks for MATLAB

Andrea Vedaldi, Karel Lenc

arXiv:1412.4564v3cs.CVcs.LGcs.MScs.NE

TL;DR

CNN research requires implementations that are both efficient at large scale and easy to adapt, while existing CNN operations can be difficult to implement efficiently. MatConvNet addresses this by exposing composable CNN building blocks in MATLAB and supporting CPU/GPU computation; AlexNet reaches 264 images/s with CuDNN, about 40% faster than its vanilla GPU implementation and more than 10 times faster than CPUs.

  • Problem

    CNNs require efficient implementations because they are learned from millions of images and may contain several million parameters, while implementing even simple operations efficiently is nontrivial.

  • Method

    MatConvNet integrates CNN building blocks into MATLAB as simple composable functions, with optimized CPU/GPU implementations and support for backpropagation and large-model training.

  • Results

    264 images/s: AlexNet trains with CuDNN, about 40% faster than the vanilla GPU implementation and more than 10 times faster than CPUs.

  • Takeaways & Limitations

    MatConvNet provides a MATLAB-based environment for rapidly prototyping CNN architectures while training complex models on large datasets.

  • Takeaways & Limitations

    The library currently relies on MATLAB from the user perspective, although its clean separation from the C++ and CUDA core may allow future independent processing.

Abstract

from arXiv · show

MatConvNet is an implementation of Convolutional Neural Networks (CNNs) for MATLAB. The toolbox is designed with an emphasis on simplicity and flexibility. It exposes the building blocks of CNNs as easy-to-use MATLAB functions, providing routines for computing linear convolutions with filter banks, feature pooling, and many more. In this manner, MatConvNet allows fast prototyping of new CNN architectures; at the same time, it supports efficient computation on CPU and GPU allowing to train complex models on large datasets such as ImageNet ILSVRC. This document provides an overview of CNNs and how they are implemented in MatConvNet and gives the technical details of each computational block in the toolbox.

5 Geometry

Section 5 develops methods for analyzing receptive fields and their transformations in CNNs.

  • 5.1 Preliminaries: The section begins with preliminaries for receptive-field analysis.
  • 5.2 Simple filters: It introduces simple filters and discusses pooling in Caffe.
  • 5.3–5.4: It covers transposed convolution and transposition of receptive fields.
  • 5.5–5.6: It concludes with methods for composing and overlaying receptive fields.

6 Implementation details

Section 6 presents implementation details for core CNN computational blocks, including convolution, pooling, activations, and resampling.

  • 6.1–6.2: The section describes convolution and transposed convolution operations.
  • 6.3 Spatial pooling: It explains spatial pooling as a CNN computational block.
  • 6.4 Activation functions: It covers ReLU and sigmoid activation functions.
  • 6.5 Spatial bilinear resampling: It includes spatial bilinear resampling.

Introduction to MatConvNet

MatConvNet is designed as a MATLAB-centered, simple and flexible CNN toolbox for research, exposing composable building blocks while supporting efficient CPU/GPU training of large models. It provides accessible workflows and pretrained networks, with ImageNet training speeds that are competitive with Caffe and substantially faster with CuDNN than vanilla GPU or CPU implementations.

  • Motivation: MatConvNet implements CNNs for computer vision and emphasizes efficient computation for large-scale learning.
  • Design philosophy: Its MATLAB integration exposes convolution, normalization, pooling, and other CNN building blocks as simple commands that researchers can combine and extend.
  • Capabilities: The toolbox supports large models such as AlexNet and very deep networks trained on millions of images, while remaining simple to install and use.
  • Getting started: The sequential wrapper vl_simplenn applies configured building blocks to a CNN and returns a structure of results from the network and preprocessed image.
  • Design philosophy: The design exposes linear convolution and ReLU directly in MATLAB, enabling building blocks to be combined into complete CNNs and learning algorithms without often requiring C coding.
  • Speed: 264 images/s: AlexNet trains with CuDNN, about 40% faster than the vanilla GPU implementation and more than 10 times faster than CPUs.
  • Speed: 20 to 45 images/s: VGG-VD-16 training speed increases when distributing sub-batches across one to four GPUs, despite substantial communication overhead.

Neural Network Computations

Neural networks compose parameterized computational blocks, while CNNs apply this structure to spatial tensors and batches of feature maps. MatConvNet supports these computations, including DAG evaluation and memory-efficient backpropagation for learning.

  • Network computations: A neural network maps input data to outputs by composing parameterized computational blocks or layers.Each layer computes x_l = f_l(x_{l-1}; w_l), with the network represented as g = f_L ◦ ... ◦ f_1.
  • CNN structure: CNN tensors encode height, width, and feature channels, with an additional batch dimension for processing multiple samples efficiently.Spatial dimensions identify feature-vector locations, while channels store the feature-vector components at each location.
  • MatConvNet blocks: MatConvNet provides MATLAB layers for convolution, pooling, activations, normalization, softmax, loss functions, and related operations.These layers can implement state-of-the-art networks or support importing architectures from other toolboxes such as Caffe.
  • Learning: CNN training minimizes a loss over labelled examples, typically using stochastic gradient descent and derivatives with respect to network parameters.Efficiency matters because networks may contain millions of parameters and require millions of training images.
  • Network structures: A network may use a directed acyclic graph rather than a simple sequence, provided every layer is evaluated only after its inputs are available.DAGs support functions with multiple inputs or outputs, shared parameters, and evaluation through a topological ordering.
  • Backpropagation: Backpropagation applies the chain rule to compute parameter derivatives without explicitly storing the full Jacobian, whose size can reach approximately 17 × 10^9 elements.For the stated moderate tensor dimensions, storing that Jacobian would require 68 GB in single precision; the completed procedure yields derivatives for network variables.

Wrappers and pre-trained models

MatConvNet provides wrappers that assemble CNN computational blocks into either simple chains or more flexible directed acyclic graphs. It also includes pre-trained models and support for evaluating and extending CNNs.

  • Pre-trained models: MatConvNet includes pre-trained models for image classification, image segmentation, text spotting, and face recognition.
  • MatConvNet offers SimpleNN for linear chains and DagNN for computational blocks organized in more complex directed acyclic graphs.
  • SimpleNN: SimpleNN evaluates CNNs and their derivatives and includes support for moving networks between CPU and GPU and displaying network information.
  • DagNN: DagNN supports arbitrary graph topologies through an object-oriented layer design, trading slight overhead on tiny networks for greater flexibility and extensibility.

22 CHAPTER 3. WRAPPERS AND PRE-TRAINED MODELS

MatConvNet supports practical CNN use through pre-trained-model workflows and training examples. Its examples cover image preparation, model execution, classification, and large-scale training, with reported throughput of 200-300 images/sec.

  • Pre-trained models: Pre-trained CNNs can be downloaded, configured in MATLAB, and applied to images after model-specific preprocessing.
  • Inference: The network returns layer outputs, and the final layer can be used for image classification using class names stored in the model structure.
  • Inference: Multiple crops can be averaged for improved results, and network outputs can also serve as generic image-encoding features.
  • Learning models: Backpropagation derivatives make learning algorithms straightforward, with flexible stochastic-gradient-descent examples for NMINST, CIFAR, ImageNet, and other datasets.
  • Large-scale training: 200-300 images/sec is the expected training throughput for the ImageNet example when the recommended setup is ready.

Computational blocks

MatConvNet exposes CNN operations as MATLAB functions with forward and derivative-computation interfaces. Its convolution and convolution-transpose blocks support configurable spatial transformations, channel grouping, and applications such as interpolation and decoding.

  • Computational blocks: Each computational block maps input arrays and parameters to an output array, while optional derivatives enable backward computation with respect to inputs and parameters.
  • Computational blocks: Block functions may accept optional property-value arguments, omit parameters, or handle multiple inputs and parameters.
  • Convolution: vl_nnconv computes convolutions between an input map and a bank of multidimensional filters with biases, supporting arbitrarily shaped inputs and filters.
  • Convolution: Convolution supports zero padding, output subsampling strides, valid filter placement, receptive-field analysis, fully connected layers as a special case, and grouped channels.
  • Convolution transpose: Convolution transpose computes the transpose of the linear convolution operator and supports input upsampling and output cropping.
  • Convolution transpose: The convolution-transpose output height is determined by a closed-form expression, with an analogous formula for width and corresponding receptive-field analysis.

Geometry

This section formalizes how CNN operators map input components to output components through receptive-field geometry. It derives rules for common filters, pooling, convolution transpose, and composed or transposed fields, while noting limitations from runtime behavior and rounding.

  • Receptive-field representation: Receptive fields represent each output component as depending on a rectangular input window characterized by stride, offset, and field size.The geometry is parameterized by (αh, αv), (βh, βv), and (∆h, ∆v).
  • Filters: Convolutional filters use integer size, stride, and padding to determine receptive-field geometry and output dimensions.The same reasoning applies independently along vertical and horizontal directions.
  • Pooling: MatConvNet treats pooling as filter-like, but its output-size and boundary rules differ from Caffe when windows reach padded boundaries.Caffe may use ceil-based output sizing and allow the final pooling window to extend outside the input, producing compatibility adjustments.
  • Convolution transpose: Convolution transpose reverses convolution dependencies, with cropping and upsampling becoming padding and downsampling in the reverse direction.Its receptive-field analysis uses the convolution relation after exchanging input and output roles.
  • Convolution transpose: Rounding prevents an exact tight sliding-window description for convolution-transpose receptive fields, so the analysis relaxes the relations and may produce a slightly larger field.The output height can also have multiple possible values for a fixed input height because some samples may be ignored.
  • Composition and transposition: Receptive-field transformations compose through stride, offset, and size rules, and transposing a field yields the reverse-direction transformation.The convolution example recovers the convolution-transpose formulas from these general transformations.

Implementation details

This section describes matrix-based implementations and derivatives for MatConvNet operators. It shows how reshaping and sparse or structured linear operators support efficient convolution, pooling, nonlinearities, resampling, normalization, and loss gradients.

  • Convolution implementation: The im2row operator extracts every input patch into rows, enabling convolution to be expressed using matrix multiplication.The corresponding row2im operator provides the transposed patch-accumulation operation.
  • Convolution implementation: Matrix formulations of convolution derivatives support implementation with BLAS and GPU BLAS, which is efficient when the number of filters is large.Although the formulation may appear inefficient, the implementation leverages optimized linear-algebra routines.
  • Convolution transpose: Convolution transpose is defined as multiplication by the transpose of the matrix representing the corresponding padded and downsampled convolution.The index-level derivation uses the identity vec y^T(M vec x) = vec x^T(M^T vec y).
  • Pooling and nonlinearities: Max pooling uses a selector matrix, so its backward derivative applies the selector transpose to the output derivative.The relation holds except at the nondifferentiable ties, a null set of points.
  • Pooling and nonlinearities: ReLU uses a diagonal matrix formed from the positive-input indicator, and its derivative applies the same gating pattern to the upstream derivative.The indicator vector is s = [vec x > 0].
Loading 1412.4564v3…