PaperlayerView exact revision on arXiv

Source-linked AI summary

Network In Network

Min Lin, Qiang Chen, Shuicheng Yan

arXiv:1312.4400v3cs.NEcs.CVcs.LG

TL;DR

Conventional CNN filters provide limited local abstraction when concept variations are nonlinear. NIN replaces them with shared MLP-based micro-networks and achieves state-of-the-art classification on CIFAR-10, CIFAR-100, and SVHN.

  • Problem

    Conventional CNN filters model receptive-field patches linearly, limiting abstraction when variants of the same concept lie on nonlinear manifolds.

  • Method

    NIN stacks mlpconv layers that apply shared multilayer perceptrons to local patches and uses global average pooling instead of fully connected classification layers.

  • Results

    NIN achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and SVHN, including an 8.81% CIFAR-10 test error with augmentation and 35.68% on CIFAR-100.

  • Takeaways & Limitations

    NIN supports stronger local modeling and interpretable category confidence maps while global average pooling serves as a structural regularizer against overfitting.

Abstract

from arXiv · show

We propose a novel deep network structure called "Network In Network" (NIN) to enhance model discriminability for local patches within the receptive field. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to abstract the data within the receptive field. We instantiate the micro neural network with a multilayer perceptron, which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner as CNN; they are then fed into the next layer. Deep NIN can be implemented by stacking mutiple of the above described structure. With enhanced local modeling via the micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers. We demonstrated the state-of-the-art classification performances with NIN on CIFAR-10 and CIFAR-100, and reasonable performances on SVHN and MNIST datasets.

1 Introduction

Network In Network replaces CNN’s linear local filters with shared multilayer perceptrons that model receptive fields more nonlinearly. By stacking these micro-network layers and using global average pooling for classification, NIN aims to improve local abstraction while avoiding traditional fully connected layers.

  • Motivation: CNN convolutional filters are generalized linear models whose abstraction is limited when local concept variations are not linearly separable.The paper defines abstraction as invariance to variants of the same concept.
  • mlpconv layers: An mlpconv layer maps each local receptive field to an output feature vector using a shared multilayer perceptron with nonlinear activations.The MLP consists of multiple fully connected layers and is applied by sliding across the input similarly to CNN convolution.
  • Network In Network: NIN stacks multiple mlpconv layers, embedding multilayer perceptrons as micro networks within the overall deep network.The resulting feature maps are fed into subsequent layers.
  • Classification: NIN replaces traditional fully connected classification layers with global average pooling over the final feature maps before softmax.The pooled feature-map averages serve as category confidence values and are described as easier to interpret than fully connected layers.

2 Convolutional Neural Networks

Conventional CNNs alternate convolutional and pooling layers, using linear filters plus nonlinear activations to form feature maps. Because linear convolution may inadequately model nonlinear latent concepts, the section motivates more general local function approximators, including NIN’s micro networks.

  • Conventional CNNs: Conventional CNNs alternate convolutional and spatial pooling layers, with convolutional filters followed by nonlinear activation functions producing feature maps.The activations mentioned include rectifier, sigmoid, and tanh functions.
  • Limitations: Linear convolution is sufficient when latent concepts are linearly separable, but strong abstractions are generally highly nonlinear functions of the input.Conventional CNNs may compensate by using over-complete filter sets that detect different variations of the same concept.
  • Related Work: Maxout networks use maximum pooling over affine feature maps to create piecewise-linear approximators capable of approximating any convex function.This gives maxout greater separation capacity for concepts lying within convex sets than conventional convolutional layers.
  • Motivation for NIN: Maxout imposes a convex-set prior that may not hold for complex latent-concept distributions, motivating more general function approximators.NIN introduces a micro network within each convolutional layer to compute more abstract features for local patches.
  • Related Work: Sliding neural networks over input patches had appeared in SMLP and neural-network face-detection filters, but those methods targeted specific problems and used only one sliding-network layer.NIN is presented from a more general perspective.

3 Network In Network

Network In Network replaces conventional linear convolutional filters with multilayer perceptrons applied to local patches, then uses global average pooling for classification. Its overall architecture stacks flexible mlpconv layers above the pooling and objective layers.

  • 3.1 MLP Convolutional Layer: NIN uses a multilayer perceptron as a universal function approximator for extracting abstract representations from local patches.The authors choose MLPs because they are compatible with convolutional neural network structure and can be trained with backpropagation.
  • 3.1 MLP Convolutional Layer: The MLP convolution can be interpreted as cascaded cross-channel parametric pooling with learnable feature-map interactions and rectified linear units.Each layer performs weighted linear recombination followed by a rectifier, and the process is repeated across layers.
  • 3.1 MLP Convolutional Layer: Cross-channel parametric pooling is equivalent to convolution with a 1x1 kernel, providing a straightforward interpretation of the NIN structure.This equivalence connects the micro-network operation to a standard convolutional formulation.
  • 3.2 Global Average Pooling: Global average pooling replaces fully connected layers by averaging one category-associated feature map per class and feeding the resulting vector directly to softmax.The method is presented as a structural regularizer that encourages feature maps to represent category concepts.
  • 3.3 Overall NIN: The overall NIN stacks mlpconv layers followed by global average pooling and an objective cost layer, with optional subsampling and flexible depths.The illustrated architecture contains three mlpconv layers, each with a three-layer perceptron, while the number of layers can be tuned for specific tasks.

4 Experiments

NIN was evaluated on CIFAR-10, CIFAR-100, SVHN, and MNIST using stacked mlpconv networks, achieving strong test performance across datasets. Experiments also show that dropout and global average pooling improve generalization, while feature maps reflect category-level activations.

  • Experimental setup: NIN uses three stacked mlpconv layers with spatial max pooling, dropout between all but the last mlpconv layers, and global average pooling in the classification layer.The four benchmark evaluations use this general architecture, with each pooling layer downsampling the input by a factor of two.
  • CIFAR-10: 8.81% test error on CIFAR-10 with translation and horizontal-flipping augmentation establishes new state-of-the-art performance.Without augmentation, the model obtains 10.41% test error; the model without dropout obtains 14.51%.
  • CIFAR-100: 35.68% test error on CIFAR-100 surpasses the current best performance without data augmentation.The CIFAR-100 network uses the CIFAR-10 settings, changing the final mlpconv layer to output 100 feature maps.
  • SVHN: 2.35% test error on SVHN is obtained using a three-mlpconv-layer network followed by global average pooling, without augmented-data comparisons.The result is compared with methods that did not augment the data.
  • MNIST: 0.47% test error on MNIST is comparable to, but worse than, the current best result of 0.45%.The experiment uses the CIFAR-10 network structure with fewer feature maps and no data augmentation.
  • Ablation and visualization: Global average pooling achieves the lowest testing error at 10.41%, compared with 11.59% for an unregularized fully connected layer and 10.88% with dropout.Replacing a fully connected layer with global average pooling also reduces CNN error from 17.56% to 16.46%.

5 Conclusions

The paper proposes Network In Network (NIN), combining mlpconv layers with global average pooling for classification. Mlpconv layers improve local-patch modeling, while global average pooling replaces fully connected layers and acts as a structural regularizer against overfitting.

  • Conclusions: NIN is a novel deep network for classification tasks.Its structure uses mlpconv layers and global average pooling.
  • Conclusions: Mlpconv layers use multilayer perceptrons to convolve inputs and model local patches better.They are a central component of the proposed NIN structure.
  • Conclusions: Global average pooling replaces fully connected layers in conventional CNNs and acts as a structural regularizer that prevents global overfitting.The pooling layer is the second principal component of NIN described in the conclusion.
Loading 1312.4400v3…