PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolution in Convolution for Network in Network

Yanwei Pang, Manli Sun, Xiaoheng Jiang, Xuelong Li

arXiv:1603.06759v1cs.CV

TL;DR

NiN improves CNN filtering with nonlinear MLPs, but its fully connected MLPs require many parameters. CiC replaces them with sparse convolutionally formed MLPs, and experiments report lower error on CIFAR10, augmented CIFAR10, and CIFAR100 datasets.

  • Problem

    NiN’s fully connected MLP filters require many parameters to compute and store, creating a parameter burden.

  • Method

    CiC replaces dense shallow MLP filters with sparse shallow MLPs implemented using unshared channel-wise and shared spatial convolution.

  • Results

    CiC-3D outperforms NiN by 1.95 percent on CIFAR10 and achieves a 31.40% test error rate on CIFAR100, versus NiN’s 35.68%.

  • Takeaways & Limitations

    The experiments demonstrate the effectiveness of CiC across CIFAR10, augmented CIFAR10, and CIFAR100 evaluations.

Abstract

from arXiv · show

Network in Netwrok (NiN) is an effective instance and an important extension of Convolutional Neural Network (CNN) consisting of alternating convolutional layers and pooling layers. Instead of using a linear filter for convolution, NiN utilizes shallow MultiLayer Perceptron (MLP), a nonlinear function, to replace the linear filter. Because of the powerfulness of MLP and $ 1\times 1 $ convolutions in spatial domain, NiN has stronger ability of feature representation and hence results in better recognition rate. However, MLP itself consists of fully connected layers which give rise to a large number of parameters. In this paper, we propose to replace dense shallow MLP with sparse shallow MLP. One or more layers of the sparse shallow MLP are sparely connected in the channel dimension or channel-spatial domain. The proposed method is implemented by applying unshared convolution across the channel dimension and applying shared convolution across the spatial dimension in some computational layers. The proposed method is called CiC. Experimental results on the CIFAR10 dataset, augmented CIFAR10 dataset, and CIFAR100 dataset demonstrate the effectiveness of the proposed CiC method.

I. INTRODUCTION

CNNs use learned linear filters, while NiN replaces them with nonlinear shallow MLP filters to improve representation. CiC addresses NiN’s fully connected MLP parameter burden by introducing sparse connectivity.

  • NiN uses a shallow MLP as a convolutional filter, making the filter itself a network and increasing nonlinear representational power.
  • NiN’s fully connected MLP requires many parameters to compute and store, limiting its performance.
  • CiC modifies NiN by replacing dense shallow MLPs with locally connected sparse MLPs obtained by applying kernels within the MLP.
  • CNNs alternate convolutional and pooling layers, with convolutional filters providing feature representation and pooling improving translation invariance and efficiency.
  • CNN performance can be pursued through changes to depth, width, convolution, pooling, parameter count, or activation functions; CiC directly modifies convolution in NiN.

III. PROPOSED METHOD: CIC

CiC replaces NiN’s dense MLP filters with full or partial sparse MLPs formed through convolution. Its design uses shared spatial convolution and unshared channel-wise convolution, with unshared convolution preferred experimentally for lower test error.

  • CiC uses sparse shallow MLPs as convolutional filters, with the MLPs themselves obtained by convolution.
  • CiC applies shared convolution spatially and unshared convolution across channels; experiments found unshared convolution better for reducing test error.
  • The proposed sparse MLPs include full sparse and partial sparse forms, with labels indicating which layers are locally connected.
  • A dense two-hidden-layer MLP has 80 parameters, whereas the parameter-shared full sparse counterpart has 9 parameters and the unshared version has 36.
  • MLP-010 denotes a partial sparse MLP whose first hidden layer is dense, second hidden layer sparse, and output layer dense.

B. CiC with Sparse and Shallow MLP (One-dimensional Filtering across Channels with Large Filter): CiC-1D

CiC-1D embeds sparse MLP-010 blocks in a CNN and uses one-dimensional filtering across channels with 1 × 1 spatial convolutions. Its kernels implement sparse channel connections without channel-direction padding, alongside standard normalization, activation, pooling, and dropout components.

  • CiC-1D treats sparse MLPs as convolution filters inserted into the CNN framework, using unshared convolution to connect channels.
  • CiC-1D uses three building blocks, each receiving a spatial feature map and channel input and applying an MLP-010 computation.
  • Each block contains kernels for the input, first hidden, and second layers of MLP-010, represented as spatial-channel tensors with optional output-channel indexing.
  • The first kernel processes color-image inputs with a spatial size larger than 1 and densely connects the three input color channels.
  • The second kernel uses a 1 × 1 spatial filter whose channel length is smaller than the input channel count, producing sparse channel connections.
  • CiC-1D applies no padding along the channel direction, while its CNN blocks also use batch normalization, ReLU, max-pooling, and dropout.

C. Generalized CiC with Three-dimensional Filtering across Channel-Spatial Domain: CiC-3D

CiC-3D generalizes CiC-1D by replacing spatially independent 1 × 1 convolutions with n × n convolutions, while retaining channel-dimension filtering. The generalized kernels integrate spatial and channel-spatial processing, with n = 5 reported as effective.

  • CiC-3D design: CiC-3D changes selected kernels from 1 × 1 spatial filters to n × n filters with n > 1.The modified kernels are K1, K4, and K7 in the CiC blocks.
  • CiC-3D design: The generalized kernels combine spatial filtering with channel-dimension filtering instead of independently processing each spatial location.CiC-1D uses 1 × 1 spatial convolution and one-dimensional channel convolution; CiC-3D removes the spatial independence through larger kernels.
  • Kernel specification: The kernel specification uses three-order tensors and corresponding four-order tensors for the convolutional filters.Table I presents the kernel sizes and constraints for these tensors.
  • CiC-3D design: n = 5 is reported as generally yielding good performance for CiC-3D.

IV. EXPERIMENTAL RESULTS

The experiments compare CiC-1D and CiC-3D with several neural-network baselines across CIFAR10, CIFAR100, and augmented versions of these datasets. The implementations use the MatConvNet toolbox.

  • Compared methods: CiC-1D and CiC-3D are compared with NiN, Maxout Network, Probabilistic Maxout Network, Deeply Supervised Network, and NiN+LA units.
  • Datasets: The evaluation covers CIFAR10, CIFAR100, and augmented versions of both datasets.
  • Implementation: The proposed methods are implemented using the MatConvNet toolbox.

A. Datasets

The study evaluates CiC on CIFAR10, augmented CIFAR10 variants, and CIFAR100. These datasets vary in augmentation procedure, class count, and classification difficulty.

  • CIFAR10: CIFAR10 contains 60,000 32 × 32 color images across 10 classes, with 50,000 for training and 10,000 for testing.
  • Augmented datasets: CIFAR10+ is created by randomly flipping training images and is used for parameter tuning and intermediate results.
  • Augmented datasets: CIFAR10++ is created by padding images by 4 pixels, then randomly cropping and flipping them on the fly.
  • CIFAR100: CIFAR100 has the same numbers of training and testing images as CIFAR10 but contains 100 classes grouped into 20 superclasses.Each image has a fine class label and a coarse superclass label.
  • CIFAR100: CIFAR100 provides fewer images per class than CIFAR10 and is described as more challenging.

B. Configurations and Intermediate Results on the CIFAR10+ Dataset

The authors tune CiC configurations on CIFAR10+ by varying kernel length, channel count, and the blocks using sparse MLP-010. These experiments select parameters and motivate the CiC-3D configuration used thereafter.

  • Parameter selection: The authors greedily select CiC-1D kernel parameters, then reuse the obtained optimal parameters for CiC-3D.Each parameter is varied while the others remain fixed.
  • Kernel length: A channel-dimension kernel length of 3 produces the lowest error rate, 9.07%, for MLP-010.This value is selected whenever MLP-010 provides sparse connections.
  • Channel count: The best classification performance occurs when MLP-010 has 224 input channels.The tested channel counts are 160, 192, 224, and 256.
  • Block configuration: Applying MLP-010 in more blocks is beneficial, and CiC-3D is configured with MLP-010 in all three blocks.The selected sparse-layer configuration uses 224 input channels and kernel length 3.
  • CiC variant: CiC-3D significantly outperforms CiC-1D, so subsequent experiments use CiC-3D with MLP-010 in every block.The main parameters and operations are shown in Fig. 6.

C. Comparison with Other Methods on the CIFAR10 Dataset

On CIFAR10, the experiments compare CiC-3D with NiN and other methods using the original dataset. CiC-3D achieves lower test error than the cited baselines and converges faster than NiN.

  • Training behavior: CiC-3D has smaller training error rates and converges much faster than NiN on CIFAR-10.The comparison is shown through training-error curves versus training epochs.
  • Test performance: CiC-3D outperforms NiN by 1.95 percent on the CIFAR10 test error rate.Table VI compares CiC-3D with NiN, DSN, NiN-LA, and RCNN-160.
  • Test performance: CiC-3D improves over NiN-LA by 1.13 percent on CIFAR10 test error rate.NiN-LA is one of the methods listed in Table VI.

D. Comparison with Other Methods on the CIFAR10++ Dataset

On the larger CIFAR10++ dataset, CiC-3D has lower test error than NiN, and the reported advantage increases as the training set grows.

  • Test performance: CiC-3D reaches 6.68% test error versus NiN’s 8.81% on CIFAR10++.The reported difference is a 2.13% improvement over NiN.
  • Scaling with training data: The superiority of CiC-3D over NiN grows as the CIFAR10++ training set increases.CIFAR10++ is constructed through padding, random cropping, and random flipping.

E. Comparison with Other Methods on the CIFAR100 Dataset

On CIFAR100, CiC-3D records the lowest test error among the compared methods and improves over both NiN and NiN-LA.

  • Test performance: CiC-3D achieves the lowest listed CIFAR100 test error rate, 31.40%.The comparison includes NiN, NiN-LA, Highway, and RCNN-160.
  • Baseline comparisons: CiC-3D outperforms NiN by 4.28 percent and NiN-LA by 3 percent on CIFAR100.These comparisons are based on the test error rates in Table VIII.

V. CONCLUSION AND FUTURE WORK

CiC uses sparse shallow MLPs for convolution, including partial variants in experiments and one-dimensional and three-dimensional forms. The authors identify broader sparse-MLP implementations and integration with other CNNs as future directions.

  • CiC uses sparse shallow MLPs for convolution, including fully sparse and several partial variants.
  • The experiments adopt the partial sparse MLP variant MLP-010.
  • CiC-1D connects channels with a one-dimensional filter, while CiC-3D extends filtering across the channel-spatial domain.
  • Future work includes implementing full and other sparse MLP types and integrating the idea into other state-of-the-art CNNs.
Loading 1603.06759v1…