PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolution in Convolution for Network in Network

Yanwei Pang, Manli Sun, Xiaoheng Jiang, Xuelong Li

arXiv:1603.06759v1cs.CV

TL;DR

NiN improves CNN filtering with nonlinear MLPs, but its fully connected MLPs require many parameters. CiC replaces them with sparse convolutionally formed MLPs, and experiments report lower error on CIFAR10, augmented CIFAR10, and CIFAR100 datasets.

Abstract

from arXiv · show

Network in Netwrok (NiN) is an effective instance and an important extension of Convolutional Neural Network (CNN) consisting of alternating convolutional layers and pooling layers. Instead of using a linear filter for convolution, NiN utilizes shallow MultiLayer Perceptron (MLP), a nonlinear function, to replace the linear filter. Because of the powerfulness of MLP and $ 1\times 1 $ convolutions in spatial domain, NiN has stronger ability of feature representation and hence results in better recognition rate. However, MLP itself consists of fully connected layers which give rise to a large number of parameters. In this paper, we propose to replace dense shallow MLP with sparse shallow MLP. One or more layers of the sparse shallow MLP are sparely connected in the channel dimension or channel-spatial domain. The proposed method is implemented by applying unshared convolution across the channel dimension and applying shared convolution across the spatial dimension in some computational layers. The proposed method is called CiC. Experimental results on the CIFAR10 dataset, augmented CIFAR10 dataset, and CIFAR100 dataset demonstrate the effectiveness of the proposed CiC method.

I. INTRODUCTION

CNNs use learned linear filters, while NiN replaces them with nonlinear shallow MLP filters to improve representation. CiC addresses NiN’s fully connected MLP parameter burden by introducing sparse connectivity.

  • NiN uses a shallow MLP as a convolutional filter, making the filter itself a network and increasing nonlinear representational power.Source: §I · p. 1, §II · p. 2
  • NiN’s fully connected MLP requires many parameters to compute and store, limiting its performance.Source: §I · p. 1
  • CiC modifies NiN by replacing dense shallow MLPs with locally connected sparse MLPs obtained by applying kernels within the MLP.Source: §I · p. 1
  • CNNs alternate convolutional and pooling layers, with convolutional filters providing feature representation and pooling improving translation invariance and efficiency.Source: §II · pp. 1–2
  • CNN performance can be pursued through changes to depth, width, convolution, pooling, parameter count, or activation functions; CiC directly modifies convolution in NiN.Source: §II · p. 2

III. PROPOSED METHOD: CIC

CiC replaces NiN’s dense MLP filters with full or partial sparse MLPs formed through convolution. Its design uses shared spatial convolution and unshared channel-wise convolution, with unshared convolution preferred experimentally for lower test error.

B. CiC with Sparse and Shallow MLP (One-dimensional Filtering across Channels with Large Filter): CiC-1D

CiC-1D embeds sparse MLP-010 blocks in a CNN and uses one-dimensional filtering across channels with 1 × 1 spatial convolutions. Its kernels implement sparse channel connections without channel-direction padding, alongside standard normalization, activation, pooling, and dropout components.

C. Generalized CiC with Three-dimensional Filtering across Channel-Spatial Domain: CiC-3D

CiC-3D generalizes CiC-1D by replacing spatially independent 1 × 1 convolutions with n × n convolutions, while retaining channel-dimension filtering. The generalized kernels integrate spatial and channel-spatial processing, with n = 5 reported as effective.

CiC-3D design

  • CiC-3D changes selected kernels from 1 × 1 spatial filters to n × n filters with n > 1.The modified kernels are K1, K4, and K7 in the CiC blocks.Source: C. Generalized CiC with… · p. 4
  • The generalized kernels combine spatial filtering with channel-dimension filtering instead of independently processing each spatial location.CiC-1D uses 1 × 1 spatial convolution and one-dimensional channel convolution; CiC-3D removes the spatial independence through larger kernels.Source: C. Generalized CiC with… · pp. 4–5

Kernel specification

  • The kernel specification uses three-order tensors and corresponding four-order tensors for the convolutional filters.Table I presents the kernel sizes and constraints for these tensors.Source: Table I · p. 5

CiC-3D design

IV. EXPERIMENTAL RESULTS

The experiments compare CiC-1D and CiC-3D with several neural-network baselines across CIFAR10, CIFAR100, and augmented versions of these datasets. The implementations use the MatConvNet toolbox.

Compared methods

  • CiC-1D and CiC-3D are compared with NiN, Maxout Network, Probabilistic Maxout Network, Deeply Supervised Network, and NiN+LA units.Source: §IV · p. 5

Datasets

  • The evaluation covers CIFAR10, CIFAR100, and augmented versions of both datasets.Source: §IV · p. 5

Implementation

  • The proposed methods are implemented using the MatConvNet toolbox.Source: §IV · p. 5

A. Datasets

The study evaluates CiC on CIFAR10, augmented CIFAR10 variants, and CIFAR100. These datasets vary in augmentation procedure, class count, and classification difficulty.

CIFAR10

  • CIFAR10 contains 60,000 32 × 32 color images across 10 classes, with 50,000 for training and 10,000 for testing.Source: A. Datasets · p. 5

Augmented datasets

  • CIFAR10+ is created by randomly flipping training images and is used for parameter tuning and intermediate results.Source: A. Datasets · p. 5
  • CIFAR10++ is created by padding images by 4 pixels, then randomly cropping and flipping them on the fly.Source: A. Datasets · p. 5

CIFAR100

  • CIFAR100 has the same numbers of training and testing images as CIFAR10 but contains 100 classes grouped into 20 superclasses.Each image has a fine class label and a coarse superclass label.Source: A. Datasets · p. 5
  • CIFAR100 provides fewer images per class than CIFAR10 and is described as more challenging.Source: A. Datasets · p. 5

B. Configurations and Intermediate Results on the CIFAR10+ Dataset

The authors tune CiC configurations on CIFAR10+ by varying kernel length, channel count, and the blocks using sparse MLP-010. These experiments select parameters and motivate the CiC-3D configuration used thereafter.

Parameter selection

Kernel length

Channel count

Block configuration

  • Applying MLP-010 in more blocks is beneficial, and CiC-3D is configured with MLP-010 in all three blocks.The selected sparse-layer configuration uses 224 input channels and kernel length 3.Source: B. Configurations and Intermediate… · p. 7

CiC variant

C. Comparison with Other Methods on the CIFAR10 Dataset

On CIFAR10, the experiments compare CiC-3D with NiN and other methods using the original dataset. CiC-3D achieves lower test error than the cited baselines and converges faster than NiN.

Training behavior

Test performance

D. Comparison with Other Methods on the CIFAR10++ Dataset

On the larger CIFAR10++ dataset, CiC-3D has lower test error than NiN, and the reported advantage increases as the training set grows.

Test performance

Scaling with training data

E. Comparison with Other Methods on the CIFAR100 Dataset

On CIFAR100, CiC-3D records the lowest test error among the compared methods and improves over both NiN and NiN-LA.

Test performance

Baseline comparisons

V. CONCLUSION AND FUTURE WORK

CiC uses sparse shallow MLPs for convolution, including partial variants in experiments and one-dimensional and three-dimensional forms. The authors identify broader sparse-MLP implementations and integration with other CNNs as future directions.

  • CiC uses sparse shallow MLPs for convolution, including fully sparse and several partial variants.Source: §V · p. 8
  • The experiments adopt the partial sparse MLP variant MLP-010.Source: §V · p. 8
  • CiC-1D connects channels with a one-dimensional filter, while CiC-3D extends filtering across the channel-spatial domain.Source: §V · p. 8
  • Future work includes implementing full and other sparse MLP types and integrating the idea into other state-of-the-art CNNs.Source: §V · p. 8

Related papers

Loading 1603.06759v1…