PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee

arXiv:1603.05201v2cs.LGcs.CV

TL;DR

CNNs achieve strong empirical results, but their intrinsic properties remain insufficiently understood. The paper identifies opposite-phase filter pairs, introduces CReLU to preserve both phases, and reports improved recognition with fewer parameters.

  • Problem

    Despite CNNs’ empirical success, their intrinsic properties remain insufficiently understood.

  • Method

    The paper examines CNN filter pairing, proposes CReLU to preserve positive and negative phase information, and theoretically analyzes its reconstruction property.

  • Results

    CReLU improves recognition performance on CIFAR-10/100 and ImageNet and enables notable parameter reduction without sacrificing classification performance when applied appropriately.

  • Takeaways & Limitations

    Better understanding of CNN properties can realize higher potential through a simple architecture modification.

  • Takeaways & Limitations

    Reconstruction analysis with max-pooling requires additional input-space constraints, which are explained in the supplementary materials.

Abstract

from arXiv · show

Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

1. Introduction

The paper studies CNNs to address limited fundamental understanding and finds paired, opposite-phase filters in lower layers. It proposes CReLU as a simple modification that improves recognition and can reduce parameters.

  • Despite CNNs’ empirical success, their intrinsic properties remain insufficiently understood.
  • AlexNet’s first convolution filters form highly negatively correlated pairs representing opposite phases.
  • CReLU preserves positive and negative phase information while maintaining non-saturated nonlinearity.
  • Replacing ReLU with CReLU in lower convolution layers substantially improves classification on CIFAR-10/100 and ImageNet.
  • CReLU can reduce trainable parameters without sacrificing classification performance when applied appropriately.

2. CRelu and Reconstruction Property

The paper interprets negatively correlated lower-layer filters as redundancy caused by ReLU’s loss of negative responses and introduces CReLU to preserve both phases. It analyzes reconstruction and relates the design to prior activation and scattering approaches.

  • Pairing phenomenon: AlexNet’s lower convolution filters increasingly form opposite-phase pairs, while the pairing phenomenon weakens in deeper layers.
  • Pairing phenomenon: The authors hypothesize that ReLU models recover both positive and negative phase information through redundant negatively correlated filter pairs.
  • CReLU: CReLU copies convolution responses, negates the copies, concatenates both parts, and applies ReLU to preserve both activation directions.
  • Reconstruction property: CReLU preserves positive and negative responses, making the reconstruction power of convolution layers more straightforward to characterize.
  • Reconstruction property: Without max-pooling, the analysis shows that the input component spanned by filter shifts is preserved and reconstructible.
  • Reconstruction property: With max-pooling, obtaining a non-trivial reconstruction bound requires additional constraints on the input space.

3. Benchmark Results

Across CIFAR-10, CIFAR-100, deeper VGG-style networks, and ImageNet, replacing ReLU with CReLU improves recognition while enabling parameter reductions in suitable configurations.

  • CIFAR-10 and CIFAR-100: CReLU consistently improves CIFAR recognition over ReLU with the same number of activations, while some configurations halve the parameter count.On CIFAR-10, the CReLU half model has 0.7M parameters versus 1.4M for the baseline while maintaining the same number of neurons.
  • CIFAR-10 and CIFAR-100: AVR can outperform ReLU on CIFAR-100 and single-model CIFAR-10 evaluation, but becomes clearly inferior to CReLU under averaged or voted 10-fold evaluation.This comparison spans all reported evaluation metrics for CIFAR-100 and single-model evaluation on CIFAR-10.
  • CIFAR-10 and CIFAR-100: CReLU produces substantial gains in deeper VGG networks when applied after selected convolution layers while halving the number of filters.The evaluated placements are after conv1, conv3, and conv5.
  • ImageNet: On ImageNet, integrating CReLU through conv1–4 achieves the best results, while applying it deeper does not further improve classification performance.The authors relate this to weaker filter pairing in deeper convolutional layers.
  • ImageNet: CReLU removes the lower-layer filter-pairing phenomenon on ImageNet, with the two histogram distributions aligning across conv1–conv4.The reported Figure 3 analysis concerns the best CReLU model using those four layers.
  • ImageNet: CReLU also supports parameter-efficient ImageNet models: the 4.6M-parameter CReLU-all model outperforms FastFood-32-AD and Pruned AlexNet.The comparison is reported despite parameter reduction not being the primary goal.

4. Discussion

The discussion examines CReLU through regularization, invariant representations, filter organization, and reconstruction, linking these properties to performance and parameter efficiency.

  • Regularization: CReLU models show less overfitting than ReLU models in all-convolution CIFAR experiments despite having twice as many parameters.The authors contemplate that retaining both phase directions makes training more challenging and may improve regularization on small datasets.
  • Regularization: The CReLU-plus-linear complexity bound equals the corresponding ReLU-plus-linear bound, so doubling parameters does not necessarily increase model complexity.The bound is presented through a Rademacher complexity analysis.
  • Invariant representation learning: CReLU models consistently achieve higher invariance scores than ReLU models, although scores generally increase non-monotonically with network depth.Max-pooling produces pronounced score increases in some CIFAR configurations.
  • Invariant representation learning: ImageNet invariance maxima at conv1, conv4, and conv7 motivate placing CReLU after those layers while halving filters to limit model capacity.This architecture achieves the best top-1 and top-5 recognition results when averaging scores from 10 patches.
  • Reconstruction property: CReLU removes the negatively correlated filter-pairing pattern: lower-layer filters span distinct directions while CReLU implicitly performs pair-grouping.The CReLU filter distributions align with those of random Gaussian filters.
  • Reconstruction property: CReLU retains distinct treatment of positive and negative phases, unlike AVR, because outgoing-weight correlations for both paired and unmatched components remain far below 1.The empirical gap between CReLU and AVR supports the importance of both modulus and phase information.
  • Reconstruction property: CReLU reconstruction fully recovers the input component spanned by filter shifts, while direct reconstruction after max-pooling requires additional analysis.For convolution-plus-CReLU-plus-max-pooling, reconstruction power is evaluated indirectly through convolution-filter properties.

A. Reconstruction Property Proofs

The proof reconstructs the component of an input lying in the convolution-filter span by inverting CReLU and applying a pseudoinverse.

  • Proposition A.1: The input is decomposed into a component x′ in range(W) and a residual component x − x′ in ker(W).The proposition targets reconstruction of x′ rather than the full input.
  • Reconstruction algorithm: The reconstruction algorithm first applies the inverse CReLU function to the network output to recover the pre-activation representation.It denotes this intermediate vector as z = ρ^-1_c(fcnn(x)).
  • Reconstruction algorithm: It then applies the Moore–Penrose pseudoinverse of W^T, yielding x′ through x′ = (W^T)+z.The associated projector Q = (W^T)+W^T is the orthogonal projector onto range(W).

A.2. Max-Pooling Case

This section models convolution followed by CReLU and max-pooling, then establishes conditions under which the input can be reconstructed with bounded error. The analysis uses frame-theoretic operators and pseudoinverse-based reconstruction.

  • Layer formulation: The layer applies convolution, separates positive and negative phases through CReLU, and selects maxima within each filter block.The resulting mapping is fcnn: R^D → R^2K.
  • Assumption: The reconstruction analysis assumes each input is a linear combination of shifted convolution filters, with at most one participating shift per max-pooling region.This constraint makes it possible to derive a non-trivial bound when max-pooling is present.
  • Frame-theoretic setup: A frame provides lower and upper bounds on coefficient energy, while its analysis, synthesis, and frame operators connect convolutional representations to reconstruction.The frame bounds correspond to the smallest and largest eigenvalues of the frame operator.
  • Frame-theoretic setup: The convolution filters and their shifts form the matrix representation used to analyze the layer through singular values and operator norms.The analysis operator is represented by W^T U, and the frame operator by T̃* T̃.
  • Reconstruction theorem: Theorem A.5 states that, under the input assumption, fcnn(x) yields a reconstruction x′ with an error controlled by the singular values of the active filter matrices.The proof reverses CReLU, uses a Moore–Penrose pseudoinverse, and compares the resulting projection with a reconstruction from selected active filters.
  • Reconstruction theorem: Because the selected-filter range is contained in the full convolutional range, the layer’s reconstruction is at least as accurate as the corresponding restricted projection.The proof then uses orthogonal-projection geometry to relate reconstruction error to the input and projected components.

B. Proof of Model Complexity Bound

This section bounds the complexity of CReLU followed by a linear transformation using Rademacher-complexity tools. The resulting bound matches the corresponding ReLU-plus-linear bound despite CReLU doubling the parameter count.

  • Complexity tools: Rademacher complexity measures the expected ability of a function class to correlate with independent random signs over a sample.The section introduces empirical Rademacher complexity before applying composition and network-layer bounds.
  • Complexity tools: The composition lemma transfers a Lipschitz constant through function composition, while the network-layer proposition bounds a linear transform using input dimension and operator norm.The linear-layer bound scales as √d_in B times the preceding class complexity.
  • ReLU comparison: Because ReLU is 1-Lipschitz, ReLU followed by a linear transformation inherits the preceding class bound up to the linear-layer factor.This provides the comparison bound used for CReLU.

C. Reconstruction Ratio

This section evaluates how well pooled CReLU features preserve information by reconstructing inputs from intermediate representations. Learned filters recover substantially more of the input domain than random filters in the reported comparison.

  • Experimental setup: The experiment studies all-convolutional CReLU-half models on CIFAR-10/100, where max-pooling follows conv2 and conv5.The analysis uses the reconstruction theorem as motivation for evaluating pooled intermediate features.
  • Experimental setup: Reconstruction ratio is computed as ∥x − x′∥2/∥x∥2 after reconstructing sampled test images from pooled features via Algorithm 2.The same procedure is applied to learned and random convolution filters.
  • Results: Random filters recover only 1% of the original input, whereas learned filters span more of the input domain.The comparison uses conv2 and conv5 settings with D = 48(96) × 5 × 5 and K = 48(96).
  • Random-filter baseline: The random-filter baseline is analyzed as projection onto a random subspace, with each random-filter entry sampled from a standard normal distribution.Theoretical analysis relates expected reconstruction to the dimension of that random subspace.

D. Invariance Score

This section defines an invariance-score procedure for testing whether convolutional filters respond consistently to transformed versions of images. Scores are averaged across filters within each convolution layer.

  • Evaluation protocol: Invariance scores are computed on all 50k CIFAR-10/100 test images and center crops from 5k randomly sampled ImageNet validation images.The evaluation uses different image subsets for the two dataset groups.
  • Evaluation protocol: Each filter’s firing threshold is chosen so it activates one percent of the time, using negative-response masking for ReLU and absolute responses for CReLU.This sets the global firing rate to 0.01.
  • Stimulus transformations: Semantically similar stimuli are generated through horizontal flips, 15-degree rotations, and translations, with translation ranges differing between CIFAR and ImageNet.CIFAR uses three-pixel horizontal and vertical shifts, while ImageNet uses four-corner crops.
  • Scoring rule: A convolutional filter counts as fired only when the original and transformed images activate it at the same spatial location.This location-matching rule adapts invariance measurement to convolutional feature maps.
  • Scoring rule: The final invariance score for each convolution layer is the average of the individual filter scores at that layer.The procedure therefore produces one aggregate score per convolution layer.

E. Implementation Details on ImageNet Models

The ImageNet networks use different optimization setups depending on whether convolution filters after CReLU are kept unchanged or reduced by half.

  • Networks in Tables S8–S11 keep the number of post-CReLU convolution filters unchanged and use SGD with mini-batches of 64 and momentum 0.9.The learning rate and weight decay follow a seven-stage schedule across epochs 1–45.
  • Networks in Tables S12–S13 halve the number of post-CReLU convolution filters and use Adam with initial learning rate 0.0002, mini-batches of 64, and 100 epochs.

F. Details of Network Architecture

The supplementary architecture details organize CIFAR-10/100 and ImageNet models by baseline, CReLU, AVR, and the convolutional layers where these variants are applied.

  • CIFAR-10/100: Table S2 identifies the baseline or AVR models and baseline (double) models used for the CIFAR-10/100 experiment, with “avg” denoting average pooling.
  • CIFAR-10/100: Table S3 identifies the CReLU and CReLU (half) models used for the CIFAR-10/100 experiment.

G. Image Reconstruction

This section provides additional image reconstruction examples.

  • Additional image reconstruction examples are provided in this section.
Loading 1603.05201v2…