PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Convolutional Neural Networks

Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, Stella X. Yu

arXiv:1911.12207v3cs.CV

TL;DR

CNNs face training instability and feature redundancy, while kernel orthogonality does not fully ensure orthogonal convolutions. OCNN instead regularizes the doubly block-Toeplitz convolutional operator, achieving broad performance gains with little overhead and no extra parameters. It also learns more diverse, expressive features with better stability, robustness, and generalization.

  • Problem

    CNNs suffer from training instability and redundant features, and existing kernel-orthogonality methods do not fully enforce orthogonal convolutions.

  • Method

    OCNN represents convolution using a doubly block-Toeplitz matrix and directly regularizes the convolutional operator toward orthogonality.

  • Results

    OCNN consistently outperforms alternative orthogonal regularizers across diverse tasks and settings without additional parameters and with little computational overhead.

  • Takeaways & Limitations

    The method produces more diverse and expressive features alongside better training stability, robustness, and generalization.

Abstract

from arXiv · show

Deep convolutional neural networks are hindered by training instability and feature redundancy towards further performance improvement. A promising solution is to impose orthogonality on convolutional filters. We develop an efficient approach to impose filter orthogonality on a convolutional layer based on the doubly block-Toeplitz matrix representation of the convolutional kernel instead of using the common kernel orthogonality approach, which we show is only necessary but not sufficient for ensuring orthogonal convolutions. Our proposed orthogonal convolution requires no additional parameters and little computational overhead. This method consistently outperforms the kernel orthogonality alternative on a wide range of tasks such as image classification and inpainting under supervised, semi-supervised and unsupervised settings. Further, it learns more diverse and expressive features with better training stability, robustness, and generalization. Our code is publicly available at https://github.com/samaonline/Orthogonal-Convolutional-Neural-Networks.

1. Introduction

The paper identifies redundant, correlated filters and nonuniform convolutional spectra as sources of instability, then proposes OCNN to enforce orthogonality directly on the convolutional operator. Across varied tasks and settings, OCNN improves feature quality and performance with little overhead.

  • Motivation: Deeper CNN layers develop correlated, redundant filters and long-tailed operator spectra, contributing to exploding or vanishing gradients.These observations motivate orthogonality as a way to stabilize training and improve capacity utilization.
  • Motivation: OCNN regularizes convolutional layers during training so filters become decorrelated, responses become less redundant, and features become more expressive.The paper links these changes to better utilization of model capacity and improved task performance.
  • Method: The proposed method represents the convolution kernel as a doubly block-Toeplitz matrix and directly regularizes the input-to-output transformation.This formulation keeps the input and output intact while enforcing orthogonality on the operator connecting them.
  • Limitations of Existing Methods: Kernel orthogonality is necessary but not sufficient for orthogonal convolutions because the resulting convolutional spectrum can remain nonuniform.The paper therefore distinguishes orthogonality of the kernel matrix from orthogonality of the full convolutional operator.
  • Method: Efficient algorithms implement orthogonal convolutions without additional parameters and with little computational overhead.The paper presents this as an implementation contribution for CNNs.
  • Results: OCNN consistently outperforms other orthogonal regularizers across classification, generation, retrieval, and inpainting under supervised, semi-supervised, and unsupervised settings.The reported gains are associated with more diverse features, more uniform spectra, improved stability, robustness, and generalization.

2. Related Works

Prior CNN work uses im2col and kernel orthogonality to improve computation, stability, optimization, and performance, while related studies also address redundancy and training challenges.

  • Im2col-Based Convolutions: Im2col transforms convolution into a GEMM problem by unrolling input patches into columns and multiplying them by a reshaped kernel matrix.The output is reshaped back into the desired convolution tensor.
  • Kernel Orthogonality: Kernel orthogonality penalizes deviation of the kernel Gram matrix from identity, using row or column regularizers depending on matrix dimensions.Row orthogonality applies when M ≤ Ck^2; otherwise column orthogonality may be used.
  • Kernel Orthogonality: Orthogonal kernels have been studied for stabilizing RNN and CNN training, improving optimization, initialization, and norm preservation, though orthogonality may not persist during training.Stiefel-manifold optimization is one approach for maintaining orthogonality throughout training.
  • Kernel Orthogonality: Related CNN methods relax exact orthogonality through Frobenius-norm Gram penalties, mutual coherence, restricted isometry, or off-diagonal penalties, including applications to GANs.These approaches are reported to improve performance in image generation and other settings.
  • Kernel Orthogonality: Prior convolutional studies generally adopt kernel orthogonality, while DBT matrices had been used to analyze convolutional singular values without imposing orthogonality.This distinction motivates treating convolution itself as a structured linear operator.
  • Other Stabilization Methods: Other related work targets CNN feature redundancy, unstable gradients, and covariate shifts through diversity methods, normalization, reparameterization, and gradient clipping.These methods address different aspects of CNN efficiency and training stability.
  • DBT Representation: A DBT representation expresses convolution by flattening the input, converting the kernel tensor into a structured matrix, multiplying, and reshaping the output.The illustrated example uses a C × 4 × 4 input, an M × C × 2 × 2 kernel, and stride 1.

3. Orthogonal Convolution

The paper treats convolution as multiplication by a doubly block-Toeplitz matrix and regularizes that operator directly, using efficient local conditions for row and column orthogonality.

  • 3. Orthogonal Convolution: Convolutional orthogonality views convolution as matrix-vector multiplication and regularizes the resulting DBT operator to stabilize its spectrum.The method focuses on 2D convolution, with concepts and conditions generalizing to other cases.
  • 3.1. Convolution as a Matrix-Vector Multiplication: The DBT matrix is constructed from the kernel tensor after flattening the input; multiplying produces a vector that is reshaped into the output tensor.Each DBT row corresponds to a filter at a particular spatial location.
  • 3.2. Convolutional Orthogonality: For fat operators, uniform spectrum requires row orthogonality; for tall operators, it requires column orthogonality, with the latter preserving norm as a normalized frame.The matrix is fat when MH′W′ ≤ CHW and tall when MH′W′ > CHW.
  • 3.2. Convolutional Orthogonality: Brute-force DBT orthogonality is avoided because the matrix is structured and sparse, enabling equivalent conditions that are efficiently computable.The paper develops fast constraints for both row and column orthogonality.
  • Row Orthogonality: Row orthogonality only requires checking filter pairs whose spatial patches overlap, because nonoverlapping flattened vectors have zero inner product.The relevant region depends on kernel size and stride and can be realized using padding.
  • Row Orthogonality: Minimizing the difference between the self-convolution Z and the identity-centered tensor Ir0 yields a near row-orthogonal convolution in the DBT sense.The identity occupies the center M × M entries, while the remaining entries are zero.
  • Comparison to Kernel Orthogonality: Kernel orthogonality conditions are necessary but generally insufficient for orthogonal convolutions, becoming equivalent only when convolution stride equals kernel size.The comparison distinguishes kernel-level constraints from the stronger convolutional conditions.
  • Orthogonal Regularization in CNNs: A soft orthogonal-convolution loss is added to the task loss so optimization can pursue both task performance and operator orthogonality.The regularization weight is denoted λ > 0.

4. Experiments

Experiments evaluate OCNN across supervised, semi-supervised, and unsupervised tasks, including classification, retrieval, and inpainting. OCNN consistently improves over plain, kernel-orthogonal, and prior methods.

  • Classification on CIFAR100: OCNN achieves 78.1%, 78.7%, and 79.5% CIFAR100 accuracy with ResNet18, ResNet34, and ResNet50, respectively.These results outperform plain baselines by 3%, 2%, and 1%, and kernel-orthogonal regularizers by 2%, 1%, and 1%.
  • Classification on CIFAR100: OCNNs gain 3% and 1% over plain baselines and kernel-orthogonal regularizers with WideResNet28 on CIFAR100.WideResNet uses column orthogonality because some convolutional DBT matrices are tall.
  • Classification on ImageNet: OCNN outperforms plain and other orthogonal regularizations by 1% on ImageNet Top-1 and Top-5 error comparisons with ResNet34.The method achieves 25.87% top-5 error and 7.89% top-1 error without modifying the backbone.
  • Semi-supervised learning: OCNN constantly outperforms the baseline by 2%–3% when CIFAR100 labeled-data fractions range from 10% to 80%.The comparison varies the proportion of labeled data under semi-supervised learning.

4.6. Image Generation

The analysis studies OCNN in image generation, adversarial robustness, feature diversity, hyper-parameter sensitivity, and computational cost. OCNN improves GAN metrics and attack resistance while adding no parameters and modest training overhead.

  • Image Generation: OCNN achieves an inception score of 8.63 ± 0.007 and FID of 11.75 ± 0.04, outperforming the baseline and reaching state-of-the-art performance.The model is evaluated five times with 50k images each, and convergence is faster with OCNN regularization.
  • Adversarial robustness: OCNN requires approximately 7x attack time and 1.7x attack queries compared with the baseline, while outperforming it by 2% after equal iterations.The robustness experiment uses a black-box pixel attack on correctly classified CIFAR100 test images.
  • Feature analysis: OCNN shifts filter-similarity distributions leftward and broadens them, indicating greater filter diversity and reduced feature redundancy.The analysis compares plain ResNet34 with different regularizers and examines guided back-propagation patterns.
  • Hyper-Parameter Analysis: CIFAR100 accuracy is highest at regularization weight λ = 0.1 with a ResNet18 backbone.The hyper-parameter sweep evaluates λ from 0.05 to 1.0.
  • Space and Time Complexity: The regularizer adds no parameters or test-time cost; training time increases 3% for kernel orthogonality and 9% for conv-orthogonality.The complexity comparison uses ResNet34 on ImageNet with a GTX 1080 Ti and batch size 256.

5. Summary

OCNN imposes convolutional orthogonality through the doubly block-Toeplitz representation rather than common kernel orthogonality. It consistently improves performance and feature quality across diverse tasks with little overhead.

  • Summary: OCNN uses the doubly block-Toeplitz matrix representation to impose filter orthogonality, whereas kernel orthogonality is necessary but not sufficient for orthogonal convolutions.The approach requires no additional parameters and little computational overhead.
  • Summary: OCNN consistently outperforms state-of-the-art alternatives across classification and inpainting under supervised, semi-supervised, and unsupervised settings.It also learns more diverse and expressive features with better training stability, robustness, and generalization.

A. Intuitive Explanations of our Approach

CNN convolution spectra can be highly non-uniform, while OCNN enforces orthogonality on the DBT operator to make spectra more uniform. This reduces feature redundancy and is associated with gains across visual tasks and adversarial robustness.

  • Problem intuition: Non-uniform convolution spectra can scale different inputs unevenly and potentially cause exploding or vanishing gradients in deep networks.The spectrum reflects the scaling property of the convolutional layer and the scaling of gradients.
  • Approach intuition: OCNN enforces orthogonality on the DBT kernel matrix, producing more uniform spectra and reducing feature redundancy.The paper connects uniform spectra with improved feature quality and performance.
  • Observed consequences: OCNN improves visual feature quality in image retrieval and inpainting, generates realistic images, and is more robust to adversarial attacks.These observations extend beyond classification performance.

B. Network Dissection

Network dissection evaluates feature expressiveness through semantic alignment of channels. Compared with a plain ResNet34 baseline, OCNNs produce more unique detectors and more uniform concept distributions, implying reduced redundancy and enhanced expressiveness.

  • Evaluation method: Network dissection measures channel interpretability by comparing activation regions with annotated semantic concepts using mIoU.Channels with mIoU≥0.04 are treated as effective and called unique detectors.
  • Results: OCNNs have more unique detectors across network layers than the plain baseline ResNet34.
  • Results: OCNNs show more uniform distributions across six concept categories than the plain baseline ResNet34.The categories are scene, object, part, material, texture, and color.
  • Interpretation: These detector and concept-distribution patterns imply that orthogonal convolutions reduce feature redundancy and enhance feature expressiveness.

C. Deep Metric Learning

The orthogonal regularizer was evaluated for retrieval and clustering on Cars196 under established deep metric learning settings. The authors report performance gains when the regularizer is added.

  • Evaluation: The study evaluates the orthogonal regularizer on Cars196 using retrieval and clustering tasks.Training and evaluation follow the settings of prior work [41].
  • Results: Performance gains are observed when the orthogonal regularizer is added.

D. Proof of the Orthogonality Equivalence

The proof establishes an MSE-equivalence between row and column orthogonality for an arbitrary matrix, with a constant offset determined by its dimensions. This supports using a convenient orthogonality formulation.

  • Equivalence: Row and column orthogonality are equivalent in the MSE sense.
  • Equivalence: The two formulations differ by an additive constant U, where U = M−N.
  • Motivation: For square matrices, KKT = I if and only if KTK = I′, motivating generalization to rectangular matrices.
  • Setup: For an arbitrary K ∈ R^M×N, the proof defines row loss Lr as ∥KKT−IM∥2 and column loss Lc as ∥KTK−IN∥2.

E. Filter Similarity Visualizations

Guided back-propagation visualizations compare channel patterns in early and late ResNet34 layers. Earlier-layer channels are more diverse, whereas later-layer channels focus on particular regions, consistent with increasing filter similarity with depth.

  • Visualization setup: Figure 12 visualizes guided back-propagation patterns for an ImageNet-trained ResNet34.
  • Visualization setup: The first row shows the first three channels from layer 7, while the second row shows the first three channels from layer 33.
  • Observed patterns: Channels in earlier layers have more diverse patterns than channels in later layers.
  • Observed patterns: Later-layer patterns usually focus on certain regions, and filter similarity increases with network depth.
Loading 1911.12207v3…