Source-linked AI summary
DiracNets: Training Very Deep Neural Networks Without Skip-Connections
Sergey Zagoruyko, Nikos Komodakis
TL;DR
Prior analyses question whether ResNet's original depth motivation holds, as widening has been reported more effective than deepening. The paper introduces Dirac parameterization for plain deep networks and reports strong comparisons with residual networks on CIFAR and ImageNet.
Problem
Prior analyses question whether ResNet's original depth motivation holds, with widening reported more effective than deepening beyond 50 layers.
Source: §1 · p. 1Method
The paper introduces generic Dirac weight parameterization and uses it in plain VGG- and ResNet-like networks that train end-to-end at hundreds of layers.
Source: §1 · p. 2Results
A 28-layer DiracNet surpasses 1001-layer ResNet and approaches Wide ResNet accuracy; on ImageNet, DiracNets almost match ResNet with the same number of parameters.
Source: §1 · p. 2, §3.5 · p. 6Takeaways & Limitations
Dirac parameterization can also be combined with explicit ResNet skip-connections, eliminating the need for careful initialization.
Source: §1 · p. 2On smaller CIFAR datasets, DiracNets significantly trail ResNet; the authors link this to weaker regularization from weight normalization and free scaling parameters.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution-ReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with 28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at https://github.com/szagoruyko/diracnets
1 INTRODUCTION
The introduction questions whether skip-connections deliver their original promise of training deeper networks and motivates plain networks that can train at great depth. It presents Dirac parameterization and DiracNets as a route to that goal, with reported results competitive with residual networks.
1 INTRODUCTION
- Prior analyses question whether depth itself benefits ResNets: widening is reported more effective than deepening, with no benefit beyond 50 layers.The introduction also lists feature reuse and uncertainty about effective depth as problems of skip-connected networks.Source: §1 · pp. 1–2
- 28-layer DiracNets surpass 1001-layer ResNet and approach Wide ResNet accuracy.The authors report training these plain networks end-to-end, without layer-wise pretraining.Source: §1 · p. 2
- The paper introduces Dirac parameterization and plain VGG- and ResNet-like DiracNets to study whether very deep networks can train without explicit skip-connections.The motivation is to explore possible gains in parameter efficiency, generalization, and computational efficiency.Source: §1 · p. 2
- Dirac parameterization is presented as broadly applicable, compatible with explicit ResNet skip-connections, and foldable into a VGG-like chain of convolution-ReLU pairs.The authors also report that plain DiracNets train end-to-end at hundreds of layers and generalize with massive parameter counts.Source: §1 · p. 2
2 DIRAC PARAMETERIZATION
Dirac parameterization adds an identity component to convolutional weights rather than an explicit skip connection, helping information propagate through deep networks. Its connection to ResNet is implicit, though the two differ in the order of nonlinearities.
2 DIRAC PARAMETERIZATION
- Dirac weights add an identity component instead of an explicit skip connection, helping information propagate and alleviating vanishing gradients.Source: §2 · p. 2
- Each filter has a learned scale for the Dirac term; when the scale exceeds 1 and W is small, the output approaches the input.Source: §2 · p. 3
- Weight normalization stabilizes training of very deep networks with more than 30 layers.Source: §2 · p. 3
- The parameterization adds only two scaling multipliers per channel during training, which can be folded into filters at test time.Source: §2 · p. 3
2.1 CONNECTION TO RESNET
- Dirac parameterization and ResNet differ in the order of nonlinearities, and each Dirac-parameterized layer has an unavoidable nonlinearity.Unlike ResNet's skip connection, the Dirac skip is implicit; the parameterization can be folded into one weight vector at inference.Source: §2.1 · p. 3
3 EXPERIMENTAL RESULTS
The experiments compare plain and Dirac-parameterized networks with ResNet and Wide ResNet on CIFAR and ImageNet, using the same training hyperparameters as Wide ResNet. They also examine scaling coefficients and test Dirac parameterization as a way to reduce ResNet’s initialization sensitivity.
3.1 PLAIN NETWORKS WITH DIRAC PARAMETERIZATION
- At 100 layers, plain networks do not converge and achieve 40% train/validation accuracy, whereas DiracNet achieves 92.4% validation accuracy; at 34 layers, they achieve 91.2% and 92.8%, respectively.At 52 layers, plain-network accuracy falls to 88% while DiracNet accuracy keeps growing.Source: §3.1 · p. 4, Figure 1 · p. 4
3.2 PLAIN DIRAC NETWORKS AND RESIDUAL NETWORKS
- 95.25% accuracy for DiracNet-28-10 with 36.5M parameters is close to WRN-28-10’s 96.0% with 36.5M parameters; 28-layer DiracNet also closely matches 1001-layer ResNet.Table 2 reports CIFAR performance with horizontal flips and crops; DiracNets outperform other plain networks and approach residual architectures.Source: §3.2 · p. 4, Table 2 · p. 5
3.3 ANALYSIS OF SCALING COEFFICIENTS
- Coefficient analysis found no layer with a degraded a-to-b ratio, indicating all layers remain involved in training.The analysis tracks average a and b values across DiracNet-34 layers during CIFAR-10 training.Source: §3.3 · p. 5, Figure 2 · p. 5
3.4 DIRAC PARAMETERIZATION FOR RESNET WEIGHT INITIALIZATION
- Dirac parameterization eliminates careful-initialization requirements for ResNet; very small σ values such as 10^-8 preserve accuracy, σ around 1 is slightly worse, and zero weights do not converge.Using the same orthogonal initialization as DiracNet gives an even wider range of scaling gains.Source: §3.4 · pp. 5–6
3.5 IMAGENET RESULTS
- On ILSVRC2012, DiracNet nearly matches ResNet accuracy with similar convergence curves and the same parameter count; plain VGG-like DiracNets match accuracy with 10 times fewer parameters.Figure 3 compares training and validation top-5 error using the same optimization hyperparameters.Source: §3.5 · p. 6, Figure 3 · p. 6
4 DISCUSSION
DiracNets closely match ResNets on ImageNet but lag on smaller CIFAR datasets; both architectures show little or no accuracy benefit beyond some depth. The authors also report that DiracNets generalize with massive parameter counts and hypothesize that optimization noise and stabilizing factors contribute.
4 DISCUSSION
- After a certain depth, both ResNets and DiracNets show little or no accuracy benefit, likely because deeper layers underuse parameters.The authors say both architectures are prone to this issue, but to different extents.Source: §4 · p. 6
- DiracNets closely match ResNet accuracy on ImageNet but fall significantly behind on smaller CIFAR datasets, which the authors attribute to weaker regularization.They point to weight normalization and free scaling parameters as making DiracNet less regularized than ResNet.Source: §4 · p. 6
- DiracNets generalize with massive parameter counts without negative effects of overfitting; the authors hypothesize this relates to high-learning-rate momentum noise and stabilizing factors.The proposed stabilizers include residual or Dirac parameterization and batch normalization.Source: §4 · p. 7