PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Connections Encourage Iterative Inference

Stanisław Jastrzębski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, Yoshua Bengio

arXiv:1710.04773v2cs.CV

TL;DR

Deep compositional networks are difficult to train, and the paper investigates whether Resnets instead perform iterative feature refinement. Through analytical and empirical study, it finds that residual blocks approximate negative-gradient updates, with early layers learning representations and higher layers refining them, while naive sharing can cause instability and overfitting.

  • Problem

    Training compositional networks beyond roughly 15–20 layers remains challenging, motivating a deeper understanding of how Resnets process features.

  • Method

    The paper combines Taylor-expansion analysis with empirical measurements across Resnet architectures to study residual-block updates, representation behavior, and block sharing.

  • Results

    Residual blocks naturally move representations toward the negative loss gradient; lower blocks learn representations while higher blocks perform iterative refinement.

  • Takeaways & Limitations

    Resnets combine hierarchical representation learning with iterative inference, but naive residual-layer sharing can cause representation explosion and overfitting.

  • Takeaways & Limitations

    Naively shared residual blocks perform poorly, and the causes and mitigation of this behavior require further investigation.

Abstract

from arXiv · show

Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively, overfitting, and we show that simple existing strategies can help alleviating this problem.

1 INTRODUCTION

Resnets address the difficulty of training very deep compositional networks by using additive residual blocks. This paper formalizes and empirically studies their iterative-refinement behavior, while distinguishing it from representation learning and examining shared-block training.

  • Traditional compositional networks deeper than 15–20 layers remain difficult to train because of vanishing and exploding gradients.ReLU activations, improved initialization, and normalization were proposed to address this problem.
  • Residual blocks additively transform representations, helping Resnets learn very deep networks and maintain performance when blocks are removed or shuffled.
  • Residual connections naturally encourage hidden representations to move along the negative loss gradient, approximating gradient steps in activation space.The paper derives this analytically for any loss function and empirically tests alignment with the loss gradient.
  • Lower residual blocks primarily learn representations, whereas higher blocks tend to refine existing representations by moving them along the negative gradient direction.With shortcut connections, representation learning is dominated by the shortcut layer and most residual blocks perform iterative refinement.
  • Naively sharing residual blocks produces poor performance, motivating analysis of the failure and investigation of a preliminary fix.The paper studies sharing blocks during training and unrolling blocks beyond the number of steps used during training.

2 BACKGROUND AND RELATED WORK

Prior work interprets Resnets as ensembles of shallow paths or approximations to recurrent networks, while this paper emphasizes a precise gradient-based definition of iterative inference. The related perspective also connects iterative refinement with recursive processing and weight sharing.

  • Residual Networks and their analysis: Some prior analyses view Resnets as ensembles of relatively shallow networks because their unraveled structure contains exponentially many input-to-output paths.The relative insensitivity to dropping or shuffling residual blocks supports this interpretation.
  • Residual Networks and their analysis: Other work suggests that residual networks approximate recurrent networks, providing an alternative interpretation of their computation.
  • Residual Networks and their analysis: This paper distinguishes its approach by defining iterative inference as a residual block approximating a gradient-descent step in activation space.The authors relate this view to gradient boosting because both use outputs aligned with the negative loss gradient.
  • Iterative refinement and weight sharing: Iterative refinement can be modeled through recursive application of lateral connections, analogous to shared weights in a recurrent model.This perspective motivates parameter sharing for iterative inference in deep networks.
  • Iterative refinement and weight sharing: A shortcut connection is a convolution layer between residual blocks that changes the hidden-space dimension.

3 ITERATIVE INFERENCE IN RESNETS

The paper formalizes iterative inference in Resnets by analyzing stacked additive residual blocks and Taylor-expanding the loss. This analysis shows that residual updates are encouraged to move representations toward lower loss, resembling gradient steps.

  • Architecture: Resnet architectures stack residual blocks after an initial convolutional layer, with optional shortcut connections between blocks.The analysis focuses on how additive residual transformations affect hidden representations across the network.
  • Residual transformation: A residual block transforms a representation by adding its learned residual update to the incoming representation.This additive structure is the basis for analyzing iterative inference in Resnets.
  • Loss analysis: Taylor expansion recursively expresses the final loss in terms of intermediate representations and residual updates across stacked blocks.The expansion is performed around successive hidden representations, from the final block backward toward an earlier block.
  • Loss analysis: When residual updates are small, first-order Taylor terms provide a good approximation; otherwise, higher-order terms also contribute.The higher-order terms are collected as O(.) terms in the expansion.
  • Iterative inference: Minimizing the loss approximately encourages residual updates to point opposite the loss gradient, so updated representations move toward the negative-gradient direction.The analysis describes this as minimizing the dot product between the residual update and the gradient of loss with respect to the hidden representation.
  • Iterative inference: The paper formalizes iterative inference as moving down the loss surface, noting that residual blocks resemble stochastic gradient descent.The interpretation is especially justified for higher layers, whose residual outputs have small magnitude.

4 EMPIRICAL ANALYSIS

Experiments on CIFAR-10 examine how residual blocks change representations, refine features, target borderline examples, and behave when unrolled or shared. Results support distinct lower-layer representation learning and higher-layer iterative refinement, while naive sharing causes instability and poor generalization.

  • Experimental setup: Experiments evaluate four Resnet architectures on CIFAR-10, including original, single-representation, average-pooling, and wide Resnets.The analysis uses different depth, pooling, width, and shortcut configurations.
  • Cosine loss of residual blocks: Higher residual blocks have consistently negative cosine loss, especially in deeper architectures, indicating stronger iterative refinement behavior.Their smaller representation changes make the first-order Taylor term dominant, supporting movement toward −∂L(h_i)/∂h_i.
  • Representation learning vs. feature refinement: The first few blocks can change representations by up to twice the original norm, whereas higher blocks make progressively smaller changes in single-representation and pooling Resnets.This pattern is less pronounced in architectures with shortcut convolutions, suggesting lower blocks compensate when compositional layers are limited.
  • Borderline examples: Higher-layer refinement primarily corrects borderline examples, while already-correct examples can experience worse loss after later blocks.The authors note that test loss increases after the last block despite uniformly decreasing training loss.
  • Unrolling residual networks: Unrolling the last residual block improves test performance on borderline examples from 43% to 53% across an average of 51 such examples.Training loss improves from 0.0012 to 0.001, while test loss increases; scaled unrolling maintains negative cosine loss.
  • Sharing residual layers: Naively sharing higher residual blocks causes overfitting, underfitting, and worse validation performance, while repeated operations make activations and gradients explode at initialization.Unshared Batch Normalization mitigates the activation-explosion problem.

5 CONCLUSION

The paper formalizes iterative refinement in Resnets as movement toward the negative loss gradient in activation space and validates this view experimentally. It also finds that naïve residual-block sharing causes overfitting, while a batch-normalization variant mitigates the associated activation explosion.

  • Residual blocks naturally encourage representations to move in the half space of the negative loss gradient, implementing gradient descent in activation space.The paper reports that each block reduces loss and improves accuracy.
  • The iterative-refinement theory is validated experimentally across a wide range of Resnet architectures.
  • Naïvely sharing residual blocks during training leads to overfitting and representation explosion.The paper studies both training-time sharing and unrolling a block beyond its training horizon.
  • A variant of batch normalization helps mitigate the problems caused by naïve residual-block sharing, while further investigation remains future work.

A FURTHER ANALYSIS

The analysis connects movement of a convolutional layer’s output in the negative loss-gradient direction with a gradient update to that layer’s parameters. The authors note this equivalence but do not develop it further in the paper.

  • Moving the convolution output in the negative loss-gradient half space is equivalent to updating its parameters with gradient descent.The output is defined as ho = Wx + b, and the parameter update uses step size η.
  • The authors identify this equivalence as an interesting insight but leave it for future work.

B ANALYSIS ON CIFAR-100

The CIFAR-100 appendix reports experiments corresponding to the paper’s CIFAR-10 analyses. It presents the results in Figures 10, 11, and 12 and reports the same conclusions as the main text.

  • CIFAR-100 experiments reproduce the analyses from Sections 4.2 and 4.1 in Figures 10, 11, and 12.
  • The CIFAR-100 experiments yield the same conclusions as those reported for CIFAR-10.

C ANALYSIS OF INTERMEDIATE METRICS ON CIFAR-10 AND CIFAR-100

Intermediate-block metrics show gradually increasing accuracy across higher residual blocks, consistently negative cosine loss, and increasing ℓ2 ratios during training. These measurements are reported for selected blocks in CIFAR-10 and CIFAR-100 experiments.

  • The metrics are plotted for residual blocks in the last space of each architecture on CIFAR-10 and CIFAR-100.This restriction keeps residual-block outputs dimensionally compatible with the classifier.
  • Accuracy after individual residual blocks increases gradually from lower to higher residual blocks.
  • Cosine loss remains consistently negative across architectures after individual residual blocks.
  • The ℓ2 ratio tends to increase for residual blocks as training progresses.

D ITERATIVE INFERENCE IN SHARED RESNET

The section reports measurements of cosine loss, residual-to-input ℓ2 ratios, intermediate accuracy, and block dropping across several Resnet variants. It also examines naive block sharing, unshared batch normalization, and extending a trained block beyond its original unrolling depth.

  • Figures 11, 14, and 16 track the residual-block output-to-input ℓ2 norm ratio across Resnet variants on CIFAR-100 and CIFAR-10.The plots distinguish lower and higher residual blocks where indicated.
  • Figure 12 measures final prediction accuracy after individually dropping residual blocks in four CIFAR-100 Resnet variants.The compared models are the original, single-representation, average-pooling, and wide Resnets.
  • Figure 17 measures intermediate prediction accuracy by placing a classifier after hidden states in the last stage of four Resnet variants during CIFAR-100 training.Curves are ordered from lower to higher residual blocks.
  • Figures 18 and 19 compare cosine loss, ℓ2 ratios, and intermediate accuracy for shared Resnet-110 with unshared versus naively shared batch normalization.Each curve represents a different block, with red denoting the block closest to the output.
  • Figure 20 tests whether a Resnet-110 block trained for a fixed depth generalizes when unrolled for additional steps and examines scaled final-block behavior.The plots show cosine loss and ℓ2-ratio evolution, with a vertical line marking the number of training steps.

E UNROLLING RESIDUAL NETWORKS

This section reports additional results for unrolling residual networks. Figure 20 tracks cosine loss and the ℓ2 ratio for Resnet-110 when its last block is unrolled for 20 additional steps.

  • Figure 20 reports cosine-loss and ℓ2-ratio evolution for Resnet-110 with its last residual block unrolled for 20 additional steps.The figure evaluates unrolling beyond the originally trained depth.
Loading 1710.04773v2…