Source-linked AI summary
Designing Network Design Strategies Through Gradient Path Analysis
Chien-Yao Wang, Hong-Yuan Mark Liao, I-Hau Yeh
TL;DR
The paper addresses network architecture design beyond feed-forward data paths by examining how backpropagated gradients guide learning. It proposes gradient-path strategies at the layer, stage, and network levels, with experiments and theory supporting their feasibility and effectiveness. The paper also identifies limitations of data-path design, including possible training degradation and hardware optimization difficulties.
Problem
Most network design strategies focus on forward data paths and feature-processing units, while the backward gradient paths driving parameter learning receive less design attention.
Method
The paper designs architectures by analyzing gradient paths during backpropagation at layer, stage, and network levels.
Results
The proposed gradient-path strategies are reported to improve network learning ability across layer-, stage-, and network-level designs, with YOLOR-PRN improving object-detection AP by 0.5% and YOLOR-CSP by 1.5% versus YOLOR-v3.
Takeaways & Limitations
Gradient-path design can reuse existing computing units while re-planning gradient flow to reduce parameters, computation, and hardware resources and improve inference speed.
Takeaways & Limitations
Data-path design can sometimes cause unpredictable training degradation and may require complex architectures or additional hardware circuits for specialized arithmetic units.
Abstract
from arXiv · showhide
Designing a high-efficiency and high-quality expressive network architecture has always been the most important research topic in the field of deep learning. Most of today's network design strategies focus on how to integrate features extracted from different layers, and how to design computing units to effectively extract these features, thereby enhancing the expressiveness of the network. This paper proposes a new network design strategy, i.e., to design the network architecture based on gradient path analysis. On the whole, most of today's mainstream network design strategies are based on feed forward path, that is, the network architecture is designed based on the data path. In this paper, we hope to enhance the expressive ability of the trained model by improving the network learning ability. Due to the mechanism driving the network parameter learning is the backward propagation algorithm, we design network design strategies based on back propagation path. We propose the gradient path design strategies for the layer-level, the stage-level, and the network-level, and the design strategies are proved to be superior and feasible from theoretical analysis and experiments.
1. Introduction
The paper argues that neural-network architecture design should analyze backward gradient paths, not only forward data paths, to improve learning ability. It introduces layer-, stage-, and network-level strategies for this purpose.
- Motivation: DNN architecture design is important because researchers seek efficient, accurate, and low-cost solutions for diverse devices and tasks.Related systems such as SqueezeNet, MobileNet, and ShuffleNet explicitly address parameter count, latency, or hardware characteristics.
- Data-path design: Modern network design commonly focuses on feature fusion, receptive-field enhancement, attention, and branch selection along the feed-forward data path.These approaches generally assume shallow layers extract low-level features and deep layers extract high-level features.
- Gradient-path perspective: Backpropagation propagates objective-function information through gradients to shallow layers, motivating architecture design based on gradient paths.The paper treats gradient analysis as a way to understand how the objective function guides weight updates.
- Three design levels: Layer-level design adjusts layer counts and residual-connection channel ratios to create gradient-flow shunting strategies, producing Partial Residual Network.PRN is presented as the layer-level strategy for testing the gradient-path hypothesis.
- Three design levels: Stage-level design maximizes gradient combinations while minimizing hardware computation, producing Cross Stage Partial Network.This strategy adds hardware considerations to accelerate inference.
- Three design levels: Network-level design considers whole-network gradient propagation efficiency and path length to balance learning ability, producing Efficient Layer Aggregation Network.The strategy evaluates gradient propagation at the network scale rather than only within layers or stages.
2. Methodology
The paper divides network design into data-path and gradient-path strategies, then develops gradient-path designs at layer, stage, and network levels. These designs target gradient combinations, hardware cost, and gradient propagation efficiency through PRN, CSPNet, and ELAN.
- Network design strategies: Gradient-path design analyzes gradient sources and propagation to guide network architecture, complementing data-path feature extraction, selection, and fusion.
- Layer-level design: PRN uses masked and asymmetric residual layers to increase gradient combinations and support feature maps with different channel counts.Masking identity connections changes gradient sources and timestamps, while asymmetric residual operations avoid some transition layers.
- Stage-level design: CSPNet extends gradient-combination design from layers to stages while additionally optimizing hardware inference speed.Its cross-stage partial operation separates feature maps, processes one part computationally, and combines it with the other part before the next stage.
- Stage-level design: CSP networks use fusion and gradient-flow truncation structures to reduce duplicated gradient flow across paths.The described variants differ in the ordering of transition and concatenation operations.
- Network-level design: ELAN designs network-level layer aggregation by considering shortest and longest gradient paths to improve gradient propagation during model scaling.The motivation includes deteriorating convergence and accuracy when computational blocks are repeatedly stacked beyond a critical depth.
3. Analysis
The analysis characterizes gradient combinations through timestamps and sources, then connects these properties to PRN and CSPNet design choices. It also examines how gradient-path length and hardware efficiency affect learning and inference.
- Gradient combination: Gradient combinations consist of gradient timestamps and gradient sources, providing two dimensions for analyzing backward-propagation information.The paper uses these dimensions to examine how architectures propagate information and update weights.
- Gradient combination: ResNet shares many gradients with identical timestamps and sources, whereas DenseNet passes same-timestamp gradients from different sources to corresponding layers.The distinction is used to explain differences in information utilization across connection types.
- Gradient combination: PRN increases gradient-combination diversity by splitting channels, varying gradient timing, and diverting gradient sources while preserving the ResNet topology.Its masked residual layer enables this redistribution without changing the overall network topology.
- Cross-stage partial strategy: CSPNet combines gradient diversity with hardware efficiency by reducing duplicated information and splitting feature maps across computational stages.The strategy targets both learning ability and inference speed.
- Cross-stage partial strategy: A 50%-to-50% cross-stage split halves computational-block input and output channels, reducing computational and parameter costs while minimizing memory access cost.The analysis uses Darknet-53 as an example of this resource-saving structure.
- Gradient-path length: Shorter overall gradient paths do not necessarily produce stronger learning, and very deep stacking can degrade learning ability even when path-combination length is fixed.The paper motivates stochastic training with shallower random subnetworks as a response to this issue.
4. Experiments
Experiments evaluate layer-, stage-, and network-level gradient-path strategies on COCO using YOLOR-based baselines. PRN, CSPNet, and ELAN improve reported accuracy or efficiency, with ELAN providing the strongest overall comparison result.
- Layer-level design: 0.5% AP improvement on object detection is reported for YOLOR-PRN versus YOLOR-v3, with additional instance-segmentation gains of 0.1% AP and 0.3% AP75.The PRN experiment keeps the network parameters and topology unchanged while reducing residual additions by half.
- Stage-level design: 22% lower computation and 1.5% higher object-detection AP are reported for YOLOR-CSP versus YOLOR-v3.The experiment uses a 50%-to-50% gradient split ratio.
- Network-level design: 0.7% higher object-detection AP is reported for ELAN with 35% less computation than YOLOR-v3.ELAN varies computational-block stacking to trade off accuracy and computation.
- Network-level design: Under the 2,1s stack setting, YOLOR-ELAN improves object-detection AP by 1.9% and instance-segmentation AP by 0.6% while reducing computation by 26%.The comparison varies stacking times in the backbone and neck.
- Comparison: Across the comprehensive comparison, gradient-path-designed models outperform the baseline methods in computation, parameter count, and accuracy, with YOLOR-ELAN performing best overall.The comparison includes YOLOR-PRN, YOLOR-CSP, and YOLOR-ELAN against YOLOR-v3 and YOLOv5 (r6.2).
5. Conclusions
The paper proposes designing network architectures through gradient paths, using simple path replanning to improve learning while reducing resource demands relative to data-path strategies.
- Gradient path design strategies improve network architecture and learning ability at the layer, stage, and network levels.
- Compared with data-path strategies, gradient-path design can rely on existing computing units rather than adding complex topology.
- Simple gradient-path replanning can reduce parameters, computation, and hardware resources while improving inference speed and network learning effect.
- The proposed strategy establishes an effective and concise rule for network architecture design.