PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic ReLU

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu

arXiv:2003.10027v2cs.CV

TL;DR

Static ReLU and its generalizations behave identically across inputs, motivating a more adaptive rectifier. The paper proposes DY-ReLU, which generates piecewise-linear activation parameters from global input context, and reports improved classification and keypoint detection with modest added computation.

  • Problem

    Static ReLU and its generalizations perform identically for different inputs, raising whether rectifiers should instead adapt to input samples.

  • Method

    DY-ReLU uses a hyper function over all input elements to generate parameters for a piecewise-linear activation function.

  • Results

    DY-ReLU improves ImageNet classification and COCO keypoint detection across multiple networks with only a 5% increase in computational cost.

  • Takeaways & Limitations

    DY-ReLU provides greater representation capability than static activations while remaining suitable for lightweight CNNs.

  • Takeaways & Limitations

    The method uses fixed hyperparameters, including αk, βk, λa, and λb, with default values specified for K = 2.

Abstract

from arXiv · show

Rectified linear units (ReLU) are commonly used in deep neural networks. So far ReLU and its generalizations (non-parametric or parametric) are static, performing identically for all input samples. In this paper, we propose dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements. The key insight is that DY-ReLU encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly. Compared to its static counterpart, DY-ReLU has negligible extra computational cost, but significantly more representation capability, especially for light-weight neural networks. By simply using DY-ReLU for MobileNetV2, the top-1 accuracy on ImageNet classification is boosted from 72.0% to 76.2% with only 5% additional FLOPs.

1 Introduction

The paper asks whether static rectifiers should adapt to inputs and proposes DY-ReLU, which dynamically adjusts activation functions using global input context. It reports improved classification and keypoint detection with only a slight computational increase.

  • 1 Introduction: Static ReLU and its generalizations apply identically to different inputs, motivating adaptive rectifiers.The paper frames dynamic rectifiers as an investigation of whether rectifiers should be fixed or input-adaptive.
  • 1 Introduction: DY-ReLU computes piecewise-linear activation parameters from a hyper function over all input elements.The hyper function encodes global context to adapt the activation function, with negligible computational cost.
  • 1 Introduction: DY-ReLU includes spatial- and channel-sharing variations whose suitability differs across image classification and keypoint detection.Channel-wise variants suit image classification; backbone and head networks for keypoint detection favor different variants.
  • 1 Introduction: 5% additional computational cost accompanies DY-ReLU's reported improvements across multiple networks and tasks.On MobileNetV2, the method gains 4.2% top-1 accuracy for image classification and 3.5 AP for keypoint detection.

2 Related Work

The paper relates DY-ReLU to prior activation, hypernetwork, attention, dynamic convolution, and efficient-CNN methods. Its distinction is adapting activation functions rather than convolutional kernel weights.

  • 2 Related Work: Prior activation-function work progresses from fixed and leaky slopes to learnable, random, grouped, and maximum-based rectifications.Examples include absolute value rectification, LeakyReLU, PReLU, RReLU, and Maxout.
  • 2 Related Work: Hypernetworks, squeeze-and-excitation, and dynamic convolution make network components input-dependent in different ways.These methods generate parameters, reweight channels using global context, or adapt convolution kernels through input-dependent attention.
  • 2 Related Work: DY-ReLU shifts dynamic adaptation from convolutional kernel weights to activation functions.This positions the method relative to dynamic convolution while targeting efficient CNN architectures.

3 Dynamic ReLU

DY-ReLU makes a piecewise-linear activation input-dependent by generating its parameters with a hyper function. The design encodes global context while retaining low computational cost and connects to several existing activation mechanisms.

  • 3 Dynamic ReLU: DY-ReLU is a dynamic piecewise function whose parameters depend on the input without increasing network depth or width.The paper presents it as an efficient way to increase model capability.
  • 3.1 Dynamic Activation: A dynamic activation combines a hyper function that computes parameters with an activation function that applies them across channels.The hyper function and activation function together define the dynamic activation pipeline.
  • 3.1 Dynamic Activation: The hyper function encodes global context from all input elements to select an appropriate activation function.This is presented as the source of greater representation power than static activations, especially in lightweight models.
  • 3.2 Definition and Implementation of Dynamic ReLU: DY-ReLU extends parametric piecewise-linear rectification by adapting its coefficients from all input elements.The coefficients are produced by the hyper function rather than remaining static.
  • 3.2 Definition and Implementation of Dynamic ReLU: For K functions and C channels, the hyper function outputs channel-wise slope and intercept parameters for the dynamic activation.The implementation uses a lightweight squeeze-and-excitation-like network with global average pooling, fully connected layers, and normalization.
  • 3.2 Definition and Implementation of Dynamic ReLU: The residual parameter range is controlled by λa and λb, while αk, βk, λa, and λb are hyperparameters.For K = 2, the default initialization corresponds to static ReLU, with λa = 1.0 and λb = 0.5.
  • 3.3 Relation to Prior Work: Static ReLU, LeakyReLU, and PReLU appear as special cases when the hyper function becomes static.The paper also characterizes DY-ReLU as an efficient dynamic Maxout variant using one convolutional kernel.

4 Variations of Dynamic ReLU

The paper introduces three DY-ReLU variants that differ in how activation functions share spatial and channel information, trading representation capability against computational cost. Experiments show channel-wise variants are preferred for classification and backbones, while spatial-wise adaptation is important in keypoint-detection heads.

  • Network Structure and Complexity: DY-ReLU-A shares one activation function across all spatial positions and channels, reducing computational cost but representation capability.Its hyper function outputs 2K parameters.
  • Network Structure and Complexity: DY-ReLU-B shares activation functions spatially but uses channel-wise functions, with 2K parameters generated per channel.This corresponds to 2KC hyper-function outputs.
  • Network Structure and Complexity: DY-ReLU-C assigns each input element a unique activation function, then decouples spatial locations from channels using channel parameters and spatial attention.The decoupling avoids the otherwise large 2KCHW output dimension of the fully connected layer.
  • Ablations: On ImageNet MobileNetV2 ×0.35, channel-wise DY-ReLU-B and DY-ReLU-C outperform channel-shared DY-ReLU-A, while spatial-wise adaptation adds no improvement.The table reports means and standard deviations over three runs, with bracketed improvements over baseline.
  • Ablations: On COCO keypoint detection, channel-wise variants are better in the backbone, whereas spatial-wise DY-ReLU-C is more effective in the head.Using DY-ReLU-C in both backbone and head achieves 4 AP improvement; spatially shared variants can perform worse than baseline in the wrong component.

5 Experimental Results

Experiments across ImageNet classification and COCO keypoint detection show that DY-ReLU improves performance over static activations and prior activation functions with modest computational cost. Ablations examine dynamic function design, placement, hyperparameters, and the activation's learned behavior.

  • ImageNet Classification: DY-ReLU outperforms static counterparts across ResNet, MobileNetV2, and MobileNetV3 with approximately 5% extra computational cost.On MobileNetV2, it gains more than 4.2% top-1 accuracy; MobileNetV3 gains 2.3% and 0.7% for Small and Large models, respectively.
  • ImageNet Classification: DY-ReLU outperforms prior activation functions on MobileNetV2, including Maxout despite Maxout's significantly higher computational cost.The comparison uses MobileNetV2 width multipliers ×0.35 and ×1.0 with channel-wise DY-ReLU-B.
  • Inspecting DY-ReLU: DY-ReLU outputs vary across images for the same input value, with dynamic ranges differing across network blocks.This behavior distinguishes it from static ReLU, and 51% of DY-ReLU functions have a segment with negative slope or slope above 1.
  • Inspecting DY-ReLU: Activation bending decreases from lower to higher network levels, indicating that learned dynamic functions differ across blocks.The figure measures the angle, or slope difference, between the two DY-ReLU segments.
  • Ablation Studies: 63.8% top-1 accuracy results when the first piecewise segment is made dynamic, up from 60.3%, with the second dynamic segment adding 1.9%.The intercept is helpful consistently, while the gap between K = 2 and K = 3 is small.
  • Ablation Studies: Using DY-ReLU in more layers improves accuracy, while placement after the 3 × 3 depthwise convolution performs best when only one layer is used.A reduction ratio of R = 8 provides a reported trade-off between representation capacity and computational cost.
  • Ablation Studies: Classification accuracy is stable across many slope and intercept initializations, but slope ranges that are too wide or narrow are suboptimal.The reported good choice is to keep λa between 1 and 2.
  • COCO Keypoint Detection: 3.5 AP and 4.0 AP gains are achieved on COCO keypoint detection with MobileNetV2 width multipliers ×1.0 and ×0.5, respectively.MobileNetV3-Large and Small gain 1.5 and 3.6 AP, respectively.

6 Conclusion

The paper introduces DY-ReLU, which dynamically adapts a piecewise linear activation for each input. It reports improved image classification and human pose estimation with negligible extra computation, positioning DY-ReLU as a component for efficient CNNs.

  • Conclusion: DY-ReLU dynamically adapts a piecewise linear activation function for each input.It is designed to integrate into existing CNN architectures.
  • Conclusion: Replacing ReLU or h-swish with DY-ReLU improves image classification and human pose estimation in ResNet and MobileNet architectures.The paper reports solid improvements across these tasks.
  • Conclusion: DY-ReLU offers greater representation capability with negligible extra computation cost and is described as friendly to efficient CNNs.The authors propose it as a useful component for efficient network architecture.
Loading 2003.10027v2…