PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Neural Networks: A Survey

Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, Yulin Wang

arXiv:2102.04906v4cs.CV

TL;DR

Dynamic networks address the limitations of static inference, while a systematic review of this rapidly developing area remains lacking. This survey organizes dynamic networks by instance-, spatial-, and temporal-wise adaptation, reviews their architectures and training problems, and identifies open directions.

  • Problem

    Static inference fixes the computational graph and network parameters, while a comprehensive review of dynamic-network advances and challenges remains lacking.

  • Method

    The survey systematically reviews dynamic networks by instance-wise, spatial-wise, and temporal-wise adaptation, including architecture design, decision making, optimization, and applications.

  • Results

    The survey provides an overview of dynamic-network advances, connects related subareas, and summarizes key challenges and possible future research directions.

  • Takeaways & Limitations

    Dynamic-network research spans adaptive architectures and computation, with future work including dynamic Transformer structures and principled decision functions.

  • Takeaways & Limitations

    The survey leaves out detailed discussion of self-attention and focuses instead on feature re-weighting in dynamic inference.

Abstract

from arXiv · show

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

1 INTRODUCTION

Dynamic networks adapt their structures or parameters to inputs instead of using fixed inference graphs and parameters. This survey reviews their benefits, three adaptive-computation granularities, research techniques, applications, and open problems.

  • Static deep models use fixed computational graphs and parameters during inference, which may limit representation power and efficiency.
  • Dynamic networks allocate computation on demand by selectively activating layers, channels, or sub-networks for each input.This can reduce computation for easy samples and less informative spatial or temporal locations.
  • Dynamic networks improve representation power through data-dependent architectures or parameters, including feature-conditioned attention over convolutional kernels.Soft attention over channels, spatial areas, or temporal locations also fits this dynamic-inference framework.
  • Dynamic models support accuracy-efficiency trade-offs under changing computational budgets and can adapt to different hardware platforms and environments.
  • The survey covers sample-wise, spatial-wise, and temporal-wise adaptive computation, then reviews decision strategies, training techniques, applications, and future directions.It is motivated by the lack of a systematic and comprehensive review of this rapidly developing area.

2 SAMPLE-WISE DYNAMIC NETWORKS

Sample-wise dynamic networks process different inputs with data-dependent computation. They either adjust architectures to reduce redundant computation or adapt parameters while retaining a fixed computational graph.

  • Sample-wise dynamic networks adapt their computation to each input through architecture changes or parameter changes.

2.1 Dynamic Architectures

Dynamic architectures conditionally vary network depth, width, or routing paths according to each sample. These designs include early exiting, layer skipping, adaptive precision, mixture-of-experts, and neural routing structures.

  • Dynamic architectures adjust network depth, width, or routing within a SuperNet to match computation with each sample.They aim to save redundant computation for easy samples while preserving representation power.
  • Early exiting outputs easy samples at shallow exits, whereas layer skipping selectively omits intermediate layers.Early exits terminate inference at an intermediate layer; layer skipping uses per-layer execution decisions.
  • Dynamic depth: Early-exit designs include cascaded networks, intermediate classifiers, and multi-scale architectures with adaptive exiting policies.Intermediate classifiers reuse features within one backbone, while multi-scale designs address interference among classifiers.
  • Dynamic depth: Layer skipping can use halting scores, gating functions, or policy networks to determine which residual blocks execute.Gating functions produce binary execution decisions, while policy networks can generate decisions for all backbone layers.
  • Dynamic width: Adaptive bit-width and fractional skipping vary numerical precision or block computation according to resource budgets and input features.
  • Dynamic width: Dynamic width selectively activates neurons, channels, or branches, including mixture-of-experts structures with input-dependent expert selection.Soft mixture-of-experts weights all branches, whereas hard gates can avoid executing experts whose gate value is zero.

2.2 Dynamic Parameters

Dynamic parameters adapt a network’s parameters to each input while keeping its computational graph fixed. The survey covers parameter adjustment, direct weight prediction, and feature rescaling through soft attention, along with their efficiency and implementation trade-offs.

  • Dynamic-parameter models keep architectures fixed while adapting parameters to each input, aiming to improve representation power with little additional computation.
  • Parameter adaptation adjusts trained weights, predicts parameters with an independent model, or rescales features using input-dependent soft attention.These three implementations are summarized in Fig. 6.
  • Parameter adjustment: Weight adjustment includes attention over kernels or spatial weight locations, while kernel-shape adaptation changes effective receptive fields through dynamic sampling.
  • Parameter adjustment: Fusing N convolutional branches can be replaced by one convolution with an aggregated kernel, consuming approximately 1/N of the computation.
  • Implementation considerations: Kernel-shape adaptation can require customized CUDA kernels because of irregular memory access and computation patterns.Hardware–algorithm co-design on FPGAs has been reported to improve practical efficiency.
  • Weight prediction: Weight prediction directly generates input-adaptive parameters at test time and has been extended to general architectures and task-specific feature embeddings.
  • Dynamic features: Feature rescaling with soft attention requires minor computational-graph modifications and can be equivalent to convolution with dynamic weights.For convolutional features, a lightweight pooling-and-linear function produces channel-wise attention coefficients.
  • Dynamic features: Soft-attention implementations include channel-wise, spatial-wise, and multi-branch feature reweighting, with performance varying across backbone networks.

3 SPATIAL-WISE DYNAMIC NETWORKS

Spatial-wise dynamic networks adapt inference to different image locations because spatial contributions and required computation vary across an image. The survey organizes these methods by pixel, region, and resolution granularity.

  • Not all image locations contribute equally to CNN predictions, creating potential to reduce redundancy by processing only selected pixels or regions.
  • Spatial-wise dynamic networks perform adaptive inference at image locations and are categorized into pixel-level, region-level, and resolution-level methods.

3.1 Pixel-level Dynamic Networks

Pixel-level dynamic networks adapt architectures or parameters at individual image locations. They exploit spatial redundancy through sparse computation, additional refinement, and location-dependent weights or receptive fields.

  • Pixel-level dynamic networks are divided into pixel-specific dynamic architectures and dynamic parameters.
  • Pixel-wise dynamic architectures: Dynamic sparse convolution computes only on selected pixels, while additional refinement allocates extra layers or channels to selected spatial positions.
  • Pixel-wise dynamic architectures: Pixel-wise halting can skip deeper processing for confident locations, while interpolation fills unselected positions to reduce performance degradation.
  • Pixel-wise dynamic architectures: Pixel-wise early exiting and PointRend allocate computation according to prediction confidence, with PointRend refining uncertain pixels near object borders.
  • Pixel-wise dynamic parameters: Pixel-level dynamic weights use test-time adjustment, prediction, or dynamic features to produce location-dependent parameters and receptive fields.
  • Region-wise dynamic networks: Region-level dynamic inference selects or transforms regions before subsequent network processing, while region-aware convolution can generate weights for segmented regions.

3.2 Region-level Dynamic Networks

Region-level dynamic networks adapt computation across image regions or patches, either transforming selected regions or using hard attention to select informative patches. These approaches target improved prediction effectiveness and/or inference efficiency.

  • Region-level approaches: Region-level methods apply parameterized transformations to feature-map regions or select patches through hard spatial attention.The two main directions are region transformations and patch-level hard attention.
  • Dynamic transformations: Dynamic transformations can undo image variations or emphasize salient regions for better generalization or discriminative representations.Spatial transformers generate transformation parameters with a localization network before applying the transformation.
  • Hard spatial attention: Hard spatial attention strategically selects input patches because informative features may be concentrated in particular image regions.RNN-based classifiers can iteratively classify cropped patches and choose the next attentional location.
  • Hard attention with RNNs: RNN-based hard-attention methods formulate classification as sequential decision making over cropped patches, with later variants supporting adaptive stopping.At each step, the classifier observes a cropped patch and decides the next location until termination.

3.3 Resolution-level Dynamic Networks

Resolution-level dynamic networks adapt feature resolution to input difficulty or content, reducing redundant processing for samples that do not require high resolution. They use adaptive scaling or selective activation of resolution-specific subnetworks.

  • Adaptive resolution: Resolution-level methods process whole images with adaptive feature resolutions instead of applying one resolution uniformly to every input.Low resolution may suffice for many easy samples, whereas conventional CNNs use the same resolution for all inputs.
  • Adaptive scaling: Adaptive scaling predicts suitable resolution ratios and zooms inputs or adjusts convolutional strides accordingly.Examples include face-scale prediction followed by adaptive zooming and stride prediction within ResNet stages.
  • Resolution-specific subnetworks: Parallel or cascading subnetworks with different resolutions can be selectively activated according to each input.Elastic performs soft branch selection, while another design uses hard low-to-high resolution selection to avoid redundant computation.
  • Resolution-specific subnetworks: Hard resolution selection conditionally activates subnetworks that process representations from low to high resolution during inference.This design is intended to avoid redundant computation while preserving access to higher-resolution processing when needed.

4 TEMPORAL-WISE DYNAMIC NETWORKS

Temporal-wise dynamic networks adapt computation across sequence positions in text and video by skipping, simplifying, or terminating updates and by selecting where or what to read. The survey organizes these methods around dynamic state updates, adaptive reading, and video frame selection.

  • Overview: Temporal-wise models allocate less or no computation to unimportant temporal locations in texts and videos.They adapt computation along the temporal dimension rather than treating every input position identically.
  • Dynamic state updates: Dynamic RNNs reduce per-step cost by skipping hidden-state updates, partially updating states, or selectively updating hierarchical levels.These mechanisms correspond to skipped updates, data-dependent depth or width, and hierarchical temporal dependencies.
  • Dynamic state updates: Hierarchical RNNs selectively update higher-level states based on lower-level outputs to encode dependencies at different timescales.The hierarchy can reflect structures such as sentence, word, and character levels.
  • Adaptive reading: Adaptive reading uses early exiting or dynamic jumping so models can stop before the sequence ends or skip task-irrelevant tokens.Jumping methods may choose a stride, reread the current input, or skip a flexible number of words.
  • Dynamic video recognition: Video recognition adapts temporal computation through recurrent state updates or pre-sampling of key frames and clips.Recurrent methods use glimpses, early exits, or jumps, while pre-sampling methods select salient frames with temporal attention or sampling modules.
  • Joint adaptation: Jointly adapting temporal sampling with spatial computation, resolution, or patch selection is identified as a promising direction for exploiting redundancy.The survey highlights simultaneous adjustment of multiple model and input factors.

5 INFERENCE AND TRAINING

Dynamic inference relies on data-dependent decisions, while training must address discrete choices, sparsity, multiple exits, and non-differentiable controllers. The survey reviews confidence criteria, policy networks, gating functions, training objectives, gradient approximations, and reinforcement learning.

  • Overview: Dynamic networks make inference decisions conditioned on inputs, but training them is generally more challenging than optimizing static networks.The survey focuses especially on discrete decisions and their training strategies.
  • Decision making: Confidence-based criteria enable early exits when intermediate predictions satisfy thresholds based on SoftMax confidence, entropy, score margin, or prediction stability.Thresholds are typically tuned on validation data to control the accuracy-efficiency trade-off.
  • Decision making: Policy networks process each sample and directly determine which parts of a backbone or SuperNet should be activated.They can adapt network depth, width, or routing, but some designs are specific to a particular backbone.
  • Decision making: Gating functions use intermediate features to produce binary decisions controlling channels, layers, paths, or input locations.They are flexible plug-ins but generally require specialized training because their decisions are non-differentiable.
  • Training objectives: Multi-exit networks commonly minimize a weighted cumulative loss over intermediate classifiers, while sparsity objectives constrain activated units or resource use.A typical objective is L = Ltask + γLsparse, with alternatives including target activation rates, gate norms, or FLOPs-aware losses.
  • Optimization: Non-differentiable decisions are trained using gradient estimation, reparameterization, or reinforcement learning, each introducing optimization considerations.Reparameterization can lengthen training and increase hyperparameter sensitivity, while reinforcement learning may require costly multi-stage training and carefully designed rewards.

6 APPLICATION OF DYNAMIC NETWORKS

Dynamic networks support adaptive inference across images, text, video, and multimodal tasks, while also addressing broader deep-learning problems such as overthinking, long-tailed classification, and robustness.

  • Image and Video Applications: Image-recognition methods primarily use sample-wise or spatial-wise adaptive inference, with early exiting enabling temporal adaptation through sequential decision procedures.These inference paradigms can also generalize to other applications.
  • Text Applications: Temporal-wise dynamic RNNs reduce temporal redundancy in text and can also process audio, while adaptive-depth Transformers and BERT reduce redundant computation.Adaptive depths are extensively studied for large language models.
  • Image and Video Applications: Video tasks can combine instance-wise, spatial-wise, and temporal-wise dynamic inference, although most non-recurrent 3D CNNs remain static.The survey identifies dynamic 3D CNNs as an interesting future research direction.
  • Multimodal Applications: Dynamic networks have been used to fuse multimodal features in RGB-D segmentation and image or video captioning.Attention mechanisms are among the approaches used for dynamic multimodal fusion.
  • Fundamental Deep-Learning Problems: Multi-exit models address overthinking, long-tailed classification, and robustness, while dynamic routing reduces training cost in some multitask settings.The passage lists these as applications to fundamental deep-learning problems.

7 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies open challenges in theory, architecture, generalization, task applicability, hardware support, robustness, and interpretability, and proposes corresponding research directions.

  • Theories for Dynamic Networks: Dynamic-network theory remains limited, motivating principled analyses of representation power and decision functions with theoretical guarantees.Existing confidence criteria, policy networks, and gating functions can be effective but may lack theoretical justification.
  • Theories for Dynamic Networks: Heuristic early-exit decisions can suffer from overconfidence, threshold sensitivity, and poor transferability, while learned gating may add computation and unstable training.These limitations motivate more principled decision-function design.
  • Generalization Issues: Dynamic inference can violate the usual i.i.d. assumption because different sub-networks process non-uniform subsets of test samples.Smaller sub-networks tend to handle easy samples, whereas larger ones handle hard inputs.
  • Architecture Design for Dynamic Networks: Future architectures may be designed manually or with NAS, including dynamic Vision Transformers using adaptive early exiting or token sparsification.The survey expects architectures specialized for dynamic networks to improve effectiveness and efficiency.
  • Dynamic Networks versus Model Compression: Unlike model compression, dynamic networks perform data-dependent computation and may prioritize computational efficiency even when model size slightly increases.Compression typically produces compact static models that treat all inputs identically.
  • Applicability for More Diverse Tasks: Most sample-wise adaptive networks target classification and do not directly extend to detection or segmentation, where images contain objects and pixels with differing difficulty.A unified dynamic backbone for diverse vision tasks remains challenging.
  • Hardware and Systems: Static-model-oriented hardware and libraries can make practical runtime lag behind theoretical efficiency, especially for sparse computation and sequential inference.Adaptive computational graphs also conflict with parallel computation, although batching has been enabled for early-exiting networks.
  • Adversarial Robustness: Dynamic networks raise distinct robustness concerns because adversarial attacks can force full-layer activation or confuse early exits, while standard adversarial training may not defend against them.The survey characterizes robustness of dynamic networks as understudied.
Loading 2102.04906v4…