PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Networks with Adaptive Inference Graphs

Andreas Veit, Serge Belongie

arXiv:1711.11503v3cs.CVcs.LG

TL;DR

Fixed feed-forward ConvNets may execute many layers devoted to unrelated concepts even after recognizing an image’s high-level category. ConvNet-AIG instead builds an input-conditioned inference graph by selecting general and specialized layers, achieving improved classification with lower computation and greater adversarial robustness than ResNets. The paper also identifies a scope for future attacks targeting its gating functions.

  • Problem

    Fixed feed-forward ConvNets can become larger and slower as category counts rise and may compute features for unrelated concepts after high-level recognition.

  • Method

    ConvNet-AIG learns general and category-specialized layers and uses input-dependent gates to decide which layers to execute, with relevance estimated from globally pooled channel statistics.

  • Results

    On ImageNet, ConvNet-AIG 50 and 101 outperform their ResNet counterparts while using 20% and 38% less computations, and show higher robustness to adversarial examples.

  • Takeaways & Limitations

    Grouping parameters for related classes and executing only relevant layers improves efficiency and overall classification quality while enabling distinct inference graphs.

  • Takeaways & Limitations

    Future work should investigate adversarial attacks that specifically target the gating functions, and extend the approach beyond ResNets.

Abstract

from arXiv · show

Do convolutional networks really need a fixed feed-forward structure? What if, after identifying the high-level concept of an image, a network could move directly to a layer that can distinguish fine-grained differences? Currently, a network would first need to execute sometimes hundreds of intermediate layers that specialize in unrelated aspects. Ideally, the more a network already knows about an image, the better it should be at deciding which layer to compute next. In this work, we propose convolutional networks with adaptive inference graphs (ConvNet-AIG) that adaptively define their network topology conditioned on the input image. Following a high-level structure similar to residual networks (ResNets), ConvNet-AIG decides for each input image on the fly which layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns distinct inference graphs for different categories. Both ConvNet-AIG with 50 and 101 layers outperform their ResNet counterpart, while using 20% and 38% less computations respectively. By grouping parameters into layers for related classes and only executing relevant layers, ConvNet-AIG improves both efficiency and overall classification quality. Lastly, we also study the effect of adaptive inference graphs on the susceptibility towards adversarial examples. We observe that ConvNet-AIG shows a higher robustness than ResNets, complementing other known defense mechanisms.

1 Introduction

ConvNet-AIG addresses the inefficiency of fixed feed-forward ConvNets by selecting input-conditioned subsets of layers. It combines gated residual-style computation with specialized layers, improving ImageNet accuracy and reducing computation.

  • Motivation: ConvNets can become larger and slower as category counts rise because many parameters encode category-specific high-level features.For each image, more computed features may also focus on unrelated concepts.
  • Motivation: Input-conditioned layer selection could let a network move from high-level recognition directly to layers distinguishing fine-grained categories.The proposed motivation resembles decision trees selecting informative features, while avoiding unrelated intermediate computation.
  • Approach: ConvNet-AIG learns general and category-specialized convolutional layers, then selects the layers needed for each input without label-hierarchy supervision.Its topology is adaptively defined from the input image.
  • Approach: Gates determine whether each residual layer executes or skips, producing inference graphs conditioned on individual input images.The architecture retains identity skip-connections similar to ResNets.
  • Results: 20% and 38% less computations accompany higher accuracy for ConvNet-AIG 50 and 101 than their respective ResNet counterparts on ImageNet.The model also learns distinct graphs and specialized layers for related categories without specific supervision.

2 Related Work

ConvNet-AIG differs from prior adaptive and attention-based approaches by constructing a hard, input-conditioned layer subset rather than using a fixed graph, early stopping, or soft rescaling.

  • Input-conditioned computation: Unlike visual-question-answering and zero-shot approaches that construct fixed graphs, ConvNet-AIG builds its convolutional graph during execution from the input image.The distinction is whether graph construction is fixed up front or conditioned on the current image.
  • Input-conditioned computation: Unlike adaptive early-stopping methods, ConvNet-AIG selects a subset of layers, enabling category-specific parameter grouping and distinct inference graphs.The related spatially adaptive approach determines when to stop, whereas ConvNet-AIG determines which layers to execute.
  • Regularization: Stochastic depth drops residual layers independently of the input to increase redundancy, whereas ConvNet-AIG conditions layer execution on the image to reduce redundancy.Both approaches use stochastic nodes during training, but they pursue different objectives.
  • Attention: Soft attention methods rescale every residual layer according to importance, while ConvNet-AIG uses hard attention that can avoid executing selected layers.This hard selection decouples computation time from the number of categories.

3 Adaptive Inference Graphs

ConvNet-AIG extends residual-style networks with input-conditioned gates that select which layers to execute. Its gates estimate layer relevance and use stochastic, differentiable decisions to construct adaptive inference graphs with specialized computation paths.

  • Adaptive topology: ConvNet-AIG selects an input-dependent subset of residual layers instead of executing every layer in a fixed sequence.The gate determines whether each layer is needed for the current image.
  • Gated computation: Each gate estimates a layer’s relevance from the preceding feature map and then decides whether to execute that layer.The gate uses channel-wise statistics rather than the full feature map to reduce computational cost.
  • Estimating layer relevance: The relevance estimator compresses features through global average pooling into a 1 × 1 × C channel descriptor before modeling channel dependencies.A lightweight nonlinear function produces execution and skipping scores for the layer.
  • Estimating layer relevance: The gate compares two unnormalized scores: β1 for computing the layer and β0 for skipping it.Execution is favored when the execution score exceeds the skipping score.
  • Gumbel sampling: Gumbel-Max sampling supplies discrete gate decisions, while its softmax relaxation enables gradient propagation through those decisions.The straight-through estimator uses discrete samples in the forward pass and relaxed gradients in the backward pass, although this introduces bias.

4 Experiments

The experiments evaluate ConvNet-AIG’s performance, specialization, training and inference modes, ablations, and robustness to adversarial attacks.

  • Experimental evaluation: The evaluation measures accuracy, specialized layers, category-specific inference graphs, training and inference modes, target rates, thresholds, and adversarial robustness.

4.1 Results on CIFAR

On CIFAR-10, ConvNet-AIG improves over ResNet counterparts while executing fewer layers, and its all-layer variant also surpasses stochastic depth.

  • Results: ConvNet-AIG 110 clearly outperforms ResNet 110 while using only 82% of the layers in expectation.The additional gates add 0.01% floating-point-operation overhead and 4.8% parameter overhead relative to ResNet-110.
  • Results: ConvNet-AIG outperforms its ResNet counterparts even when executing only a subset of layers.
  • Results: ConvNet-AIG 110∗, which executes all layers, outperforms stochastic depth, indicating benefits beyond layer-dropping regularization.
  • Results: The model learns to always execute important downsampling layers while avoiding layers that are dropped every time.

4.2 Results on ImageNet

On ImageNet, ConvNet-AIG improves accuracy and reduces computation relative to matching ResNets while learning input-conditioned, category-specific layer usage. Its computation–accuracy trade-offs can be adjusted through target rates and inference thresholds, and learned graphs specialize across categories.

  • Quantitative comparison: ConvNet-AIG 50 and 101 outperform their ResNet counterparts while using 20% and 38% less computation, respectively.Deterministic inference consistently outperforms stochastic inference, especially for the 101-layer model.
  • Quantitative comparison: Decreasing the target rate reduces computation and can initially improve accuracy before further reductions decrease accuracy.
  • Quantitative comparison: Thresholds from 0.3 to 0.7 slightly adjust inference time for trained ConvNet-AIG 101 models without a large accuracy decrease.
  • Analysis of learned inference graphs: Learned execution rates differ between man-made objects and animals, including birds, mammals, and reptiles, revealing category-specific layer specialization.
  • Analysis of learned inference graphs: Downsampling and final layers execute for all images, whereas inter-class variation is mostly concentrated in later layers after the second downsampling layer.
  • Analysis of learned inference graphs: During training, important downsampling and final layers increase execution rates while less critical layers approach the target rate.
  • Analysis of learned inference graphs: ConvNet-AIG 50 executes 10.81 of 16 residual layers on average, with bird images using one fewer layer than consumer-goods images.

4.3 Robustness to adversarial attacks

ConvNet-AIG’s stochastic inference is more robust to adversarial attacks than ResNet, although it performs slightly worse than deterministic inference. Its gates remain stable under attack, suggesting Gumbel noise contributes to the observed robustness.

  • Stochastic inference has slightly lower performance than deterministic inference but may improve robustness to adversarial attacks.
  • ConvNet-AIG remains consistently more robust than ResNet across adversary strengths, including after JPEG compression.Both networks benefit from JPEG compression, but ConvNet-AIG retains higher robustness; the attack uses FGSM on ImageNet-trained ConvNet-AIG 50 and ResNet 50.
  • 74.62% to 11%: accuracy drops sharply under FGSM while ConvNet-AIG’s per-layer execution rates remain similar.The execution rates are averaged over all bird categories for ConvNet-AIG 50 under an FGSM attack with epsilon 0.047.
  • The added Gumbel noise may outweigh attack-induced noise, explaining why increased robustness appears only during stochastic inference.

5 Conclusion

The conclusion argues that ConvNet-AIG can assemble input-conditioned inference graphs instead of using fixed feed-forward structures. On ImageNet, this adaptive execution groups related classes into specialized layers, improving efficiency and classification quality, while future work includes broader architectures, constant-time execution, and attacks targeting gates.

  • ConvNet-AIG adaptively assembles stochastic or deterministic inference graphs on the fly based on each input image.
  • Grouping parameters for related classes lets the model execute only relevant specialized layers, decoupling inference time from the number of learned concepts.
  • 38% less computations for ResNet 101 achieves the same classification quality.
  • Future work includes extending adaptive inference beyond ResNets, fixing the number of executed layers for constant inference times, and targeting gating functions adversarially.
Loading 1711.11503v3…