PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatially Adaptive Computation Time for Residual Networks

Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, Ruslan Salakhutdinov

arXiv:1612.02297v2cs.CVcs.LG

TL;DR

Deep networks incur high computational costs, while existing attention methods may be unsuitable for multi-output or per-pixel prediction tasks. SACT dynamically allocates Residual Network layers across image positions, improving FLOPs-quality trade-offs on ImageNet and COCO while producing computation maps correlated with human fixations.

  • Problem

    Existing attention approaches either focus on a few rectangular regions, limiting multi-output and per-pixel prediction, or require computation at all positions.

  • Method

    SACT extends Adaptive Computation Time to Residual Networks, deterministically halting computation independently at spatial positions through an end-to-end trainable architecture.

  • Results

    SACT outperforms adaptive and non-adaptive baselines on ImageNet and achieves a superior FLOPs-quality trade-off on COCO object detection, with computation maps correlated with human eye fixations.

  • Takeaways & Limitations

    Spatially varying computation can support efficient Residual Network-based image classification and object detection while capturing image regions associated with human fixations.

  • Takeaways & Limitations

    Deterministic ACT and SACT training can leave later residual units undertrained, and recovering these dead units is very slow.

Abstract

from arXiv · show

This paper proposes a deep learning architecture based on Residual Network that dynamically adjusts the number of executed layers for the regions of the image. This architecture is end-to-end trainable, deterministic and problem-agnostic. It is therefore applicable without any modifications to a wide range of computer vision problems such as image classification, object detection and image segmentation. We present experimental results showing that this model improves the computational efficiency of Residual Networks on the challenging ImageNet classification and COCO object detection datasets. Additionally, we evaluate the computation time maps on the visual saliency dataset cat2000 and find that they correlate surprisingly well with human eye fixation positions.

1. Introduction

The paper introduces Spatially Adaptive Computation Time (SACT), which dynamically stops Residual Network computation independently across spatial positions through a deterministic, end-to-end trainable policy. This preserves image-feature alignment, supports multi-output and per-pixel tasks, and improves computational efficiency on ImageNet and COCO.

  • Motivation: Deep convolutional networks achieve high accuracy across diverse vision tasks but incur huge computational cost.They are used for classification, detection, segmentation, image-to-text, visual question answering, and image generation.
  • Motivation: Glimpse-based attention is unsuitable for multi-output and per-pixel prediction, while soft spatial attention cannot save computation.Glimpse methods require carefully selected rectangular regions and separate prediction or heuristic procedures for glimpse positions; soft attention evaluates all spatial positions.
  • Method: SACT applies Adaptive Computation Time to Residual Networks and adapts the number of evaluated layers between spatial positions.The mechanism dynamically stops computation at each position when its features become “good enough.”
  • Method: SACT is deterministic, end-to-end trainable, and maintains image-feature alignment, making it suitable for multi-output and per-pixel prediction problems.The architecture incorporates attention into Residual Networks without requiring task-specific modifications.
  • Evaluation: SACT outperforms ACT and non-adaptive baselines on ImageNet classification and achieves a superior FLOPs-quality trade-off in COCO object detection.The method is used as a feature extractor in the Faster R-CNN pipeline for evaluation on COCO.

2. Method

The method augments ResNet with Adaptive Computation Time, which learns how many residual units to execute, and extends it spatially so computation varies by image position. The resulting policy is deterministic, differentiable for training, and can recover standard ResNet or ACT as special cases.

  • ResNet backbone: ResNet-101 uses pre-activation residual units arranged in four blocks, with an initial convolution and max-pooling stage of total stride four.The architecture is also extended for object detection and image segmentation, and the proposed models can apply to any ResNet architecture.
  • Adaptive Computation Time: ACT adds a halting-score branch to each residual unit and evaluates units sequentially until cumulative halting scores reach the stopping threshold.The final unit is forced to halt, and the number of evaluated units N is the first index whose cumulative score exceeds 1 − ε.
  • Adaptive Computation Time: A ponder cost ρ provides an almost-everywhere differentiable upper bound on N and is added to the original loss with coefficient τ.ACT is applied independently to each block, with ponder costs summed; the resulting loss is optimized by conventional backpropagation.
  • Spatially Adaptive Computation Time: SACT applies ACT independently at each spatial position, allowing per-position computation and learning to focus computation on regions of interest.When the spatial residual contribution f_W^l is zero, all positions share halting scores and SACT reduces to ACT.

3. Related work

Prior work on computational efficiency primarily uses static architectural techniques, while newer methods vary computation across images, spatial regions, or network components. The proposed approach is closest to BranchyNet but differs in applicability and preliminary comparisons with ACT-based block selection.

  • Static computation: Static efficiency techniques include convolution-kernel decomposition and connection pruning, with factorized kernels used in Inception and ResNet.These methods focus on fixed computational structures rather than dynamically varying computation.
  • Dynamic computation: Recent approaches vary computation through cascaded rejection, image-level dynamic capacity, spatially varying PerforatedCNNs, or fully connected networks.Cascaded classifiers quickly reject easy negative proposals in object detection, while Dynamic Capacity Networks use the same computation for every image and a classification-specific heuristic.
  • BranchyNet: BranchyNet is the closest related approach, but it applies only to classification and halts evaluation when intermediate-classification entropy falls below a threshold.Preliminary ACT-based experiments found choosing the number of blocks inferior to using fewer units per block.

4. Experiments

Experiments show that SACT improves the computation–accuracy trade-off over ACT and non-adaptive ResNet baselines, particularly for higher-resolution images and object detection. Its ponder cost maps also focus computation on relevant regions and perform competitively as an unsupervised visual saliency model.

  • ImageNet classification: SACT achieves a better FLOPs-accuracy trade-off than ACT and improves accuracy on images with resolutions higher than those used for training.Figure 7 reports that SACT outperforms ACT and the corresponding baselines as the resolution difference increases.
  • ImageNet classification: The experiments use ResNet-101 as the basic ACT and SACT architecture, with negative halting-score bias initialization and two-stage training addressing dead residual units.The bias bl = −3 initially corresponds to using approximately 21 units, while network weights are initialized from pretrained ResNet-101 weights.
  • ImageNet classification: SACT typically focuses computation on image regions of interest, as shown by its ponder cost maps across residual blocks.The first block reacts to low-level features, while the last two blocks attempt to localize the object.
  • COCO object detection: SACT τ = 0.005 has slightly higher FLOPs than ResNet-50 and 2.1 points better mAP, while outperforming the originally reported ResNet-101 result of 27.2 mAP.In Faster R-CNN on COCO, SACT improves the FLOPs-mAP trade-off compared with a non-adaptive ResNet feature extractor.
  • Visual saliency: SACT ponder cost maps outperform the naïve center baseline and perform competitively with the state-of-the-art deep model DeepFix as predictors of human eye fixations.The models are reused from ImageNet and COCO without training on cat2000, whose ground truth records fixation positions from 24 human subjects viewing 4,000 images.

5. Conclusion

The paper presents an end-to-end trainable, deterministic Residual Network model with spatially varying computation time. It demonstrates effectiveness for image classification and object detection, while computation allocation correlates well with human eye fixations.

  • Model and contributions: The proposed Residual Network model uses spatially varying computation time and is end-to-end trainable and deterministic.It can also be viewed as a black-box feature extractor.
  • Model and contributions: On the cat2000 saliency dataset, SACT performs surprisingly well on out-of-domain images such as art and fractals.The figure presents a SACT ponder cost map from the COCO model with τ = 0.005 and postprocessing.
  • Model and contributions: The model is effective for image classification and object detection problems.
  • Model and contributions: Per-position computation correlates well with human eye fixation positions, suggesting that the model captures important parts of images.

A. Implementation details · A.1. Image classification (ImageNet)

The ImageNet classification experiments train ResNet, ACT, and SACT with shared optimization hyperparameters and evaluate them using a central crop. Training uses asynchronous SGD with specified regularization, batching, learning-rate scheduling, and batch-normalization settings, while augmentation follows Inception v3 procedures.

  • A.1. Image classification (ImageNet): ResNet, ACT, and SACT use the same hyperparameters during training.
  • A.1. Image classification (ImageNet): Training uses 50 workers running asynchronous SGD with momentum 0.9.
  • A.1. Image classification (ImageNet): The optimization uses weight decay 0.0001 and batch size 32.
  • A.1. Image classification (ImageNet): Training halts upon convergence after 150−160 epochs, starting with learning rate 0.05 and reducing it by 10 after every 30 epochs.
  • A.1. Image classification (ImageNet): Batch normalization uses epsilon 1e-5 and moving average decay 0.997.
  • A.1. Image classification (ImageNet): ImageNet training uses Inception v3 augmentation with horizontal flipping, scale, aspect-ratio, and color augmentation.
  • A.1. Image classification (ImageNet): For evaluation, the method takes one central crop covering 87.5% of the original image area and resizes it to the target resolution.

A.2. Object detection (COCO)

For COCO object detection, ResNet and SACT use matched hyperparameters with resized images and horizontal-flip augmentation. Training uses asynchronous momentum SGD, while Faster R-CNN follows the original setup with specified anchor configurations.

  • Implementation: ResNet and SACT use the same hyperparameters, with images upscaled so their smaller side is at least 600 pixels.Random horizontal flipping is used for augmentation, and atrous convolution is not employed.
  • Implementation: Training uses 9-worker asynchronous SGD with momentum 0.9, batch size 1, and 1.2 million total iterations.The learning rate starts at 0.0003 and decreases tenfold after the 800 thousandth and 1 millionth iterations; ImageNet batch-normalization parameters remain fixed.
  • Implementation: Faster R-CNN anchors are sampled from a regular grid with stride 16 and sizes {64, 128, 256, 512} across aspect ratios {0.5, 1, 2}.Other hyperparameters closely follow those recommended in the original Faster R-CNN paper, apart from the training method and an additional anchor size.

A.3. Visual saliency (cat2000)

The visual saliency experiments postprocess ponder cost maps by normalizing and blurring them, then adding a weighted centered-Gaussian baseline. The procedure uses Gaussian-filter scale s and baseline weight γ, tuned by grid search, with s = 10 and γ = 0.005 in both models.

  • The normalized ponder cost map is blurred by convolving it with a Gaussian filter.The filter has standard deviation s.
  • Ponder cost maps are first normalized to the [0, 1] range.
  • A baseline map is formed by rescaling a reference centered Gaussian3 to H × W resolution and weighting it by γ > 0.
  • The final postprocessed map combines the normalized blurred ponder cost map with the weighted center baseline map.
  • The map depends on Gaussian-filter standard deviation s and baseline weight γ, which are tuned by grid search.The experiments use s = 10 and γ = 0.005 for both models.

B. Additional ImageNet results

This section extends ImageNet validation results for ACT, SACT, and ResNet models, including residual units per block. It also examines SACT training at 248 × 248 resolution instead of 224 × 224.

  • Results comparison: Extended ImageNet validation results compare ACT, SACT, ResNet, and their respective baselines, including residual units per block.Table 3 reports network FLOPs, residual units, Accuracy, and Recall@5; x ± y denotes the mean and one standard deviation.
  • Training resolution: SACT was additionally trained at 248 × 248 resolution, the highest resolution fitting GPU memory with batch size 32.This experiment used a higher training resolution than the 224 × 224 models.
  • Training resolution: All ImageNet models are trained with 224 × 224 resolution images, although their fully convolutional designs support arbitrary test-time resolutions.The passage notes that higher training resolution can improve model quality but requires longer training and more GPU memory.
Loading 1612.02297v2…