Source-linked AI summary
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification
Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, Gao Huang
TL;DR
High-resolution images improve CNN accuracy but impose substantial computation and memory costs. GFNet processes dynamically selected small inputs with reinforcement learning and terminates inference when confidence is sufficient. On ImageNet, it improves computational efficiency across modern lightweight CNNs and reduces MobileNet-V3 latency on an iPhone XS Max without sacrificing accuracy.
Problem
High-resolution CNN inputs improve accuracy but create high computational and memory costs, motivating reduction of spatial redundancy in image classification.
Method
GFNet sequentially processes a downsampled image followed by dynamically selected high-resolution patches, using separate encoders and a learned proposal policy with confidence-based termination.
Results
GFNet consistently improves computational efficiency across modern CNNs on ImageNet, including a reported 20% reduction in MobileNet-V3 average latency on an iPhone XS Max without sacrificing accuracy.
Takeaways & Limitations
GFNet provides a general, flexible framework for adaptive image classification that can use lightweight CNN backbones and adjust computation online without additional training.
Takeaways & Limitations
GFNet retains common CNN safety risks from adversarial attacks and privacy risks, and improper use may reduce the cost of criminal behaviors.
Abstract
from arXiv · showhide
The accuracy of deep convolutional neural networks (CNNs) generally improves when fueled with high resolution images. However, this often comes at a high computational cost and high memory footprint. Inspired by the fact that not all regions in an image are task-relevant, we propose a novel framework that performs efficient image classification by processing a sequence of relatively small inputs, which are strategically selected from the original image with reinforcement learning. Such a dynamic decision process naturally facilitates adaptive inference at test time, i.e., it can be terminated once the model is sufficiently confident about its prediction and thus avoids further redundant computation. Notably, our framework is general and flexible as it is compatible with most of the state-of-the-art light-weighted CNNs (such as MobileNets, EfficientNets and RegNets), which can be conveniently deployed as the backbone feature extractor. Experiments on ImageNet show that our method consistently improves the computational efficiency of a wide variety of deep models. For example, it further reduces the average latency of the highly efficient MobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and pre-trained models are available at https://github.com/blackfeather-wang/GFNet-Pytorch.
1 Introduction
High-resolution inputs improve CNN accuracy but incur substantial computational and memory costs. GFNet addresses spatial redundancy by selecting discriminative regions dynamically and adapting computation to each image.
- Motivation: High-resolution CNN inputs improve accuracy but increase computation and memory quadratically with image height or width.These costs translate into latency and power consumption in applications such as image search and autonomous vehicles.
- Motivation: GFNet targets spatial redundancy by processing only dynamically identified class-discriminative regions rather than the entire high-resolution image.Such regions can include features like a dog’s head or a bird’s wings and require fewer computational resources than the whole image.
- Challenges: The framework addresses both efficient region identification and adaptive computation allocation across images with differing discriminative-region requirements.The number and size of useful regions can vary from one input to another.
- Approach: GFNet uses a sequential glance-and-focus process in which each small input yields a prediction and proposes the next region.The initial glance processes a downsampled full image, while later focus steps process selected patches.
- Scope: GFNet is a general framework compatible with lightweight CNN backbones and emphasizes adaptive inference rather than fixed sequence lengths.Its classifier and region proposal network are independent modules.
- Practical benefits: GFNet can reduce memory use and adjust computational cost online by changing its termination criterion without additional training.The stated goal is flexible use of available resources or achieving required performance with minimal power consumption.
2 Related Work
Prior efficiency methods reduce computation through architectural changes, selective execution, or spatial processing. GFNet differs by combining CNN-based visual attention with adaptive, input-conditioned recurrent processing.
- Computationally efficient networks: Efficient CNN research includes lightweight architectures, pruning, quantization, and knowledge distillation to reduce inference cost.These approaches primarily target network structure or parameters rather than sequentially selecting image regions.
- Adaptive computation: Adaptive architecture methods alter model computation through multi-scale classifiers, model cascades, selective ensembles, or skipped layers and channels.They allocate different network structures or subsets to inputs of varying difficulty.
- Spatial redundancy: Spatial-redundancy methods reduce computation by lowering feature resolution, adjusting execution across regions, or skipping less important feature-map regions.These methods operate mainly within intermediate feature maps.
- Visual attention: Visual attention models direct processing toward relevant image regions, especially in image captioning and visual question answering.GFNet applies a related attention principle to image recognition.
- Comparison with recurrent attention: Compared with recurrent visual attention, GFNet uses a flexible CNN-based framework and can terminate its recurrent process conditionally for adaptive inference.The paper reports improved theoretical computational efficiency and actual inference speed on ImageNet.
3 Method
GFNet sequentially classifies smaller inputs while selecting further patches from the original image. Its separate encoders, recurrent classifier, policy-gradient proposal network, and staged training support adaptive inference.
- 3.1 Overview: GFNet processes a sequence of smaller inputs, beginning with a resized image and continuing with dynamically cropped patches.Patch locations depend on information from previous inputs.
- 3.1 Overview: The glance step uses global information from a downsampled image, while focus steps process high-resolution local patches selected for their discriminative value.Each step produces a prediction and proposes the next patch, with processing stopping when confidence is sufficient.
- 3.1 Overview: At inference, GFNet halts when the maximum softmax probability exceeds the step-specific threshold; otherwise it crops and processes another patch.This makes sequence length depend on the confidence obtained for each input image.
- 3.2 The GFNet Architecture: The architecture contains global and local CNN encoders, a classifier, and a patch proposal network.The two encoders share an architecture but use different parameters because low-resolution and high-resolution inputs exhibit a discrepancy that degrades single-encoder performance.
- 3.2 The GFNet Architecture: The recurrent classifier aggregates information from prior inputs and produces a prediction at every step, using encoded features and its hidden state.Feature maps need not be retained for classification because classification generally does not rely on their spatial information.
- 3.3 Patch Proposal Network: The recurrent proposal network uses unpooled feature maps to retain spatial information, reduces channels with a 1×1 convolution, and outputs stochastic patch-location actions during training.At test time, the policy mean determines locations deterministically.
- 3.3 Patch Proposal Network: The proposal policy receives reward equal to the increment in ground-truth softmax probability and maximizes discounted rewards.This trains it to select regions that improve confident classification with as few patches as possible.
- 3.4 Training: GFNet training proceeds in three stages: classification with random crops, policy-gradient training of the proposal network, and final classifier-and-encoder fine-tuning.The first two stages are indispensable, while the third is intended to further improve performance.
4 Experiments
Experiments on ImageNet evaluate GFNet across adaptive-inference settings, efficient CNN backbones, patch sizes, and ablations. GFNet improves accuracy–computation or latency trade-offs, while visualization and ablations clarify its behavior and design choices.
- 4.1 Main Results: ImageNet experiments use budgeted batch classification and anytime prediction across MobileNet-V3, RegNet-Y, EfficientNet, ResNet, and DenseNet backbones.Confidence thresholds support adaptive inference in both settings.
- 4.1 Main Results: At 7 × 10^7 Multiply-Adds, MobileNet-V3 GFNet reaches ∼71% Top-1 validation accuracy, outperforming vanilla MobileNet-V3 by ∼2%.GFNet also uses ∼1.4× less computation with EfficientNets and approximately 2−3× fewer Multiply-Adds with ResNets and DenseNets at matched accuracy.
- 4.1 Main Results: GFNet achieves ∼4−10% higher accuracy than MSDNet in anytime prediction when the budget ranges from 5 × 10^8 to 2.2 × 10^9 Multiply-Adds.The comparison uses a DenseNet-121-based GFNet.
- 4.1 Main Results: On an iPhone XS Max, GFNet reduces latency by 22% for MobileNet-V3-Large at 75.4% test accuracy, from 16.3ms to 12.7ms.For ResNets, it generally requires 2−3× lower latency than baselines at the same performance.
- 4.1 Main Results: Larger patches perform better at large computational budgets but worse than smaller patches when the budget is insufficient, while changing T has little effect at equal computation.These trends are measured under anytime prediction with ResNet-50 backbones.
- 4.2 Visualization: Images with large prototypical objects often classify correctly at the Glance Step, whereas harder images progressively improve through selected class-discriminative patches.Figure 7 visualizes patch locations, prediction correctness, and confidence across steps.
- 4.2 Visualization: Allowing images to exit later in the Focus Stage significantly improves performance by adjusting confidence thresholds online without additional training.The trade-off is examined through expected input-sequence length E(t) in budgeted batch classification.
- 4.3 Ablation Study: The learned patch-selection policy consistently outperforms random and centre-corner policies, while resizing the Glance input and using two encoders are important for early-step accuracy.The ablation study reports fixed-length input-sequence results and finds that L′_cls helps when t is large.
5 Conclusion
GFNet reduces spatial redundancy by sequentially processing smaller image inputs, progressively classifying and localizing discriminative regions until sufficient confidence enables adaptive termination.
- GFNet processes high-resolution images sequentially using down-sampled inputs or cropped patches.Each step performs classification and localizes a discriminative region for the next step.
- The process terminates once classification confidence is sufficient, enabling adaptive inference.
- GFNet is compatible with a wide variety of modern CNNs and is easy to implement on mobile devices.
- Extensive ImageNet experiments show that GFNet significantly improves the computational efficiency of state-of-the-art lightweight CNNs theoretically and empirically.
Broader Impact
GFNet is presented as a resource-efficient image-classification framework with potential benefits for high-volume services, mobile and edge devices, and environmental protection, while retaining CNN-related safety and privacy risks.
- GFNet may reduce computational resources for image-classification applications processing large data volumes on limited hardware.The paper mentions search engines, social media companies, and online advertising agencies as examples.
- On mobile phones and edge devices, GFNet may reduce latency and computation, potentially improving user experience and preserving battery life.
- The method may decrease power consumption, which the paper connects to environmental protection.
- GFNet may motivate research on efficient CNNs and could potentially be modified for semantic segmentation, object detection, and instance segmentation.
- GFNet retains common CNN risks, including potential adversarial-attack safety risks and privacy risks.
- Improper use of GFNet may reduce the cost of criminal behaviors.
A Implementation Details
The implementation uses recurrent components for patch proposal and classification, with a computationally cheaper cascade of fully connected classifiers replacing a large GRU for selected lightweight CNN backbones.
- For RegNets, MobileNets-V3, and EfficientNets, the patch proposal network uses a GRU with 256 hidden units.
- For ResNets, DenseNets, and RegNets, the recurrent classifier uses a GRU with 1024 hidden units.
- For MobileNets-V3 and EfficientNets, a cascade of fully connected classification layers replaces the computationally expensive large GRU classifier.
- At step t, the lightweight classifier concatenates previous feature vectors and applies a linear classifier sized tF × C.
A.2 Policy Gradient Algorithm
The patch proposal policy is trained with discounted rewards and PPO, using stochastic localization during training and deterministic mean actions at test time, with clipped updates and auxiliary value modeling.
- The patch proposal network maximizes the sum of discounted rewards for localization actions.The discount factor satisfies γ ∈ (0, 1), and T is the maximum input-sequence length.
- The policy stochastically selects normalized patch-center coordinates during training and uses the distribution mean for deterministic inference.
- PPO is used to train the patch proposal network through a surrogate objective based on the policy ratio and advantage estimator.
- A clipped surrogate objective limits policy updates because directly maximizing the unclipped objective can produce excessively large updates.
- The training objective includes an entropy bonus for exploration and a squared-error loss for the estimated state value.
- The implementation optimizes the final objective with Adam using γ = 0.7, ϵ = 0.2, c1 = 0.5, and c2 = 0.01.
A.3 Training Details
Training uses staged optimization for the encoders, classifier, and patch proposal network, with model-specific initialization and configurations summarized for the evaluated GFNets.
- The global encoder is initialized by fine-tuning pretrained models on images resized to H′×W ′, while the local encoder uses ImageNet-pretrained models.
- Stage I: Stage I trains the networks with SGD, cosine learning-rate annealing, Nesterov momentum of 0.9, batch size 256, and weight decay.
- Stage II: Stage II trains the patch proposal network π with Adam, sampling localization actions from a Gaussian distribution with standard deviation 0.1.
- Stage III: Stage III uses Stage I hyperparameters with a classifier learning rate of 0.01, but is omitted for EfficientNets because it did not improve performance.
- The GFNets use square inputs with H′ = W ′, and their input sizes, maximum sequence lengths, and encoders are summarized in Table 2.
- Figure 9 reports budgeted batch-classification results for H′×W ′ fine-tuned models, whose Glance Step performance is mainly determined by low-resolution fine-tuning.
B.1 Effects of Low-resolution Fine-tuning
Low-resolution fine-tuning improves computational efficiency independently, contributes to Glance Step performance, and leaves the Focus Stage to add accuracy and online cost adjustment.
- Low-resolution fine-tuning improves computational efficiency by itself and its gains are included in GFNet through the Glance Step.
- The Focus Stage further improves test accuracy while allowing GFNet to adjust average computational cost online.
B.2 Comparisons with MSDNet in Budgeted Batch Classification
The paper compares DenseNet-based GFNets with MSDNets under Budgeted Batch Classification, using a held-out validation set to estimate confidence thresholds.
- DenseNet-based GFNets consistently outperform MSDNets across a wide range of computational budgets.
- The comparison holds out 50,000 training images for confidence-threshold estimation and trains on the remaining samples.