Source-linked AI summary
Dynamic Computational Time for Visual Attention
Zhichao Li, Yi Yang, Xiao Liu, Feng Zhou, Shilei Wen, Wei Xu
TL;DR
RAM typically uses a fixed number of attention steps, although image difficulty can vary. DT-RAM adds a per-step continue/stop action and learns attention and stopping policies with reinforcement learning; on fine-grained recognition, it reports state-of-the-art performance with less average computation than RAM.
Problem
RAM uses a fixed number of attention steps even though image content and difficulty can call for different processing times.
Source: §1 · p. 1Method
DT-RAM adds a binary continue/stop action at each time step and uses reinforcement learning to learn attention and stopping policies.
Source: §1 · pp. 1–2, §3.3 · p. 4Results
DT-RAM achieves state-of-the-art fine-grained recognition while using 1.9 steps versus 3 steps on average for RAM on CUB-200-2011.
Source: §1 · p. 2, §4.3 · p. 6Takeaways & Limitations
The authors conclude that dynamically adjusting computational time during inference is effective for fine-grained recognition and may matter for dynamic reasoning in computer vision.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We propose a dynamic computational time model to accelerate the average processing time for recurrent visual attention (RAM). Rather than attention with a fixed number of steps for each input image, the model learns to decide when to stop on the fly. To achieve this, we add an additional continue/stop action per time step to RAM and use reinforcement learning to learn both the optimal attention policy and stopping policy. The modification is simple but could dramatically save the average computational time while keeping the same recognition performance as RAM. Experimental results on CUB-200-2011 and Stanford Cars dataset demonstrate the dynamic computational model can work effectively for fine-grained image recognition.The source code of this paper can be obtained from https://github.com/baidu-research/DT-RAM
1. Introduction
RAM uses recurrent attention to process image regions, while DT-RAM adds an on-the-fly stopping decision so computation can vary by image. The paper addresses training this policy and reports state-of-the-art fine-grained recognition with less average computation than RAM.
1. Introduction
- Different image content can call for different recognition times, such as when clutter, occlusion, or object scale makes locating a bird and extracting its parts harder.The paper contrasts a clearly visible bird with camouflaged or cluttered scenes with pose variation.Source: §1 · p. 1
- DT-RAM adds a binary continue/stop action at each time step, allowing it to stop computation and output a classification score.This is intended to reduce redundant computation, especially for easy-to-recognize examples.Source: §1 · pp. 1–2
- Training DT-RAM from scratch is difficult as longer sequences increase delayed-reward and gradient-variance problems for REINFORCE.The authors use curriculum learning to train RAM with increasing step counts, initialize DT-RAM from pretrained RAM, and fine-tune with REINFORCE; intermediate supervision is crucial, particularly for longer sequences.Source: §1 · p. 2
- On public benchmark datasets including CUB-200-2011 and Stanford Cars, DT-RAM achieves state-of-the-art performance on fine-grained image recognition and uses less average computational time than RAM, better fitting devices with computational limitations.Source: §1 · p. 2
2. Related Work
Related work situates DT-RAM among visual attention, feedback networks, adaptive-computation methods, and fine-grained recognition approaches. It highlights fixed or rule-based stopping in prior work and asks whether carefully trained attention can remain competitive for fine-grained recognition.
2.1. Visual Attention Models
- Attention models include soft, differentiable spatial or temporal weighting and hard attention, which treats attention as discrete actions and uses policy gradients such as REINFORCE.The paper places DT-RAM in the hard-attention group because its stopping action is discrete.Source: §2.1 · p. 2
2.2. Feedback Neural Networks
- Feedback neural networks use previously computed high-level features to refine low-level features, but prior examples fix recurrent steps or use simple early-stopping rules [67].Source: §2.2 · p. 2
2.3. Dynamic Computational Time
- Earlier adaptive-computation methods vary inference time but use fixed halting policies [26]; prior reinforcement-learning graph adaptation was tested on small datasets [51], whereas this work compares on larger natural fine-grained images.Source: §2.3 · pp. 2–3
2.4. Fine-Grained Recognition
- Fine-grained recognition work spans representation learning, part alignment, and data scale, while carefully trained attention can still achieve performance comparable to state-of-the-art methods.Source: §2.4 · p. 3
3. Model
The model makes its computational structure input-dependent, extending recurrent visual attention with a learned stop/continue decision. Its training combines sampled-structure optimization with curriculum learning, discounted rewards, and supervision at intermediate steps.
3.1. Learning with Dynamic Structure
- Directly optimizing over structures is difficult because it requires evaluating exponentially many possibilities, so training samples structures and approximates gradients with Monte Carlo.The gradient combines a REINFORCE term, which makes lower-loss structures more probable, with the standard fixed-structure neural-network gradient.Source: §3.1 · p. 3
3.2. Recurrent Attention Model (RAM)
- RAM treats attention as a partially observed decision process, cropping local image regions and using recurrent state to guide attention and classify.Its localization network models the attention policy, while a separate classification network computes scores from the recurrent state.Source: §3.2 · pp. 3–4
3.3. Dynamic Computational Time for Recurrent Attention (DT-RAM)
- DT-RAM adds a binary stopping action at every time step, sampling attention locations and stop decisions through separate policies during inference.The action encodes continue as 0 and stop as 1; the model is expected to stop earlier on easy images and continue searching on hard ones.Source: §3.3 · p. 4
3.4. Training
- Training uses discounted cumulative reward to trade off correct classification against taking more attentions, with reward delayed until a correct final prediction.Intermediate classification supervision is applied at every time step, and each training example can have a different stopping time.Source: §3.4 · pp. 4–5
- Training progresses from an ImageNet-pretrained base network to fine-tuning on the fine-grained dataset, RAM with increasing steps, and then DT-RAM fine-tuned with REINFORCE.This curriculum addresses the difficulty of training from scratch as longer sequences increase delayed-reward and gradient-variance problems.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate DT-RAM on MNIST and two fine-grained datasets, comparing recognition performance and average computation with RAM and other methods. Ablations examine dataset conditions, model settings, stopping policies, and training choices.
4.1. Dataset
- The benchmarks are MNIST, CUB-200-2011, and Stanford Cars; the fine-grained datasets contain cluttered images, and their available box annotations are not used for training or testing.CUB-200-2011 has 11,778 images across 200 bird categories; Stanford Cars has 16,185 images across 196 car classes.Source: §4.1 · p. 5
4.2. Implementation Details
- For fine-grained experiments, images are resized to 512×512, with a 448×448 first crop and 224×224 crops thereafter; testing selects each network’s highest-probability action.The setup uses ImageNet-pretrained ResNets and does not use the datasets’ box or part annotations.Source: §4.2 · p. 5
4.3. Comparison with State-of-the-Art
- On MNIST, DT-RAM-1 achieves 1.46% recognition error at 3.6 average steps versus RAM’s 1.54% at 4 steps, while DT-RAM-2 achieves 1.12% at 5.2 steps versus RAM’s 1.34% at 5 steps.The step-count distributions show a trade-off: DT-RAM-1 uses less computation but has higher error than DT-RAM-2.Source: §4.3 · p. 6, Table 2 · p. 6, Figure 5 · p. 6
- On CUB-200-2011, DT-RAM matches RAM’s state-of-the-art performance using 1.9 steps on average versus 3; on Stanford Cars, it matches RAM’s 93.1% accuracy using 1.9 steps.The CUB result is reported alongside a 1.5% improvement of RAM over the ResNet baseline; on Cars, that improvement is 0.8%.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 7
4.4. Ablation Study
- On CUB-200-2011, 512×512 inputs with ResNet-50 achieve 84.5% accuracy versus 81.5% at 256×256, while ResNet-50 reaches 81.5% versus ResNet-34’s 79.9%.The authors also report 82.8% for ResNet-101 on 224×224 crops.Source: §4.4 · p. 7, Table 5 · p. 7
- On CUB-200-2011, DT-RAM reaches RAM’s 81.8% accuracy using 3.5 average steps versus 6, and its stopping distribution favors 1–2 or 5–6 steps.RAM’s accuracy rises from 79.9% to 81.8% as steps increase, then no longer improves after five steps.Source: §4.4 · p. 7, Table 6 · p. 7, Figure 6 · p. 7
- At 3.6 steps on CUB-200-2011, DT-RAM achieves 81.8% accuracy versus 81.3% for a fixed stopping policy based on a classification-probability threshold.The fixed policy is applied to six-step RAM.Source: §4.4 · p. 7, Table 7 · p. 7
- Intermediate supervision lets RAM models with more steps steadily improve final performance; curriculum learning stabilizes training and gradually increases testing performance over steps from 79.9% to 81.8%.Source: §4.4 · p. 8
5. Conclusion and Future Work
The paper presents reinforcement learning for dynamically adjusting computational time during inference, applies it to recurrent visual attention for fine-grained recognition, and identifies more sophisticated models and complex tasks as future directions.
5. Conclusion and Future Work
- The paper presents a reinforcement-learning method for dynamically adjusting computational time during inference.Source: §5 · p. 8
- The method is applied to recurrent visual attention, with effectiveness shown for fine-grained image recognition.Source: §5 · p. 8
- Future work will develop more sophisticated dynamic models and apply them to complex tasks such as visual question answering.Source: §5 · p. 8