Source-linked AI summary
Deep Networks with Internal Selective Attention through Feedback Connections
Marijn Stollenga, Jonathan Masci, Faustino Gomez, Juergen Schmidhuber
TL;DR
Traditional CNNs are stationary and feedforward, motivating a model that can adapt its behavior during evaluation and decide how to focus internal features. DasNet adds feedback connections and trains their control policy with SNES, enabling iterative filter reweighting across image passes. The resulting network corrects misclassifications from a trained feedforward Maxout network and achieves state-of-the-art results.
Problem
CNNs do not change their parameters during evaluation or use feedback from higher to lower layers, motivating a non-stationary CNN with a mechanism for adaptive behavior.
Method
DasNet uses learned bottom-up and top-down feedback to reweight convolutional filters over successive passes, with its million-parameter attentional policy evolved using SNES.
Results
DasNet corrects misclassifications from a fully trained feedforward Maxout network and achieves state-of-the-art results, including a 6% relative improvement over a vanilla CNN.
Takeaways & Limitations
Sequential internal attention lets dasNet alter filter importance in hindsight and correct an initial classification guess through additional internal processing.
Takeaways & Limitations
Future work will consider more complex actions that spatially focus on or alter parts of observed images.
Abstract
from arXiv · showhide
Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.
1 Introduction
CNNs achieve strong recognition performance through stationary, feedforward processing, whereas expert visual recognition can involve repeated evaluations that seek different information. The paper therefore proposes adaptive, post-training CNN behavior and a process for deciding how to adapt it.
- Motivation: Traditional CNNs use stacked feedforward layers that learn progressively abstract representations and have achieved state-of-the-art results across several vision tasks.The passage names object recognition, segmentation/detection, and scene parsing.
- Motivation: Classifying visually similar bird species can require more than a few milliseconds, implying several feedforward evaluations that elicit different image information.
- Research gap: The paper hypothesizes that CNNs can benefit from sequential processing, requiring a non-stationary CNN and a mechanism that decides how its behavior adapts after training.
- Proposed approach: DasNet uses bottom-up and top-down connections to modulate convolutional-filter activity across successive image passes, with a reinforcement-learned control policy.The policy can enhance or suppress features over multiple passes, especially for difficult cases missed by initial supervised training.
- Training challenge: The attentional policy is evolved with SNES because controlling typical image-classification CNNs requires over 1 million parameters.Experiments on CIFAR-10 and CIFAR-100 report improved correction of difficult classification instances over a previous state-of-the-art method.
2 Maxout Networks
The paper uses Maxout networks with dropout as dasNet’s underlying CNN model. Its layers combine convolution, Maxout pooling, and final classification to transform feature maps into class probabilities.
- Architecture: DasNet is built on Maxout networks combined with dropout, using alternating convolutional and Maxout layers followed by a classification layer.The passage describes Maxout as strong for object recognition across various tasks.
- Convolutional Layer: A convolutional layer maps c input feature maps to c′ output maps using c · c′ filters of size k × k.The input and output tensors have spatial dimensions m × n and m′ × n′, respectively.
- Convolutional Layer: The convolution operation applies an element-wise nonlinearity, with output size determined by kernel size and stride.
- Pooling Layer: Maxout pooling reduces every b consecutive maps to one map by retaining the maximum value at each pixel position.This changes c input maps into c′ = c/b output maps.
- Classification Layer: The final classification stage flattens the last pooling output into a vector, projects it with F, and applies softmax to produce class probabilities.The projection is followed by a Maxout reduction similar to pooling.
3 Reinforcement Learning
Reinforcement learning frames dasNet’s feedback control as sequential decision-making driven by reward. Because its continuous policy has nearly a million parameters, the paper evolves it with SNES rather than standard reinforcement-learning methods.
- RL framework: Reinforcement learning learns sequential decisions that maximize an external reward signal through an agent’s interaction with an environment.
- RL framework: At each time step, an agent observes the environment and selects an action according to a policy mapping observations to actions.
- RL objective: The agent transitions to a new state and receives a reward, while the policy objective maximizes expected future discounted reward.The discount factor γ controls how much future rewards influence the objective.
- DasNet policy: In dasNet, observations and actions are real-valued, and the policy parameter vector contains close to a million weights because both spaces approach a thousand dimensions.
- SNES optimization: SNES evolves the policy using parameterized search distributions and diagonal covariance entries, trading theoretical power for substantially greater efficiency than standard NES.
- DasNet policy: The policy is deterministic in this implementation, producing the same action for a given observation, although stochastic policies could be used.
4 Deep Attention Selective Networks (dasNet)
dasNet repeatedly processes the same image, using a learned policy to reweight convolutional feature maps based on network observations. The policy is evolved with SNES to focus attention on discriminative features and score the final classification.
- Architecture: dasNet weights convolutional feature maps differently across successive passes, allowing the network to iteratively focus attention on discriminative features.The action changes each map’s activation strength before maxout pooling, altering the CNN’s output even when the image is unchanged.
- Architecture: The policy observes map activations, intermediate classification activations, and the class probability vector before selecting the next action.The observation includes average activations from every output map, intermediate classification-layer activations, and the class probability vector.
- Policy: The deterministic policy maps observations to actions through a scaled softmax whose action elements average to 1, keeping network outputs positive and filter activations stable.The policy uses a weight matrix θ and scales the softmax by the action-space dimensionality.
- Policy training: The weighted loss assigns greater weight to misclassified samples, focusing policy evolution on difficult images, while an L2 term regularizes the policy.The paper specifies λ_misclassified > λ_correct and includes λ_L2 as a regularization parameter.
- Sequential processing: Each image is processed for T passes, with the final Maxout output used to classify it after repeated policy-controlled reweighting.The first pass uses unit filter weights so the network initially behaves normally; later passes use the selected weighting.
- Policy training: Because the policy has close to a million parameters, dasNet evolves it with Separable Natural Evolution Strategies rather than conventional single-agent reinforcement learning.SNES repeatedly updates its search distribution using sampled fitness values until the stopping criterion is met.
5 Related Work
Related work connects dasNet’s recurrent attention to biological visual processing and prior computer-vision methods using feedback, reinforcement learning, and iterative image processing.
- Biological vision: Visual cortex contains extensive bidirectional connections, with top-down links generally more numerous and diffuse than bottom-up links.A macaque visual-cortex hierarchy identified connections among about 40% of area pairs.
- Biological vision: Response-latency evidence supports fast feedforward processing followed by an attentional phase driven by recurrent processing.The pop-out effect is established for basic features but remains controversial for categorical stimuli and faces.
- Biological vision: Top-down projections can provide an initial category expectation, while recurrent feedback and competitive inhibition may make object recognition more robust.These mechanisms are described as influences on recognition rather than as a complete computational model of it.
- Computer vision: Computer-vision research has applied reinforcement learning to saccades, feedback to lower levels, selective attention, face recognition, and object recognition.Related studies also combine reinforcement learning with traditional vision primitives.
- Computer vision: Iterative recurrent processing has also been used for image reconstruction and face localization.
6 Experiments on CIFAR-10/100
Experiments on CIFAR-10 and CIFAR-100 evaluate dasNet on ambiguous classifications, its iterative dynamics, and whether its learned gates encode useful classification information. DasNet improves over the vanilla CNN and can correct difficult initial predictions through selective filter emphasis.
- Experimental setting: DasNet was evaluated on ambiguous CIFAR-10 and CIFAR-100 cases where classes sharing features are often confused.The evaluation tests whether feedback can fix erroneous predictions without disrupting the trained model.
- Classification results: 6% relative improvement over the vanilla CNN was reported in Table 1.The paper describes this as a new state-of-the-art result for the challenging dataset.
- Iterative correction: The network initially classified a cat as a dog, then sharply increased the cat probability after feedback.The example shows lower-layer filters changing emphasis, including greater focus on surroundings and reduced emphasis on eyes.
- Internal attention: Higher-layer alterations produced complex emphasis and pattern suppression rather than uniform increases or decreases across output maps.The third convolutional layer showed the most influential changes because it is closest to the final output layers.
- Dynamics: DasNet performance peaked at the training step count and declined without exploding when evaluated for up to 10 steps.This stability was observed for a small two-layer network trained with T = 1 and T = 2.
- Gate information: Final gate values yielded 40.70% accuracy with 15-nearest neighbour classification and 45.74% with logistic regression.Their similarity to dasNet performance supports that the gates contain classification-relevant information.
7 Conclusion
DasNet adds learned feedback connections to direct selective internal attention after an initial feedforward classification. The network learned to correct misclassifications and achieved state-of-the-art results, while future work considers spatially focused actions.
- Conclusion: DasNet uses reinforcement-learned feedback connections to direct selective internal attention to image features.After the initial feedforward classification, feedback alters the importance of selected filters during additional processing.
- Conclusion: DasNet corrected misclassifications produced by a fully trained feedforward Maxout network.Its active selective attention was associated with state-of-the-art results.
- Future work: Future research will consider actions that spatially focus on or alter parts of observed images.The stated future direction extends attention beyond the current feature-level alterations.