Progressive Attention Networks for Visual Attribute Prediction

Introduction

• Attention can reduce computational cost by limiting hidden activations to attended regions [16], while query-specific attribute prediction requires identifying a particular image region.Source: §1 · pp. 1–2
• Soft attention [23] is differentiable and supports end-to-end training, but its fixed receptive-field size limits it to local regions and complicates handling objects with varied scales and shapes.Source: §1 · p. 2
• PAN applies attention across multiple CNN layers to refine target-object focus across scales and shapes, using hard attention with likelihood marginalization instead of feature aggregation.Figure 1 illustrates distractions at smaller scales being suppressed earlier and larger-scale distractions later, as receptive fields grow.Source: §1 · p. 2, Figure 1 · p. 2
• The paper identifies local context as a way to improve progressive attention stability and reports significant performance improvement over traditional single-step attention for query-specific visual attribute prediction.Source: §1 · p. 2

Attention on features

• Feature-based attention selects neural-network features, supporting translation [3] [15], speech [4], handwriting [8], memory access [9], captioning, and visual question answering [22] [23] [24].The paper describes this as the most straightforward attention mechanism: an attention model is attached to a neural-network architecture to select a subset of features.Source: §2 · pp. 2–3

Attention by image transformation

• Image-transformation methods predict regions using translation [2] [16], scale [10], or affine and other transforms, but STNs are limited to a single candidate region defined by few parameters.The proposed model instead progressively filters feature maps, without assuming roughly aligned objects or restricting attention to a constrained spatial transformation.Source: §2 · p. 3

Multiple attention processes

• Prior iterative methods condition image attention on previous states [24], retrieve memory values [9, 19], or stack STN layers; ours progressively predicts attention across CNN layers to capture target-object shapes.The paper shares the iterative focus of these approaches but operates over multiple CNN layers, deriving attention information progressively from feature maps.Source: §2 · p. 3

Training attention models

• Soft attention is end-to-end differentiable, whereas stochastic hard attention [23] [26] and transformation-based models [2] [16] typically use REINFORCE; our network uses standard backpropagation.STN offers a fully differentiable formulation; the proposed network requires no extra training technique.Source: §2 · p. 3

3.1 Progressive Attentive Process

• Across CNN layers, PAN estimates spatial attention and forwards each attended feature map to the next layer, then aggregates the final map for attribute classification.The model uses sigmoid probabilities independently at intermediate locations and softmax over the final spatial region for aggregation.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 Incorporating Hard Attention

• Hard attention preserves feature-map characteristics by marginalizing spatial-location predictions with attention probabilities, avoiding the REINFORCE technique used in [23].The soft-attention weighted sum can lose the feature map’s semantic layout; hard attention instead predicts answers from selected features and marginalizes their predictions.Source: §3.2 · p. 5

3.3 Multi-Resolution Attention Estimation

• Recursive attention over feature maps of differing resolutions lets PAN use deep semantics while attending at arbitrary spatial resolutions.Lower layers suppress features containing small distractors first; higher layers can then suppress larger distractors, using a feedforward inference procedure.Source: §3.3 · p. 5

3.4 Local Context

• Local context lets the attention network compare a feature with its spatial neighbors, improving attention quality and making predictions more discriminative.The context area is set by the composite convolution-and-pooling filter size in the next layer, and is unnecessary at the last attention layer.Source: §3.4 · p. 6

3.5 Training Progressive Attention Networks

• PAN’s soft- and hard-attention network is trained end-to-end with standard backpropagation to minimize cross-entropy for object-specific visual attributes.When initialized from a pretrained CNN, the CNN must be fine-tuned because intermediate attention maps can change the input distributions of its layers.Source: §3.5 · p. 6

4.1 MNIST Reference

• PAN variants outperform baselines on MREF color prediction, while progressive attention with local context and hard attention consistently achieves the best weighted mAP on Visual Genome.On MREF, PAN[∗]+CTX further improves performance, and PAN[H]+CTX performs best in most cases.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 9
• Progressive attention achieves the best MREF true-positive ratios, and adding hard attention and local context improves overall performance in most cases.TPR measures the share of total attention probability assigned to the ground-truth target area.Source: Table 1 · p. 8, §4.1 · p. 8
• On the MBG test subsets, progressive attention is robust to target-scale variation, especially with hard attention and local context; MREF and MDIST show similar trends.The test sets are divided into five subsets by target-object scale.Source: Table 1 · p. 8, §4.1 · p. 8

4.2 Attribute prediction on Visual Genome

• STNs show very low TPR because they fail to attend to target objects, yet achieve higher mAP than other baselines in Visual Genome’s attention-only setting.The authors attribute the higher mAP to object-class conditional priors encoded through the transformation process.Source: §4.1 · p. 8, §4.2 · p. 10
• On VOC 2007 train/validation images, progressive attention outperforms baselines in segmentation-mask TPR, with hard attention strengthening performance.Class names are used as queries, and attention is evaluated against the corresponding object segmentation masks.Source: Table 2 · p. 9, §4.2 · p. 10

4.1 MNIST Reference

• Qualitative results show progressive attention gradually suppresses irrelevant regions while retaining fine-resolution attention, whereas baselines attend only once at the top layer and produce coarser regions.This pattern is illustrated on MBG and Visual Genome examples.Source: Figure 6 · p. 8, §4.1 · p. 9, Figure 7 · p. 10

• Both soft- and hard-attention versions substantially outperform existing methods on synthetic and real datasets.Source: §5 · p. 10
• Progressive attention across CNN layers predicts regions of interest with variable sizes and shapes, while local context supports more robust attention estimation.Source: §5 · p. 10

A Network Architectures on Visual Genome

• PAN uses ImageNet-pretrained VGG-16 convolution and pooling layers, with attention layers att1, att2, and att3 stacked above pool3, pool4, and pool5, respectively.Figure 8 illustrates the network architecture.Source: A Network Architectures on Visual Genome · p. 13, Figure 8
• For att1 and att2, PAN combines the query, target feature, and local context to estimate attention scores, setting δ = 2 to match context size to the receptive fields of the next two convolution layers.Two convolutions encode the target feature and context, initialized with the corresponding CNN weights; their embedding is concatenated with the one-hot query and passed through two fully connected layers.Source: A Network Architectures on Visual Genome · p. 13
• PAN’s final attention layer, att3, estimates attention from the concatenated query and target feature; its attended feature is then classified to predict attributes.Figure 9 depicts the intermediate attention functions and the attention functions used in SAN, HAN, and PAN’s final layer.Source: A Network Architectures on Visual Genome · pp. 13–14, Figure 9 · p. 14
• SAN and HAN use the same VGG-16 CNN architecture as PAN, attaching soft attention and hard attention, respectively, at the top of the CNN.Their attention functions use two fully connected layers on the concatenated query and target feature, as in PAN’s att3.Source: A Network Architectures on Visual Genome · pp. 13–14
• Additional model variants fuse the query with PAN’s attended feature through a fully connected layer before classification, incorporating a target-class conditional prior for attribute prediction.The appendix motivates these variants by noting that the query can constrain possible attributes and filter irrelevant ones.Source: A Network Architectures on Visual Genome · p. 13

B More Qualitative Results on MBG

• Tables 3 and 4 show qualitative results for SAN, HAN, PAN[S]+CTX, and PAN[H]+CTX using queries '8' and '1', respectively; Table 3 includes attended-feature-map overlays and activation magnitudes.Source: Table 3 · p. 15, Table 4 · p. 16
• Table 3 visualizes images faded by attended feature maps and activation magnitudes before and after attention.Before-attention activations are mapped to original image space by spreading them across their receptive fields.Source: Table 3 · p. 15, B More Qualitative Results on MBG · p. 15
• For PAN[∗]+CTX, Table 3 shows only the last two attentions, which accumulate attention from earlier layers; each map is rescaled to [0,1].The rescaling uses (x−min)/(max−min).Source: B More Qualitative Results on MBG · p. 15

• Two progressive attention variants gradually attend to target objects at fine resolution on the Visual Genome dataset.Source: Table 5 · p. 17
• For PAN[∗]+CTX, Table 5 shows only the last two attentions, which accumulate attention from earlier layers.Source: Table 5 · p. 17
• Tables 6 and 7 provide additional attention visualizations.Source: Table 6 · p. 18, Table 7 · p. 19