Source-linked AI summary
Attentional Pooling for Action Recognition
Rohit Girdhar, Deva Ramanan
TL;DR
Action recognition must distinguish evidence distributed across people, objects, and context, while existing hard attention requires region or pose supervision. The paper replaces standard pooling with learned attention-weighted pooling, optionally guided by pose, and interprets it as low-rank second-order pooling. The resulting module improves recognition across still-image and video benchmarks with nearly unchanged network size and computation, including a reported 12.5% relative improvement on MPII.
Problem
Existing action-recognition models either process full images without selective focus or use hard-coded regions requiring object or pose labels, despite actions depending on varied context and interactions.
Method
The paper introduces a trainable attention-weighted pooling layer that learns bottom-up and top-down attention, optionally uses pose supervision, and is formulated as a low-rank approximation to second-order pooling.
Results
The module produces significant improvements over baseline architectures across three action-recognition benchmarks and establishes state of the art on MPII with a 12.5% relative improvement.
Takeaways & Limitations
Attention-weighted pooling is presented as an easy-to-implement alternative to standard pooling that learns where to look with few additional parameters and nearly negligible computation cost.
Takeaways & Limitations
Existing saliency-based methods were not promising for constructing a bottom-up-only model because class-agnostic saliency makes class-specific scores difficult to produce.
Abstract
from arXiv · showhide
We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks. Our proposed attention module can be trained with or without extra supervision, and gives a sizable boost in accuracy while keeping the network size and computational cost nearly the same. It leads to significant improvements over state of the art base architecture on three standard action recognition benchmarks across still images and videos, and establishes new state of the art on MPII dataset with 12.5% relative improvement. We also perform an extensive analysis of our attention module both empirically and analytically. In terms of the latter, we introduce a novel derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods (typically used for fine-grained classification). From this perspective, our attention formulation suggests a novel characterization of action recognition as a fine-grained recognition problem.
1 Introduction
The paper motivates learned soft attention for action recognition because relevant evidence may lie in people, objects, or context rather than a fixed region. It proposes an attention-weighted pooling modification that can improve recognition without extra supervision or substantial computational overhead.
- Background: Deep CNN representations have surpassed traditional detection-, pose-, trajectory-, and part-based approaches, with most video methods using discriminatively trained 2D CNNs.These models commonly process appearance frames and sometimes optical flow, aside from a few 3D-convolutional methods.
- Motivation: Hard-coded attention around pose keypoints or bounding boxes improves performance but requires object or pose labels or detectors.Such approaches also risk overlooking actions whose discriminative evidence lies in background context or interacted objects.
- Our Work: The proposed network modification learns image-specific attention maps that focus computation on task-relevant input regions without additional supervision.It extends average pooling into weighted average pooling and automatically improves the baseline architecture.
- Our Work: Pose supervision has little effect on image recognition but produces larger video gains because many video frames are non-iconic and lack centered action subjects or objects.The pose-guided variant encourages attention toward human-object interactions.
- Contributions: The study evaluates the module across three action-recognition datasets and analyzes its relationship to rank-1 approximations of bilinear pooling.The authors use this analysis to suggest viewing action recognition as a fine-grained recognition problem.
2 Related Work
Related work spans deep-learning approaches for still-image and video action recognition, hard attention using human or context regions, and soft spatio-temporal attention. The paper distinguishes its single-frame soft-attention design and connects it to efficient second-order pooling.
- Benchmarks and Features: Action-recognition benchmarks cover still images and videos, with image datasets such as HICO and MPII having more classes than common video benchmarks.The cited examples contain 600 and 393 image classes versus 101 and 51 video categories.
- Benchmarks and Features: Image recognition has benefited strongly from deep learning, while competitive video recognition still relies on handcrafted improved dense trajectory features.MPII performance is reported as rising from 5% mAP to 27% mAP with newer image-based methods.
- Hard Attention: Hard-attention methods use human, context, or pose-keypoint bounding boxes, but the proposed approach removes the need for bounding boxes.Prior examples include R*CNN with auxiliary context boxes and multiple-instance learning over humans and full-image context.
- Soft Attention: Earlier soft-attention methods predict spatial or temporal attention mainly in video settings, whereas this method predicts and applies spatial attention from a single frame.That design supports both still-image and video use cases.
- Second-Order Pooling: Using one appearance-feature set to predict and apply attention makes the output quadratic in the features, linking the method to efficient low-rank bilinear pooling.This connection motivates implementing attention through second-order pooling operations and their low-rank approximations.
3 Approach
The approach replaces standard CNN pooling with trainable attention-weighted pooling, derived as a low-rank approximation to second-order pooling. It separates bottom-up saliency from class-specific top-down attention and supports pose-regularized variants.
- Attentional pooling: Attentional pooling replaces a standard CNN pooling operation with a trainable attention-weighted pooling layer.The module is designed to plug into standard CNNs that end with average pooling.
- Low-rank second-order pooling: Rank-1 factorization W = ab^T converts second-order pooling into an attentional formulation using vectors a and b.The factorization yields an efficient low-rank approximation rather than explicitly computing the full second-order feature matrix.
- Efficient implementation: The module computes a bottom-up attention map h = Xb, uses it for a weighted feature average x = X^T h, and applies a linear classifier a^T x.This preserves the attention computation while producing the final classification score efficiently.
- Top-down attention: Class-specific top-down attention is separated from class-agnostic bottom-up saliency by sharing the bottom-up parameter vector b across classes.The resulting score is an inner product between class-specific top-down maps and bottom-up saliency maps.
- Network architecture: The network applies attention after the final spatial feature map, multiplying top-down and bottom-up maps before spatial averaging to produce class predictions.An alternative implementation first weights features by saliency and then applies a classifier.
- Pose-regularized attention: Pose-regularized attention predicts pose-keypoint heatmaps with an L2 loss while retaining an unconstrained bottom-up attention channel.The pose-guided architecture uses 17 output channels: 16 for keypoints and one for unconstrained attention.
4 Experiments
Experiments across MPII, HICO, and HMDB51 show that attentional pooling improves action and human-object interaction recognition across still images and video, with pose supervision especially useful for non-iconic video frames. The analyses also examine architecture dependence, bilinear-pooling alternatives, and overfitting from multiple attention maps.
- Datasets: The experiments cover MPII, HICO, and HMDB51 across still-image action classification, human-object interaction classification, and video recognition.MPII and HICO use mAP evaluation, while HMDB51 evaluates trimmed action classification.
- MPII: MPII attentional models outperform the baseline ResNet and prior methods, with pose-regularized attention performing best by a small margin.The pose-regularized model uses pose labels during training but does not require pose input at test time.
- HICO: 5% is the reported boost over the base ResNet model on HICO, where the model outperforms nearly all prior methods, including approaches using detection boxes.The exception is a method trained with a specialized weighted loss for HICO.
- Attention analysis: Attention peaks in HICO typically occur near the relevant object, suggesting the learned maps can provide weak supervision for object detection.The visualization crops patches around attention peaks for images containing object-specific interactions.
- HMDB51: Pose-based attention outperforms both the base network and TSN on HMDB51’s RGB-only single-frame setting, establishing a new state of the art.Pose regularization is more helpful for video because subjects can move across non-iconic frames.
- Diagnostics: A 393-map attention model reaches 27.9 mAP on MPII versus 30.3% with one map and 26.2% without attention, indicating overfitting from added parameters.Both multiple-map and one-map models achieve near-perfect training mAP, while the authors suggest multiple maps may suit larger datasets.
5 Discussion and Conclusion
The discussion connects attentional pooling to instance-level action recognition and second-order pooling, while concluding that the formulation is competitive, lightweight, and easy to integrate.
- Discussion: Unlike prior approaches, the model does not require explicit instance or bounding-box action modeling, allowing attention to objects outside person bounding boxes.The paper notes that instance-level recognition remains possible by applying attentional pooling over RoI features.
- Discussion: The formulation links attention with low-rank second-order pooling and suggests combining perspectives traditionally treated as distinct.
- Conclusion: The model achieves competitive or state-of-the-art results while learning where to look during feature pooling with few additional parameters.The authors present it as an attractive alternative to standard pooling because it is easy to implement and adds little model complexity.