Source-linked AI summary
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points
Fabien Baradel, Christian Wolf, Julien Mille, Graham W. Taylor
TL;DR
Human activity recognition often depends on pose or structured local representations, but pose may be unavailable and its suitability is uncertain. The paper instead learns unconstrained glimpse sequences from RGB, tracks them with recurrent workers through soft assignments and external memory, and reports state-of-the-art performance on NTU RGB-D and Northwestern-UCLA. The method uses pose only optionally during training, not testing.
Problem
Human activity recognition needs an alternative to pose-based representations because depth or articulated pose may be unavailable at test time and pose may not be the optimal intermediate representation.
Method
A visual attention process extracts unconstrained glimpse sequences, while recurrent tracking/recognition workers integrate them through soft assignments supported by external memory.
Results
The method achieves state-of-the-art performance on NTU RGB-D and Northwestern-UCLA, including comparisons against methods using pose, depth, or both at test time.
Takeaways & Limitations
Distributed recognition over an unstructured Glimpse Cloud outperforms a global convolutional model and simpler local glimpse-localization baselines in the reported experiments.
Takeaways & Limitations
The external memory has fixed size K and deletes the oldest slot when full; more flexible storage processes are left for future work.
Abstract
from arXiv · showhide
We propose a method for human activity recognition from RGB data that does not rely on any pose information during test time and does not explicitly calculate pose information internally. Instead, a visual attention module learns to predict glimpse sequences in each frame. These glimpses correspond to interest points in the scene that are relevant to the classified activities. No spatial coherence is forced on the glimpse locations, which gives the module liberty to explore different points at each frame and better optimize the process of scrutinizing visual information. Tracking and sequentially integrating this kind of unstructured data is a challenge, which we address by separating the set of glimpses from a set of recurrent tracking/recognition workers. These workers receive glimpses, jointly performing subsequent motion tracking and activity prediction. The glimpses are soft-assigned to the workers, optimizing coherence of the assignments in space, time and feature space using an external memory module. No hard decisions are taken, i.e. each glimpse point is assigned to all existing workers, albeit with different importance. Our methods outperform state-of-the-art methods on the largest human activity recognition dataset available to-date; NTU RGB+D Dataset, and on a smaller human action recognition dataset Northwestern-UCLA Multiview Action 3D Dataset. Our code is publicly available at https://github.com/fabienbaradel/glimpse_clouds.
1. Introduction
The paper proposes RGB-only-at-test human activity recognition using unconstrained visual glimpses, recurrent tracking/recognition workers, and soft assignments. It evaluates this approach on NTU RGB-D and Northwestern-UCLA, reporting strong benchmark performance.
- Motivation: The method recognizes activities from RGB at test time without requiring articulated pose, while optionally using pose during training.The paper motivates this setting because depth may be unavailable and pose may not be the optimal intermediate representation.
- Method: A visual attention process learns local glimpse representations and can vary its focus freely across frames without imposed spatio-temporal coherence.The glimpses form an unstructured collection of activity-relevant local entities rather than predefined body-joint subsets.
- Contributions: The approach models two attentional processes: extracting glimpses per frame and reasoning about entities over time.Feature-space attention is jointly calculated with a global model processing the full input image.
- Method: Recurrent tracking/recognition workers integrate the glimpse features over time, with each glimpse soft-assigned to all workers at different importance levels.External memory supports these assignments, and the number of glimpses can differ from the number of workers.
- Evaluation: The method outperforms the state of the art on NTU RGB-D and achieves state-of-the-art results on the Northwestern-UCLA Multiview Action 3D Dataset.NTU RGB-D is described as the largest available human activity dataset, while Northwestern-UCLA is characterized as smaller.
2. Related Work
The paper distinguishes its approach from prior attention, recurrent, memory, and tracking methods by using unconstrained glimpse clouds and learned soft assignments to recurrent workers. Unlike methods organized around pose parts or hand-crafted tracking graphs, it learns scene-part assignments automatically and does not require pose at test time.
- Pose and multimodal data: Unlike prior pose-dependent approaches, the method uses pose only during training for regularization and does not require it during testing.Its test-time input is therefore not dependent on articulated pose.
- Recurrent architectures: Unlike part-based recurrent models, the method soft-assigns scene parts across multiple recurrent workers, each of which can integrate all scene points.This contrasts with recurrent architectures whose cells are organized around anatomical parts or manually defined structures.
- Tracking and distributed recognition: The model learns assignments between glimpses and trackers automatically instead of relying on external trackers or manually created application-specific graphs.This distinguishes it from Structural RNN-style approaches that use hand-crafted graphs and separate tracking systems.
- Attention mechanisms: The approach uses hard attention for selecting input regions, in contrast to differentiable soft attention mechanisms that dynamically weight observation parts.Earlier hard-attention methods selected locations with recurrent mechanisms, while differentiable attention enabled gradient-based optimization.
- Attention and memory: The method produces an unstructured glimpse cloud in a spatio-temporal cube rather than imposing spatio-temporal coherence on attention locations.The external memory is trainable in reading and remembers past soft assignments of glimpses to recurrent workers.
3. Dynamic sequential attention
The model uses recurrent spatial attention to extract sequential local glimpses from RGB video, while recurrent workers process these features for activity recognition. Differentiable cropping and external-memory soft assignment support end-to-end learning and tracking of the unstructured glimpses.
- Feature representation: The model maps RGB video to a spatio-temporal feature map while retaining the original temporal scale for sequential processing.A 3D-convolutional network pools spatial dimensions but not time.
- Sequential attention: Each glimpse corresponds to a sub-region of the feature map, whose features are extracted using global average pooling.The glimpse region is specified by location and scale parameters.
- Sequential attention: A recurrent spatial attention model predicts a fixed-length sequence of glimpse locations and scales across video frames.Its hidden state persists across frames, supporting a coherent scrutinization process.
- Sequential attention: The recurrent attention model uses glimpse history, frame context, and high-level classification information when predicting subsequent glimpses.The recurrence runs over glimpses, while frame context indicates transitions and the classification state carries recognition information.
- Differentiable glimpse extraction: A differentiable spatial transformer crops, translates, and isotropically scales feature-map regions, allowing attention parameters to be trained with gradient descent.Bilinear interpolation samples the input feature map at transformed grid points to produce each glimpse feature map.
- Soft tracking: An external memory module assigns each new glimpse to workers according to similarities with past glimpses and their previous attention probabilities.The assignment is a soft attention distribution over workers, illustrated for one glimpse and three workers.
4. Distributed Reasoning on Unstructured Glimpse Clouds
The model tracks semantically meaningful but temporally unstructured glimpses with recurrent workers that receive soft-weighted contributions from all glimpses. An external memory promotes assignments based on similarity to worker-relevant past features, while worker outputs are aggregated for activity classification.
- Challenge: Unstructured glimpses can vary substantially across frames, making conventional temporal tracking unsuitable for associating recurring scene points.The method therefore avoids hard temporal tracking and assignments.
- Glimpse representation: “What” and “where” features combine glimpse appearance, motion, and spatial localization through a learned coordinate embedding.The spatial embedding is combined elementwise with extracted glimpse features.
- Distributed workers: Each recurrent soft-tracking worker receives weighted contributions from all glimpses, allowing the numbers of glimpses and workers to differ.Workers do not hard-assign glimpses and can potentially specialize through independent parameters.
- External-memory assignment: Memory-based attention gives higher glimpse weights to workers associated with similar past features, using a learned distance and recency weighting.The softmax distribution is computed separately for each worker, and each glimpse may contribute to every worker with different weights.
- External-memory assignment: The memory stores glimpse features with worker-specific importance weights and removes the oldest slot when its fixed capacity is reached.More flexible learned storage mechanisms are left for future work.
- Recognition: Worker hidden states support individual classification, whose logits are averaged to produce the final activity probability vector.The supplied passage describes each worker as responsible for its own classification before aggregation.
5. Training
Training combines activity supervision with pose-regression and glimpse-localization terms. Pose is used only during training, while glimpse losses encourage sparse, diverse points that remain near humans.
- Objective: The end-to-end objective sums activity classification, pose prediction, and glimpse-attraction losses.The total loss is written as L = LD(ŷ, y) + LP(ŷp, yp) + LG(l, yp).
- Objective: The activity term LD is a supervised cross-entropy loss on activity labels.It directly supervises the predicted activity distribution.
- Pose supervision: Pose regression is supervised during training to provide additional information without requiring pose at test time.The pose term is predicted from intermediate feature maps.
- Glimpse supervision: The glimpse-attraction loss encourages points to be sparse within each frame while remaining close to humans.Its components promote diversity among glimpses and prevent them from drifting too far from people.
6. Architectures - Pretraining
The architecture preserves temporal resolution in a 3D convolutional backbone and derives it by inflating ResNet-50 spatial kernels. Separate recurrent modules then process attention and tracking without parameter sharing.
- Global backbone: The 3D convolutional network maintains the temporal dimension without temporal subsampling.This preserves information across all input timesteps in the global feature maps.
- Global backbone: The backbone starts from ResNet-50 and inflates its 2D spatial convolutional kernels into 3D kernels.The inflation introduces a temporal dimension into the convolutional architecture.
- Recurrent modules: The recurrent spatial attention module is a 1024-unit GRU, the coordinate network is a 256-unit one-hidden-layer ReLU MLP, and each soft-tracker is a 512-unit GRU.The attention, coordinate, and tracking modules use distinct recurrent or feed-forward components.
- Recurrent modules: The recurrent modules do not share parameters.This applies across the spatial attention, coordinate embedding, and soft-tracking components.
7. Experimental results
Experiments evaluate the method on NTU and Northwestern-UCLA under cross-subject, cross-view, and cross-view protocols, with comparisons against state-of-the-art methods and alternative design choices. The results support distributed recognition, soft assignment, unstructured attention, and learned feature weighting.
- Benchmark results: The method outperforms state-of-the-art methods on NTU and Northwestern-UCLA, including methods using RGB, depth, and pose.NTU uses cross-subject and cross-view evaluation, while Northwestern-UCLA uses cross-view evaluation.
- Ablation study: Distributed recognition improves over the global model by 1.9 points on NTU and 4.4 points on N-UCLA.The larger N-UCLA gap is associated with the greater portion of each frame occupied by people and the efficiency of local representation.
- Ablation study: Soft-assigned Glimpse Clouds clearly outperform simpler baselines that sum glimpse features over time, while adding the global model provides no improvement.The simpler temporal integration baseline yields only a very small improvement over the global model.
- Ablation study: 89.1% is achieved with cross-entropy alone; adding pose prediction gains 0.6 points and pose attraction gains a further 0.4 points.The two additional losses are reported as complementary, although pose is not used during testing.
- Attention strategies: Unstructured attention allows glimpses to attend to new, possibly unrelated scene points in each frame instead of following axis-aligned space-time tubes.The tube baseline requires no tracking or soft assignment to worker modules.
- Attention strategies: Random local-feature sampling drops performance by more than 6 points, indicating that glimpse locations matter.The comparison tests sequential attention against random sampling and a saliency model.
- Design choices: Randomly initializing and fine-tuning the D matrix loses 0.4 points and slows convergence by 1.5×, while fixing D to inverse covariance loses 0.8 points.These results support learning the matrix rather than using either alternative initialization or a fixed value.
- Design choices: Plainly concatenating coordinates and features loses 1.1 points compared with the joint encoding.The experiment evaluates the weighting of “what” and “where” features.
8. Conclusion
The paper presents RGB-only human activity recognition without depth images or articulated pose at test time. Its unstructured Glimpse Cloud is soft-assigned to tracking and recognition workers, and distributed recognition outperforms global and simpler local alternatives on NTU and N-UCLA.
- 8. Conclusion: The method recognizes human activities without depth images or articulated pose at test time, while allowing pose information during training.An attention process creates an unstructured Glimpse Cloud that is soft-assigned to tracking and recognition workers.
- 8. Conclusion: Distributed recognition outperforms a global convolutional model and simpler glimpse-localization baselines in the reported experiments.The authors identify more complex procedures for fusing worker decisions as future work.