Source-linked AI summary
Recurrent Attention Models for Depth-Based Person Identification
Albert Haque, Alexandre Alahi, Li Fei-Fei
TL;DR
Depth-based person identification remains difficult despite its value for authentication, tracking, safety, and activity understanding. The paper uses recurrent attention and reinforcement learning to select discriminative spatio-temporal regions, achieving state-of-the-art performance on several datasets while providing interpretable attention visualizations.
Problem
Depth-based person identification addresses a difficult computer-vision task with applications in authentication, human tracking, public safety, and activity understanding.
Method
A recurrent attention model formulates depth-video identification as reinforcement learning, pruning high-dimensional 4D inputs to focus on small, discriminative regions.
Results
The model achieves state-of-the-art performance on several person-identification datasets and outperforms human, depth-based, and an RGB-D baseline.
Takeaways & Limitations
Interpretable 2D, 3D, and 4D attention visualizations provide insights into volumetric and motion-based differences between individuals.
Takeaways & Limitations
A selected glimpse path may reflect a policy that did not explore alternative paths, despite training with varied initial glimpse locations; 4D time denotes RAM iterations rather than video-frame order.
Abstract
from arXiv · showhide
We present an attention-based model that reasons on human body shape and motion dynamics to identify individuals in the absence of RGB information, hence in the dark. Our approach leverages unique 4D spatio-temporal signatures to address the identification problem across days. Formulated as a reinforcement learning task, our model is based on a combination of convolutional and recurrent neural networks with the goal of identifying small, discriminative regions indicative of human identity. We demonstrate that our model produces state-of-the-art results on several published datasets given only depth images. We further study the robustness of our model towards viewpoint, appearance, and volumetric changes. Finally, we share insights gleaned from interpretable 2D, 3D, and 4D visualizations of our model's spatio-temporal attention.
1. Introduction
The paper targets person identification from depth imagery despite variability in appearance, viewpoint, pose, and limited examples. It proposes a recurrent attention approach that learns discriminative 4D body-shape and motion signatures, evaluates challenging conditions, and reports state-of-the-art results.
- Person identification remains difficult because visual features are weak across time and are affected by illumination, viewpoint, pose, and intra-class variation.
- The approach learns body-shape and motion signatures from depth images or video, motivated by the uniqueness of individual gait.
- A reinforcement-learning formulation prunes high-dimensional depth inputs and focuses attention on small, discriminative spatio-temporal regions.
- The evaluation varies viewing angle and tests people carrying objects or wearing hats and backpacks, while also introducing a challenging dataset.
- The model achieves state-of-the-art results on several existing person-identification datasets using depth information.
2. Related Work
Prior identification methods use RGB, handcrafted depth or silhouette features, temporal embeddings, or image-similarity formulations. This work instead emphasizes depth-only input, spatio-temporal modeling, and attention-based processing of compact 4D regions.
- RGB-based identification research addresses intra-class variation through learned feature representations and similarity metrics, while silhouette methods use body-part distances.
- Earlier depth-based methods rely on handcrafted anthropometric or low-level RGB features, while spatio-temporal representations remain underexplored for person identification.
- Gait energy approaches embed temporal information into two-dimensional silhouettes or three-dimensional depth representations and commonly use nearest-neighbor prediction.
- Deep identification methods often compare multiple images using Siamese, filter-pair, triplet, or cross-input architectures.
- Unlike these approaches, the proposed model uses depth without RGB, takes a single image as input, and does not rely on metric learning.
- The method uses visual glimpses to process small 4D regions at high fidelity while covering larger regions at lower detail.
3. Our Model
The model identifies people from depth images or videos by navigating a high-dimensional 4D input with recurrent hard attention. It combines glimpses, encoding, an LSTM-based RAM, and reinforcement learning to focus on discriminative spatio-temporal regions while addressing limited training data.
- 3.1. Input Representation: A 4D depth-video input can produce 2.5×10^9 features per average video, roughly three orders of magnitude larger than a typical RGB convolutional input.This scale motivates intelligent navigation of the input space rather than processing the full representation.
- 3.2. Recurrent Attention Model: The model downsamples high-dimensional inputs into multi-scale glimpses, then compresses each glimpse with a separately trained 4D convolutional autoencoder.Glimpses preserve high resolution near the selected location while using progressively lower resolution farther away; the encoder makes the resulting features feasible for RAM training and testing.
- 3.2. Recurrent Attention Model: The core RAM uses an LSTM and two sub-networks to update its hidden state, select the next glimpse location, and predict the current identification label.The LSTM receives encoded features and the previous hidden state, while the location and action networks produce the next location and label actions.
- 3.3. Training and Optimization: Reinforcement learning trains a stochastic policy over glimpse locations and labels, rewarding a correct label at the final timestep and optimizing it with REINFORCE.The policy maps the historical glimpse path to a distribution over current actions, allowing attention to focus on salient regions in space and time.
- 3.3. Training and Optimization: Training on glimpses exposes the model to approximately 1×10^6 to 1×10^9 possible examples per video, reducing the practical impact of limited training data.The model trains on subsets of the video rather than repeatedly seeing the entire sequence.
4. Experiments
The experiments evaluate depth-based person identification across single-shot and multi-shot settings, including challenging viewpoint and appearance conditions. The recurrent attention model improves with higher-dimensional learned encoding, surpasses deep-learning and human/depth-based baselines, and reveals interpretable spatio-temporal attention patterns.
- Datasets: The DPI-T dataset tests identification across days with varied clothing, viewing angles, walking speeds, carried objects, and accessories.Its top-view camera setting introduces self-occlusion and incomplete point-cloud reconstruction challenges.
- Single-Shot Identification: Performance improves monotonically as the RAM input increases from 2D to 3D, unlike gait energy, whose higher-dimensional volume performs worse.The comparison indicates that learned encoding preserves useful information from higher-dimensional inputs.
- Single-Shot Identification: The 3D RAM outperforms a 3D CNN operating on the same point-cloud input.The authors attribute this to glimpse-based augmentation, whereas the CNN processes the entire point cloud without such augmentation.
- Multi-Shot Identification: The 4D RAM outperforms the human baseline and existing depth-based approaches on multi-shot identification.It also outperforms an RGB-D method by selecting informative glimpses instead of processing the entire noisy point cloud.
- Hard Attention Regions: In 4D RAM, time is a free parameter, allowing attention to jump forward or backward across frames while changing spatial location and effective temporal emphasis.Visualizations show attention to shoulders and feet, consistent with motion regions potentially carrying identity information.
- Hard Attention Regions: The model’s 2D attention projections nearly always visit major skeleton joints and highlight body regions such as shoes, hair, and hips.The heatmap shows that different body regions attract varying levels of attention.
5. Conclusion
The paper introduces a recurrent attention model that identifies people from discriminative spatio-temporal regions in depth video. Across 2D, 3D, and 4D inputs, it achieves state-of-the-art performance and produces visualizations that offer insights into volumetric and motion-based identity differences.
- Conclusion: The recurrent attention model identifies discriminative spatio-temporal regions and learns volumetric signatures from high-dimensional 4D depth input.Reducing dimensionality through glimpses and an encoder enables recurrent processing with an LSTM module.
- Conclusion: Across 2D, 3D, and 4D inputs, the attention model achieves state-of-the-art performance on several person identification datasets.Its attention visualizations provide insights for computer vision, biomechanics, and physiology.