Source-linked AI summary
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
Arjun Jain, Jonathan Tompson, Yann LeCun, Christoph Bregler
TL;DR
Human pose estimation in video remains challenging because of high-dimensional inputs, varied poses, and limited use of motion cues. The paper proposes a ConvNet combining RGB and motion features, introduces FLIC-motion, and reports better performance than existing pose-detection techniques. Its spatial model has a scope limitation on datasets with less constrained poses.
Problem
Video pose estimation is challenging because inputs are high-dimensional and poses vary widely, while prior methods primarily rely on appearance rather than motion cues.
Method
A ConvNet takes an RGB image and motion features to estimate 2D joint locations, evaluated on the newly introduced FLIC-motion dataset.
Results
The architecture using RGB and motion features outperforms existing state-of-the-art techniques for human body pose detection in video.
Takeaways & Limitations
Motion features can improve pose detection with a minor increase in model complexity, supporting their use when available.
Takeaways & Limitations
The spatial model is expected to fail on datasets with less constrained pose ranges than FLIC, which is primarily front-facing and standing-up.
Abstract
from arXiv · showhide
In this work, we propose a novel and efficient method for articulated human pose estimation in videos using a convolutional network architecture, which incorporates both color and motion features. We propose a new human body pose dataset, FLIC-motion, that extends the FLIC dataset with additional motion features. We apply our architecture to this dataset and report significantly better performance than current state-of-the-art pose detection systems.
1 Introduction
Human pose recognition in video is difficult because of high-dimensional inputs and varied poses, while conventional methods largely rely on appearance cues. MoDeep addresses this gap by incorporating motion features, introducing FLIC-motion, and targeting efficient pose detection.
- Motion is a powerful visual cue for extracting high-level information, including articulated pose, but prior pose methods primarily used appearance features.Appearance cues included texture patches, edges, color histograms, silhouettes, and hand-crafted local features such as HoG.
- Deep learning successfully incorporates motion features and outperforms existing state-of-the-art techniques for human pose detection.The authors also report that motion features alone outperform several traditional algorithms.
- The system enhances in-the-wild pose detection with motion features and uses an efficient, tractable algorithm approaching real-time frame rates.These properties are presented as making the method suitable for a wide variety of applications.
- FLIC-motion extends the FLIC dataset with motion features for each of 5003 images collected from Hollywood movies.
2 Prior Work
Prior pose-recognition work used geometric models, hand-crafted appearance descriptors, depth-based classifiers, and early neural networks. These approaches differ in their modeling assumptions, input domains, and reliance on manually designed local or global structure.
- Geometric Model Based Tracking: Geometric tracking systems used explicit 2D or 3D jointed body models and typically updated pose parameters incrementally from one frame to the next.Most systems also required hand initialization, with one noted exception.
- Statistical Based Recognition: Statistical recognition methods used descriptors such as oriented angle histograms, SIFT, STIP, HoG, HoF, shape-context histograms, and silhouettes.
- Pictorial Structures and DPM-based methods combined body-part appearance evidence with spatial relationship priors, but relied on hand-crafted local evidence and global structure.
- Pose Detection Using Image Sequences: Random-tree methods performed per-pixel body-part labeling from a single depth image, whereas this work uses at least two unconstrained RGB frames.The cited contrast concerns both the number of frames and the input modality.
- Deep Learning based Techniques: Deep-learning methods achieved state-of-the-art pose recognition on FLIC and LSP, while the proposed translation-invariant model is intended to improve high-precision performance.
3 Body-Part Detection Model
The model estimates 2D human-joint locations from RGB images combined with motion features using a translation-invariant, sliding-window ConvNet. It introduces FLIC-motion and adds normalization, efficient multi-resolution processing, and a torso-conditioned spatial model for evaluation.
- Motion Features: The motion representation includes RGB differences, optical-flow vectors, and optical-flow magnitude, with optional camera-motion compensation.These formulations are designed to provide cues about the true motion field for body-part localization.
- FLIC-motion Dataset: FLIC-motion extends 5,003 labeled FLIC images from Hollywood movies with corresponding motion features, including a 1,016-image test split.The dataset preserves the original FLIC collection while augmenting each image with motion information.
- Convolutional Network: The ConvNet takes an RGB image and motion features as input and produces one dense response map for each joint.The network processes a 3D input tensor and outputs per-pixel joint energies.
- Convolutional Network: Local Contrast Normalization and Local Motion Normalization are applied before normalized RGB and motion channels are concatenated and processed through convolution stages.LMN removes background camera motion and normalizes motion intensity across velocities.
- Efficient Sliding Window: The sliding-window network evaluates 64 × 64 patches every 4 pixels, producing dense joint response maps while remaining translation invariant.An efficient formulation reduces redundant computation across overlapping windows.
- Simple Spatial Model: A torso-conditioned spatial model uses empirical joint masks to select plausible part locations, but is expected to fail when pose ranges are less constrained than FLIC’s.The masks are formed from relative part-to-torso histograms and blurred to cover unseen locations.
4 Results
The experiments evaluate motion-feature variants, temporal offsets, camera-motion compensation, and comparisons with prior pose detectors on FLIC. Motion features improve localization, while RGB plus motion outperforms motion alone and existing techniques.
- Efficiency: Approximately 1.89s is required to calculate optical flow per image pair, making flow the most expensive pipeline component.Training takes approximately 12 hours, and single-image FPROP takes approximately 50ms.
- Motion-feature analysis: Motion features improve localization in occluded, cluttered, and motion-blurred examples where RGB appearance cues are unreliable.The examples include an occluded elbow, camouflaged clothing, cluttered backgrounds, and fast-motion blur.
- Motion-feature analysis: Frame differences improve over the single-RGB-frame baseline and even outperform 2D optical flow, while flow-based features generally perform best.The evaluation measures the percentage of predictions within a radius normalized to a 100-pixel torso size.
- Temporal offsets: Average precision between 0 and 20 pixel radii degrades 3.9% from a -10-pixel to a -1-pixel frame offset.Performance is relatively agnostic to frame separation, but larger frame steps degrade performance.
- Camera-motion compensation: Camera-motion compensation provides less benefit than expected, possibly because LMN removes constant background motion or the network learns to ignore residual parallax.The proposed compensation technique is evaluated with and without camera-motion compensation.
- Comparison with other techniques: The combined RGB-and-motion detector significantly outperforms prior techniques on FLIC, while motion-only input is less accurate than the combined input in the high-accuracy region.Motion-only input nevertheless outperforms several traditional methods, whereas RGB supplies fine details missing from motion features.
5 Conclusion
The authors conclude that combining RGB and motion features in a deep ConvNet improves human body pose detection in video. They also report benefits from motion alone and identify richer temporal modeling as future work.
- Conclusion: Combining RGB and motion features enables the ConvNet to outperform existing state-of-the-art techniques for human body pose detection in video.
- Conclusion: Motion features alone outperform some traditional algorithms, while simple temporal cues improve performance with only a minor increase in model complexity.
- Future work: Future work includes learned spatiotemporal convolution stages and more expressive temporal-spatial models using motion constraints.