Source-linked AI summary
Sim2real transfer learning for 3D human pose estimation: motion to the rescue
Carl Doersch, Andrew Zisserman
TL;DR
Real 3D human pose labels are difficult to obtain, while synthetic RGB-trained models often transfer poorly to real videos. The paper modifies HMR to use motion cues from optical flow and 2D keypoints, achieving competitive 3D mesh recovery on 3DPW while training only on synthetic humans.
Problem
Real-world 3D human pose annotation is extremely difficult, while synthetic-trained models often generalize poorly to real data.
Method
The paper extends HMR with temporal processing and trains on improved SURREAL video composites using optical flow and 2D keypoint inputs.
Results
74.7 is comparable to state-of-the-art methods, despite training only on annotated synthetic SURREAL images.
Takeaways & Limitations
Motion information can help neural networks learn 3D human pose estimation from synthetic images and bridge the sim2real gap when video is available.
Takeaways & Limitations
Methods using large, weakly-labeled real-video datasets still outperform the proposed approach.
Abstract
from arXiv · showhide
Synthetic visual data can provide practically infinite diversity and rich labels, while avoiding ethical issues with privacy and bias. However, for many tasks, current models trained on synthetic data generalize poorly to real data. The task of 3D human pose estimation is a particularly interesting example of this sim2real problem, because learning-based approaches perform reasonably well given real training data, yet labeled 3D poses are extremely difficult to obtain in the wild, limiting scalability. In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person's motion, notably as optical flow and the motion of 2D keypoints. Therefore, our results suggest that motion can be a simple way to bridge a sim2real gap when video is available. We evaluate on the 3D Poses in the Wild dataset, the most challenging modern benchmark for 3D pose estimation, where we show full 3D mesh recovery that is on par with state-of-the-art methods trained on real 3D sequences, despite training only on synthetic humans from the SURREAL dataset.
1 Introduction
The paper addresses the difficulty of obtaining real 3D pose labels and the poor sim2real transfer of RGB-trained models. It proposes motion cues as a bridge and evaluates this idea on 3DPW.
- Real-world 3D human pose labels are difficult to obtain because annotation is impractical and human data raises privacy, intellectual-property, and bias concerns.
- Synthetic data offers potentially infinite examples with accessible ground-truth properties while reducing privacy, intellectual-property, and bias concerns.
- Synthetic-trained neural networks often generalize poorly to real data, limiting sim2real transfer on benchmarks such as human pose estimation.
- 3DPW provides an in-the-wild evaluation dataset despite the near impossibility of manually annotating 3D poses.
- Motion extracted from video may transfer better because synthetic human models match real-human 3D geometry and plausible motion.
- The system modifies HMR by improving synthetic motion and adding optical flow and 2D keypoint tracks, while synthetic RGB inputs can harm performance.
2 Related Work
Prior work has explored synthetic data and sim2real transfer across pose estimation, robotics, and related vision tasks, but transfer remains difficult. This paper builds on motion and abstract representations as promising alternatives to RGB.
- Earlier synthetic-human 3D pose methods generally combine synthetic and real data because transfer from synthetic data remains challenging.
- Purely synthetic training often underperforms real-data training, although depth images provide an exception that requires depth cameras.
- Related work applies synthetic humans to 2D pose estimation, pedestrian detection, action recognition, 3D hand pose estimation, and eye tracking.
- Sim2real robotics research finds that abstractions such as segmentations can improve performance, alongside generative models and simulator randomization.
- Optical flow and 2D keypoints have previously supported pose estimation and sim2real transfer, though model-fitting approaches can be slow, initialization-sensitive, or brittle.
- Mainstream 3D pose methods either lift 2D poses into 3D or regress directly from pixels, with pixel-based methods generally requiring matched training and testing distributions.
3 Algorithm
The algorithm constructs realistic synthetic video sequences and extends HMR with temporal memory and motion inputs. Training includes occlusions and distractors, while optical flow and keypoints provide the principal cues.
- 3.1 Dataset Construction: The dataset includes complex motion, background motion, occluders, and missing-person frames to reproduce challenges in real videos.
- 3.1 Dataset Construction: Figure 2 generates examples by compositing a SURREAL person and synthetic occluder onto a Kinetics video, then cropping around the person.
- 3.1 Dataset Construction: Synthetic training examples composite SURREAL humans onto Kinetics videos, adding realistic backgrounds and roughly 20 billion possible video combinations.
- 3.2 Network Architecture: Motion HMR processes 31-frame clips with a per-frame ResNet-50 CNN, a bidirectional LSTM, and framewise pose estimation.
- 3.2 Network Architecture: The model extends HMR to video so temporal memory can preserve pose beliefs when individual frames provide little discriminative motion.
- 3.2 Network Architecture: Optical flow is computed by frozen FlowNet, while 2D keypoints are simulated during training and detected automatically during evaluation.
4 Results
On the challenging 3DPW benchmark, motion-based inputs substantially improve sim2real 3D pose estimation over RGB-only training. Dataset preprocessing, optical flow, and 2D keypoints each address real-world motion, occlusion, and appearance differences.
- 3DPW evaluates poses in handheld real-world videos with camera motion, lighting variation, clutter, and occlusions.
- RGB-only training performs poorly, while 2D keypoints and optical flow provide substantial and non-trivial improvements for sim2real transfer.The comparison uses PA-MPJPE, where lower is better.
- 74.7 PA-MPJPE is comparable to state-of-the-art methods using similar training pipelines, despite training only on annotated synthetic SURREAL images.The method outperforms HMR trained with real-world motion-capture and real 2D images, while only methods trained on large weakly labeled video datasets currently perform better.
- Qualitatively, the motion-based method is robust to unusual poses and occlusions, whereas the RGB-only baseline often fails to localize even 2D limb positions.
- 4.1 Ablations: Moving backgrounds, tracked camera motion, and occlusions each improve the synthetic-data pipeline, reducing shortcuts that fail on dynamic real videos.The ablation results indicate that static-background motion cues and missing occlusions are important sources of mismatch.
- 4.1 Ablations: Performance improves through 31-frame clips, roughly one second of video, but not with two seconds; longer sequences may accumulate LSTM errors.
5 Conclusions
Motion information helps neural networks learn 3D human pose estimation from synthetic images, supporting sim2real transfer when video is available. The approach may also extend to settings where camera or object motion provides pose cues.
- Motion information helps neural networks learn 3D human pose estimation from synthetic images.
- The final model loses cues such as shading because neural networks overfit to synthetic RGB appearance.
- Self-supervised learning could potentially recover some RGB advantages by training a single-frame model on poses estimated from video.
- Motion information and sim2real transfer may become essential components of pose estimation systems whenever video is available.
A Implementation details for dataset generation
The dataset-generation pipeline matches synthetic and real video geometry, augments synthetic scenes with camera motion, and creates controlled occlusions for training.
- Median optical flow is computed for Kinetics frames and used to translate the synthetic person across frames.
- SLIC superpixels are modified by integrating median flow to estimate and compensate for global camera motion.
- Total occlusions last up to 15 frames during training and evaluation, but frames with fewer than 7 visible keypoints are excluded from evaluation.
B Implementation details for network training
Network training extends HMR with motion-aware inputs, recurrent prediction, and randomized keypoint visibility, while DANN is applied to pre-LSTM representations.
- FlowNet-based optical flow is computed at bounding-box level using paired boxes extending each frame into the future.
- The LSTM uses 1024 hidden units, with an additional 1024-unit layer for each current and future prediction head.
- Domain randomization hides and randomly re-exposes keypoints to model unreliable detections and occlusions during training.
- Optical flow is adapted to the ResNet input by adding flow magnitude as a third channel after scaling the estimates.