Source-linked AI summary
Humans in 4D: Reconstructing and Tracking Humans with Transformers
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, Jitendra Malik
TL;DR
The paper addresses reconstructing 3D humans from images and tracking them over time in video. It introduces transformer-based HMR 2.0 and builds 4DHumans around 3D tracking, reporting state-of-the-art tracking and action-recognition results. The authors also identify limitations involving the SMPL model, close-contact interactions, and camera-frame reconstructions.
Problem
The paper addresses the problem of reconstructing and tracking humans from images and video, including difficult unusual poses and identity maintenance over time.
Method
HMR 2.0 uses a fully transformerized architecture for single-image human mesh recovery, while 4DHumans uses its 3D reconstructions with an adapted PHALP tracker.
Results
4DHumans achieves state-of-the-art tracking, HMR 2.0 improves pose reconstruction and action recognition, and the system reaches 42.3 mAP on AVA.
Takeaways & Limitations
The results support using transformer-based 3D pose reconstruction as a backbone for tracking people in video and pose-based action recognition.
Takeaways & Limitations
The approach is limited by the SMPL model, less successful reconstruction of people in close proximity, and reconstructions that live in the camera frame.
Abstract
from arXiv · showhide
We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.
1. Introduction
The paper introduces HMR 2.0, a fully transformer-based model for single-image 3D human mesh recovery, and 4DHumans for jointly reconstructing and tracking people in video. It reports improvements in pose reconstruction, tracking, and action recognition, including 42.3 mAP on AVA.
- Broader motivation: HMR 2.0 is presented as a large pretrained model with potential applications in robotics, computer graphics, biomechanics, and human movement analysis.The paper frames these applications as potential uses of the publicly available model.
- HMR 2.0: HMR 2.0 is an end-to-end transformerized architecture for human mesh recovery that avoids domain-specific design choices.It uses a ViT image encoder and a standard transformer decoder to predict pose and shape parameters.
- 4DHumans: 4DHumans builds on HMR 2.0 and PHALP to jointly reconstruct and track humans in video.The system links 3D reconstructions over time and is designed to handle multiple people.
- Downstream evaluation: 4DHumans achieves state-of-the-art tracking results, while HMR 2.0 improves downstream action recognition on AVA.The reported AVA result is 42.3 mAP.
2. Related Work
Prior human mesh recovery methods use direct regression, pseudo-ground-truth fitting, or specialized architectures, while related mesh methods make non-parametric predictions. The paper instead emphasizes a domain-agnostic transformer design and connects reconstruction with tracking and action recognition.
- Human Mesh Recovery: HMR directly regresses SMPL body shape from a single image, following earlier approaches that often used iterative optimization.Later methods introduced alternative pseudo-ground-truth generation strategies using temporal information, multiple views, or optimization.
- Human Mesh Recovery: Recent HMR methods add specialized mechanisms for mesh alignment, occlusion handling, SMPL hierarchy, or joint-location-aware pooling.The paper contrasts these domain-specific designs with its own architecture.
- Human Mesh Recovery: Non-parametric approaches regress mesh vertices directly, whereas this paper predicts SMPL model parameters and is therefore not directly comparable to them.Examples include graph neural network, transformer, and hybrid architectures.
- Video Tracking and Action Recognition: Prior video approaches often used off-the-shelf tracking before reconstructing people, while the proposed system combines reconstruction and tracking for videos in the wild.The related tracking literature includes systems limited to single-person videos and pipelines based on 3D reconstruction features.
3. Reconstructing People
HMR 2.0 reconstructs 3D human pose and shape from a single image with an end-to-end transformer architecture. It combines parametric body modeling with supervised, keypoint-based, and adversarial training signals, including pseudo-ground-truth annotations for unlabelled data.
- Preliminaries: The SMPL body model maps pose and shape parameters to a 3D mesh and derives joints as weighted combinations of mesh vertices.Pose includes body pose and global orientation, while shape is represented by 10 parameters.
- Preliminaries: HMR predicts SMPL pose, shape, and camera parameters from a single image.The predictor is written as Θ = [θ, β, π] = f(I).
- Architecture: HMR 2.0 uses a ViT image encoder and a standard transformer decoder that cross-attends to image tokens to output the reconstruction parameters.The architecture avoids domain-specific design choices and uses a zero input token for decoder readout.
- Losses: Training combines 2D losses, 3D losses, and an adversarial discriminator, with applicable losses selected according to each dataset’s annotations.The same loss framework is used with pseudo-ground-truth annotations.
- Losses: 2D keypoint annotations supervise projected 3D keypoints, while accurate 3D keypoints receive direct L1 supervision.These signals complement parameter supervision when the corresponding annotations are available.
- Training data: Unlabelled images are scaled through detector and keypoint-estimator outputs, followed by ProHMR fitting to obtain pseudo-ground-truth SMPL and camera parameters.The resulting θ*, β*, and π* annotations support training on additional datasets.
4. Tracking People
4DHumans tracks multiple people by lifting detections into a 3D representation and matching temporal predictions with new observations. Its transformer pose predictor also supports future-pose prediction and amodal completion of missing detections.
- Tracking pipeline: PHALP-based tracking lifts each detected person into 3D using pose, location, and appearance features, then builds a tracklet over time.Tracklet state is updated from incoming observations after matching predictions to detections.
- Pose prediction: The pose-prediction model is trained on over 1 million tracks and uses positional embeddings with masked tokens for future prediction and amodal completion.The same masking mechanism handles both future time steps and missing detections.
- Tracking pipeline: The tracker predicts each tracklet’s next pose, location, and appearance in 3D before matching those predictions with lifted detections.This top-down prediction and bottom-up detection matching operates across video frames.
- Pose prediction: PHALP′ replaces HMR-specific embeddings with SMPL pose parameters and trains a masked-token transformer to predict future poses and complete missing detections.These changes allow different mesh-recovery methods to be plugged into the tracker and support operation on videos.
- System design: 4DHumans combines reconstruction and tracking in one system rather than using tracking outputs only to reconstruct people afterward.The system is described as operating on videos with multiple people.
5. Experiments
The experiments evaluate HMR 2.0 for 2D/3D pose accuracy, 4DHumans for tracking, and HMR 2.0 for pose-based action recognition. Results show robust unusual-pose recovery, state-of-the-art tracking, and improved action recognition.
- 5.2. Pose Accuracy: HMR 2.0b consistently outperforms previous approaches on projected-keypoint accuracy, reaching PCK@0.05 of 0.53 on LSP-Extended versus 0.32 for CLIFF.The advantage narrows to 9% on COCO and 6% on PoseTrack.
- 5.2. Pose Accuracy: HMR 2.0b is qualitatively robust to unusual poses, viewpoints, partial visibility, truncation, and occlusion, while producing image-aligned and novel-view-valid reconstructions.Figure 4 organizes these conditions across unusual poses, viewpoints, and difficult visibility cases.
- 5.3. Tracking: 4DHumans achieves state-of-the-art tracking performance across all reported metrics on PoseTrack and improves ID switches by 22%.Using the improved ViTDet detector further improves performance.
- 5.4. Action Recognition: HMR 2.0 achieves 22.3 mAP on AVA pose-based action recognition, 14% better than the second-best baseline, and reaches 42.3 mAP with appearance features.The paper evaluates mesh-recovery methods through downstream action recognition on AVA.
- 5.5. Design Investigation: The study also examines more than 100 HMR 2.0 settings by visualizing the performance of 100 checkpoints per run on 3DPW and LSP-Extended.This exploration is presented as a series of lessons learned about model design.
6. Conclusion
The paper presents HMR 2.0 for human mesh reconstruction and 4DHumans for jointly reconstructing and tracking people in video, with strong tracking and action-recognition results. It also identifies limitations involving body modeling, close-proximity interactions, camera coordinates, and input resolution.
- HMR 2.0 is a fully transformerized human mesh-recovery network with strong 2D/3D pose performance.
- 4DHumans jointly reconstructs and tracks people in video, achieving state-of-the-art tracking results.
- HMR 2.0 improves action recognition over previous pose-based baselines.
- SMPL limits modeling of hand pose, facial expressions, and broader age variation.
- Independent person processing is less successful for fine-grained interactions between people in close proximity.
- Camera-frame reconstructions require camera-motion reasoning for a common world-coordinate understanding of video actions, while lower resolution can reduce reconstruction quality.
Supplementary Material for: “Humans in 4D: Reconstructing and Tracking Humans with Transformers”
The supplementary material provides additional details about the model, data, training, pose prediction, evaluation metrics, tracking, action recognition, and qualitative results.
- The supplement details the HMR 2.0 architecture, training data, and training pipeline.
- It also describes pose prediction and the metrics used for evaluation.
- Experimental settings for tracking and action recognition are documented alongside additional qualitative results.
S.1. HMR 2.0 architecture details
HMR 2.0 uses a ViT image encoder and a transformer decoder as its core architecture.
- HMR 2.0 is based on a ViT image encoder and a transformer decoder.
- The encoder is a pre-trained ViT-H/16 with 50 transformer layers and 16 × 12 output image tokens of dimension 1280.
- The model accepts 256 × 192 images as input.
- The transformer decoder has 6 layers containing self-attention, cross-attention, and feed-forward blocks.
S.2. Data details
HMR 2.0a uses established human-pose datasets, while HMR 2.0b adds pseudo-ground-truth data from additional video datasets and filters unreliable fits.
- HMR 2.0a trains on Human3.6M, COCO, MPII, and MPI-INF-3DHP images.
- HMR 2.0b additionally uses pseudo-ground-truth SMPL fits from AVA, InstaVariety, and AI Challenger.
- Pseudo-ground-truth generation uses ViTDet for bounding boxes, ViTPose for keypoints, and ProHMR for fitting.
- The pipeline discards detections with fewer than five keypoints or confidence below 0.5, along with implausible shapes, poses, and large fitting errors.
S.3. Training details
HMR 2.0 is trained end-to-end with AdamW for 1M iterations on eight A100 GPUs, while HMR 2.0a freezes its ViT encoder.
- HMR 2.0b is trained end-to-end for 1M iterations using eight A100 GPUs and an effective batch size of 384.Training takes roughly six days.
- The optimizer uses AdamW with a 1e-5 learning rate, β1 = 0.9, β2 = 0.999, and 1e-4 weight decay.
- HMR 2.0a freezes the ViT encoder, enabling an effective batch size of 4096, a 1e-4 learning rate, and fewer training iterations.
S.4. Pose prediction
The pose-prediction model learns temporal 3D pose and location completion from tracklets, while evaluation uses standard pose, tracking, and action-recognition metrics. The broader experiments assess tracking, action recognition, and qualitative behavior, including failure cases.
- Pose prediction: Each tracklet contains time-indexed 3D SMPL pose and camera-frame location, which a six-layer transformer predicts at each requested timestep.The transformer uses eight self-attention heads and a hidden dimension of 256.
- Pose prediction: Randomly masked pose tokens train the model to predict future poses and complete missing detections using a learned mask token.
- Evaluation: Evaluation reports MPJPE and PA-MPJPE for 3D pose, PCK for 2D pose, ID switches, MOTA, IDF1, and HOTA for tracking, and mAP for action recognition.Action recognition is additionally analyzed for object manipulation, person interactions, and person movement.
- Tracking: The final 4DHumans system combines a sampling-based appearance head with pose prediction to achieve state-of-the-art tracking performance on PoseTrack.Visible mesh points are textured by projecting them onto the input image and sampling corresponding pixel colors.
- Action recognition: Action-recognition experiments replace poses in existing tracklets with outputs from different human mesh recovery models and evaluate them on AVA.
- Qualitative analysis: Additional qualitative experiments compare HMR 2.0 with close competitors and show challenging successes and failure cases.