TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments

1. Introduction

• Existing methods often estimate single-frame humans in camera coordinates, while dynamic cameras make human and camera motion difficult to disentangle and severe occlusions challenge tracking.The paper notes that global coordinates are needed to capture movement for transfer to a new 3D scene.Source: §1 · p. 2
• TRACE jointly regresses pose, shape, identity, and global trajectory using a 5D representation, temporal features, camera-coordinate motion offsets, world-coordinate motion maps, and an occlusion memory unit.The method uses full-image and scene-motion information in a holistic end-to-end network.Source: §1 · p. 2
• DynaCam contains more than 500 in-the-wild videos with simulated moving cameras, precise camera-pose annotations, and pseudo-ground-truth human annotations for training and evaluating global trajectories.The human annotations are generated by fitting SMPL to detected 2D pose sequences, then using pose and camera annotations to obtain global trajectories with PnP.Source: §1 · pp. 2–3
• TRACE outperforms prior 3D-representation and tracking-by-detection methods on long-term occlusion tracking, and GLAMR on global human trajectories from DynaCam videos.The evaluations use MuPoTS-3D and DynaCam, respectively.Source: §1 · p. 3
• The paper introduces a 5D representation and motion-offset maps, reports state-of-the-art results, and releases TRACE’s code and DynaCam for research.The 5D representation captures temporal cues related to human motion and the scene.Source: §1 · p. 3

Monocular 3D mesh regression with full images

• Full-image methods extend monocular 3D human estimation beyond cropped patches, but ROMP and BEV still estimate people in camera coordinates from a single image.TRACE instead introduces maps for human motion across video in global coordinates.Source: §2 · p. 3

Tracking datasets

• 3DPW provides moving-camera real-world videos, while MuPoTS-3D supplies multi-person interaction scenes with long-term occlusions for tracking evaluation.The paper notes that existing tracking datasets often have limited scenes and human activities.Source: §2 · p. 3

Tracking 3D people through occlusions

• Tracking-by-detection and PHALP use 2D cues or multi-stage assembly, while TRACE learns 3D trajectories from temporal 5D cues end-to-end.The cited methods can be brittle with rapid motion, hand-crafted priors, or errors from early stages.Source: §2 · p. 3

Monocular global 3D human trajectory reasoning

• Global-trajectory methods include SfM and GLAMR, whose reliability can be constrained by multiple moving subjects, occlusion, or ignored scene information; TRACE combines scene cues and 3D motion in a temporal 5D representation.The related-work discussion also describes SLAHMR as a computationally expensive multi-stage optimization approach.Source: §2 · p. 3

3.1. Overview

• TRACE processes each dynamic-camera video into temporal image and motion features, then uses separate heads to recover pose, shape, identity, and trajectories.The detection and tracking branches operate in camera coordinates; the world branch accumulates global translation offsets from the first-frame position.Source: §3.1 · p. 4

3.2. Holistic 5D Representation: Details

• TRACE uses a holistic 5D representation to estimate human states, associate subjects over time, and infer their global trajectories from video.Temporal image features fuse frame features, while optical flow provides motion features; detection and localization maps estimate camera-coordinate positions, and motion and world maps encode temporal offsets and global motion.Source: §3.2 · pp. 4–5

3.3. Tracking with a Memory Unit

• The memory unit matches detections to stored subject trajectories using 3D motion offsets, maintaining identities through long-term occlusions.It uses Hungarian matching at inference; unmatched nodes can persist until a failure-time threshold, and offline tracking can interpolate reactivated trajectory segments.Source: §3.3 · pp. 5–6

3.4. DynaCam Dataset

• DynaCam simulates dynamic-camera videos from in-the-wild static-camera and panoramic footage, producing more than 500 annotated clips with over 48K frames.Camera rotations and field of view are designed for panoramic footage, while sliding-window crops simulate camera translation in static-camera videos.Source: §3.4 · p. 6

3.5. Loss Functions

• TRACE learns temporal motion with a 3D motion-offset loss and a 6D world-motion loss.The world-motion loss includes trajectory and velocity/acceleration terms, foot-keypoint velocity/acceleration, and global body orientation.Source: §3.5 · p. 6

4.1. Implementation Details

• Evaluation covers DynaCam, MuPoTS-3D, and a 16-video Dyna3DPW subset, using trajectory, tracking, and pose/shape metrics.Dyna3DPW comprises 3DPW videos with complete tracking annotations; 3DPW pose/shape evaluation uses root-relative pose because world-coordinate translation ground truth is unavailable.Source: §4.1 · p. 6, §4.2 · p. 8

4.2. Comparisons to State-of-the-art Methods

• TRACE outperforms BEV+DPVO in global trajectory accuracy and GLAMR in both accuracy and efficiency on DynaCam.The comparison evaluates world-coordinate trajectory error and runtime on DynaCam.Source: §4.2 · pp. 7–8, Table 1 · p. 7
• On MuPoTS-3D and Dyna3DPW, TRACE outperforms prior tracking methods, with significantly fewer ID switches under long-term occlusions on MuPoTS-3D.Comparisons account for unlabeled passers-by; TRACE runs online for the ByteTrack comparison on Dyna3DPW.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 8

4.3. Ablation Studies

• TRACE outperforms BEV and multi-stage BEV-based solutions on most metrics, supporting the value of a holistic temporal 5D representation.The comparison spans results in Tables 1 and 3.Source: §4.3 · p. 8
• Estimated 3D motion offsets improve tracking by 3.5% in MOTA, 2.4% in IDF1, and 2.7% in HOTA.This ablation compares tracking with and without predicted 3D motion offsets.Source: §4.3 · p. 8

• Global-coordinate human motion and consistent tracks are needed to relate people to the 3D scene and one another for applications such as behavior analysis and computer graphics.Source: §5 · p. 8
• TRACE is a single-shot method for estimating 3D HPS from video, using a 5D representation and temporal maps that represent people’s motion in camera and global coordinates.The representation includes 3D position, temporal trajectory, and identity.Source: §5 · p. 8
• Future work should explicitly estimate camera motion; the authors suggest camera and human motion may provide complementary information for metric-accurate world-coordinate recovery.Source: §5 · p. 8

• Tab. 4 results were wrong in the original version of the paper.Source: §6 · p. 8
• The per-frame PA-MPJPE evaluation bug produced a joint-wise 14x14 alignment transform for two-person annotations instead of the expected 3x3 rotation matrix.Source: §6 · pp. 8–9
• The error occurs in the widely used PA-MPJPE evaluation function when the joint matrix’s 0-dimensional shape is 2 or 3.Source: §6 · p. 9