PaperlayerView exact revision on arXiv

Source-linked AI summary

XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera

Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, Christian Theobalt

arXiv:1907.00837v2cs.CVcs.GR

TL;DR

XNect addresses the challenge of real-time multi-person 3D motion capture from a single RGB camera despite depth ambiguity and occlusion. It combines evidence-aware per-frame neural inference, per-person pose completion, and temporal skeleton fitting, achieving over 30 fps with state-of-the-art accuracy while producing joint angles. Its scope remains bounded by monocular accuracy, neck-visibility requirements, identity swaps during extended occlusions, and sensitivity to out-of-distribution poses.

  • Problem

    Multi-person monocular capture must handle depth ambiguity, occlusions, identity assignment, and real-time requirements, while prior methods often lack coherent joint-angle output.

  • Method

    XNect uses SelecSLS Net for full-frame 2D and local 3D inference, a compact global-context network for occluded-pose completion, and temporal kinematic skeleton fitting.

  • Results

    The system runs at more than 30 fps on consumer hardware while achieving state-of-the-art accuracy in challenging multi-person scenes.

  • Takeaways & Limitations

    XNect produces temporally stable full skeletal joint angles and camera-relative subject localization suitable for virtual-character control.

  • Takeaways & Limitations

    Accuracy is not yet comparable to multi-view capture, detection requires a visible neck, identity tracking can swap people during extended occlusions, and unusual poses reduce accuracy.

Abstract

from arXiv · show

We present a real-time approach for multi-person 3D motion capture at over 30 fps using a single RGB camera. It operates successfully in generic scenes which may contain occlusions by objects and by other people. Our method operates in subsequent stages. The first stage is a convolutional neural network (CNN) that estimates 2D and 3D pose features along with identity assignments for all visible joints of all individuals.We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short range skip connections to improve the information flow allowing for a drastically faster network without compromising accuracy. In the second stage, a fully connected neural network turns the possibly partial (on account of occlusion) 2Dpose and 3Dpose features for each subject into a complete 3Dpose estimate per individual. The third stage applies space-time skeletal model fitting to the predicted 2D and 3D pose per subject to further reconcile the 2D and 3D pose, and enforce temporal coherence. Our method returns the full skeletal pose in joint angles for each subject. This is a further key distinction from previous work that do not produce joint angle results of a coherent skeleton in real time for multi-person scenes. The proposed system runs on consumer hardware at a previously unseen speed of more than 30 fps given 512x320 images as input while achieving state-of-the-art accuracy, which we will demonstrate on a range of challenging real-world scenes.

1 INTRODUCTION

XNect targets the difficult problem of real-time multi-person 3D motion capture from one RGB camera under occlusion. Its staged design combines fast per-frame inference, occlusion completion, and temporal skeletal fitting to produce accurate joint-angle motion.

  • Monocular multi-person capture remains challenging because of depth ambiguity, occlusions, identity assignment, and computational demands.
  • Prior multi-person methods ran at only 10–15 fps or offline and generally produced joint positions rather than directly usable skeletal joint angles.
  • XNect combines a fully convolutional Stage I, a compact global-context Stage II, and model-based temporal fitting for complete multi-person 3D motion capture.
  • SelecSLS Net accelerates the computationally expensive first stage through selective skip connections, fewer features, and no separate per-subject bounding-box tracker.
  • The complete system maintains real-time performance without sacrificing reported reliability or accuracy, with runtime only mildly dependent on subject count.

2 RELATED WORK

Prior work established monocular pose estimation and efficient multi-person association, but occlusion, ambiguity, proposal fusion, and real-time coherent skeletal output remained unresolved. XNect addresses these gaps with evidence-aware encoding, global pose completion, and temporal fitting.

  • Single-Person 3D Pose Estimation: Monocular 3D lifting is ill-posed because image evidence often cannot disambiguate body-part depth.
  • Multi-Person 3D Pose: Multi-person approaches face difficult detection, association, occlusion, and runtime problems, while prior real-time single-person methods fail in multi-person scenes.
  • Multi-Person 3D Pose: Detection-based methods can be slow and may incorrectly merge nearby people or fail to merge proposals belonging to one person.
  • Multi-Person 3D Pose: XNect uses a bottom-up formulation that predicts all subjects jointly and avoids multiple detections per subject.
  • Multi-Person 3D Pose: Channel-sparse local encoding limits conflicts in crowded images, while Stage II uses 2D evidence, global context, and pose priors to complete occluded bodies.
  • Multi-Person 3D Pose: Model-based fitting adds kinematic constraints and temporal smoothness, producing joint angles and camera-relative positioning in real time.
  • Network Architectures: SelecSLS Net selectively connects features to reduce memory and improve practical speed, addressing limitations of dense or theoretically efficient alternatives.

3 METHOD OVERVIEW

XNect processes a live monocular multi-person video stream through three stages: per-frame pose inference, per-person 3D completion, and temporally coherent skeletal fitting.

  • Stage I: Stage I processes the complete frame jointly for all subjects, producing 2D joints, associations, and intermediate 3D encodings.
  • Stage II: Stage II runs a lightweight fully connected network per detected person to infer visible and occluded root-relative 3D joints.
  • Stage II: Stage II can execute in parallel with negligible inference-time dependence on the number of subjects.
  • Stage III: Stage III fits a kinematic skeleton to temporal histories of 2D and root-relative 3D predictions, yielding stable motion, identities, joint angles, and camera-relative localization.
  • Stage I: SelecSLS Net uses selective long- and short-range concatenation skips to reduce features and memory while preserving comparable accuracy to ResNet-50.
  • The paper evaluates the system through ablations, comparisons, and animated-character-control applications.

4 PER-FRAME POSE ESTIMATION: STAGE I & STAGE II

Stage I predicts multi-person 2D structure and local 3D evidence from the full image, while Stage II converts sparse visible-joint inputs into complete root-relative 3D poses. Channel-sparse supervision reduces encoding conflicts and preserves 2D cues for fast global reconstruction.

  • Problem Formulation: The task estimates root-relative 3D coordinates for the body joints of an unknown number of people from one image.
  • 2D Branch: The 2D branch predicts joint heatmaps and Part Affinity Fields, which locate visible joints, associate them with people, and provide detection confidences.
  • 2D Branch: If a person’s neck is not detected, that individual is discarded from subsequent processing.
  • 3D Branch: The 3D branch predicts local encodings at visible joints, with each encoding representing the joint’s relation to directly connected kinematic neighbors.
  • 3D Branch: Channel-sparse supervision restricts each encoding to bones meeting at the corresponding joint instead of encoding the full body at every location.
  • 3D Branch: This local design addresses conflicts from nearby people while distributing visible-joint 3D information across the feature maps.
  • Stage II: Stage II concatenates neck-relative 2D coordinates, confidences, and 3D features, using zero vectors for invisible joints before predicting the complete pose.
  • Stage II: Retaining 2D information lets Stage II correct conflicts and use global context and learned priors to fill missing joints.

5 SEQUENTIAL MOTION CAPTURE: STAGE III

Stage III converts per-frame pose predictions into temporally coherent skeletal motion by tracking identities, estimating scale and position, and fitting a constrained kinematic model.

  • 5 SEQUENTIAL MOTION CAPTURE: STAGE III: Stage III addresses missing person size, metric camera distance, cross-frame identity, and joint-angle reconstruction after the first two stages.The system adds appearance-based identity tracking, optional ground-plane scale inference, and space-time fitting to resolve these issues.
  • 5.1 Identity Tracking and Re-identification: 30 hue and 30 saturation bins define each subject’s HSV upper-body appearance descriptor for efficient identity matching.The descriptor uses class probabilities within the torso-joint bounding box and can represent loose or tight clothing.
  • 5.1 Identity Tracking and Re-identification: Appearance matching maintains identities through temporary disappearances, while thresholds detect occlusions, exits, and newly entering people.Known identities persist for a limited number of frames, and appearance histograms update at arrival and every 30 seconds.
  • 5.2 Scale and Position: Relative bone lengths averaged over the first 10 frames provide scale-invariant skeletal measurements for subsequent fitting.Absolute metric reconstruction remains difficult without a reference scale, person height, or ground-plane geometry.
  • 5.3 Kinematic Skeleton Fitting: The fitting stage jointly optimizes 2D reprojection and root-relative 3D errors while enforcing joint limits, temporal smoothness, and stronger depth regularization.The final model has D = 29 degrees of freedom and outputs joint angles plus global root position for each person.

6 SELECSLS NET: A FAST AND ACCURATE POSE INFERENCE CNN

SelecSLS Net accelerates the computationally dominant Stage I CNN through selective concatenation skips, retaining pose accuracy while reducing inference cost and memory.

  • 6 SELECSLS NET: A FAST AND ACCURATE POSE INFERENCE CNN: Stage I is the pipeline’s computational bottleneck, motivating a faster CNN architecture for real-time multi-person pose inference.On Nvidia 1080-MaxQ hardware, a ResNet-50 Stage I pass takes >30ms plus ≈15ms for other components, missing >25 fps.
  • 6 SELECSLS NET: A FAST AND ACCURATE POSE INFERENCE CNN: SelecSLS Net uses selective short- and long-range concatenation skips instead of additive skips, improving information flow without DenseNet’s full-connectivity cost.The architecture uses fewer features than ResNet-50 without accuracy loss.
  • 6.1 SelecSLS Module: Long-range skips connect each module to the first module at its spatial-resolution level rather than only to the preceding module.SelecSLS modules combine intra-module short-range and cross-module longer-range connectivity with interleaved 3×3 and 1×1 convolutions.
  • 6.1 SelecSLS Module: 80% memory usage for a single 512×320 image and 50% for batch size 32 distinguish SelecSLS Net from ResNet-50.The smaller memory footprint is identified as the primary source of its speed advantage.

7 RESULTS

The system delivers real-time, accurate multi-person 3D motion capture from a single RGB camera, including occluded and overlapping scenes. It exceeds 30 fps on consumer hardware while producing temporally stable joint-angle outputs suitable for character animation.

  • Real-Time Performance: More than 30 fps is achieved for 512×320 input with 10 subjects on a desktop, while a laptop reaches approximately 27 fps.On the laptop, Stage I takes 21.5 ms, feature extraction 1 ms, Stage II 1 ms, and Stage III approximately 9 ms.
  • Multi-Person Scenes and Occlusion Robustness: The method handles interacting and overlapping subjects under occlusion better than several single-person and multi-person alternatives.It avoids spurious predictions associated with top-down approaches and is reported as more robust than Mehta et al. [2018b].
  • Comparison With KinectV2: Single-RGB-camera pose quality is comparable to KinectV2, including success in certain cluttered scenarios where depth-based person identification is ambiguous.The comparison includes clutter and occlusion cases in which KinectV2 produces multiple skeletons or mispredicts pose.
  • Character Animation: The system reconstructs temporally coherent joint angles and stable camera-relative localization that can directly drive virtual avatars.Skeleton fitting reconciles 2D and 3D poses across time to produce smooth joint-angle estimates.
  • 3D Pose Accuracy: Stage II and Stage III achieve higher 3DPCK than LCRNet, LCRNet++, and Mehta et al. [2018b], while remaining comparable to Dabral et al. [2019].Moon et al. [2019] performs better but uses person detection and resized person crops.
  • Network Evaluation: SelecSLS outperforms ResNet-34 and slightly outperforms ResNet-50 on 3D pose metrics, including lower-body joints.On Human3.6m, mean per joint position error is 63.6 mm with SelecSLS, compared with 64.8 mm for ResNet-50 and 67.6 mm for ResNet-34.
  • Channel-Sparse Encoding: Channel-sparse supervision yields 82.8 3DPCK versus 80.1 for channel-dense supervision, with larger gains on difficult sitting poses.The encoding also performs better across single-person and multi-person 3D pose benchmarks.

8 DISCUSSION AND FUTURE WORK

The method establishes real-time monocular multi-person capture in challenging scenes, but its accuracy and tracking remain bounded by monocular ambiguity, occlusion, and identity-tracking failures. Future improvements target close interactions, tracking, and broader pose coverage.

  • Scope: The system performs real-time 3D motion capture of challenging multi-person scenes with one color camera, while acknowledging limitations for future work.This is presented as the first system to achieve this setting in real time.
  • Monocular Limitations: Monocular accuracy remains below multi-view capture, and incorrect 2D poses or part associations can produce incorrect 3D estimates.People may also be missed when their neck is occluded, even if most of the body is visible.
  • Close Interactions: Extremely close interactions such as hugging remain unreliable despite successful capture under difficult inter-person occlusions.Physics-based motion constraints are proposed to improve stability and fine-grained interaction capture.
  • Occlusion Effects: Strong occlusions can cause short-lived pose errors, while substantial inter-person or self-occlusion can destabilize camera-relative localization.The reconstruction remains appropriate for many real-time applications.
  • Tracking and Generalization: The simple identity tracker may swap identities during extended occlusions, drastic appearance changes, or similar clothing, motivating more sophisticated space-time tracking.Accuracy also worsens for poses very dissimilar from the training poses.
  • Architecture Outlook: SelecSLS is faster and uses substantially less memory than ResNet-50 without compromising accuracy, supporting larger batches or lower-end GPUs.The design can also accommodate further convolutional speedups and hardware-specific optimization.

9 CONCLUSION

XNect combines pose representations, network architectures, and model-based fitting for real-time multi-person 3D motion capture from one RGB camera. It handles generic-scene occlusions, provides joint angles, and exceeds 30 fps on consumer hardware.

  • XNect presents a real-time approach for multi-person 3D motion capture using a single RGB camera.
  • The system operates in generic scenes and handles occlusions caused by people and objects.
  • XNect provides joint angle estimates and localizes subjects relative to the camera.
  • SelecSLS Net uses selective long- and short-range concatenation skips to reduce memory footprint and accelerate inference without compromising accuracy.
  • The proposed system runs at more than 30 fps on consumer hardware and is demonstrated on challenging real-world scenes.

1 SELECSLS NET DESIGN EVALUATION

The supplemental evaluation selects a SelecSLS Net design using concatenation skips, first-module cross-level connectivity, and the no(W) output scheme. Experiments compare architectures and deployment speed across pose-estimation settings and hardware.

  • SelecSLS Net Design Evaluation: SelecSLS design variants differ in module skip type, stride, intermediate features, cross-module connectivity, and output count.
  • SelecSLS Net Design Evaluation: Design selection tests the Stage I network with a SelecSLS core on 2D multi-person pose estimation using the 2D branch.
  • SelecSLS Net Design Evaluation: ResNet-50 and ResNet-34 serve as core-network baselines for comparisons on 2D and 3D pose estimation.
  • SelecSLS Net Design Evaluation: Table 1 evaluates baseline architecture choices using LSP accuracy and inference-speed ratios relative to ResNet-50 on an NVIDIA K80.
  • SelecSLS Net Design Evaluation: The best SelecSLS Net uses concatenation-skip modules, first-module cross-level connectivity, and the no(W) output scheme.

2 SELECSLSNET ON IMAGE CLASSIFICATION

A SelecSLSNet variant is evaluated beyond pose estimation on ImageNet image classification. The experiment uses the architecture’s core design with a classification head replacing the pose branches.

  • The ImageNet experiment trains a SelecSLSNet variant to evaluate the architecture on image classification beyond 2D and 3D body pose estimation.
  • The classification variant uses the same core architecture design as the multi-person task and replaces the 2D and 3D branches with a five-layer classification head.

3 ABLATION OF INPUT TO STAGE II

Stage II benefits from progressively adding confidence and 3D pose features to 2D pose inputs. The ablation also examines architecture and representation variants, with gains extending to difficult pose classes.

  • Input Components: Stage II variants are evaluated by taking different subsets of Stage I outputs as input on the MPI-INF-3DHP single-person 3D pose benchmark.
  • Input Components: Using only 2D pose for Stage II achieves 76.0 3DPCK, while adding joint detection confidences increases accuracy to 77.2 3DPCK.
  • Input Components: A minor decrease to 75.5 3DPCK accompanies the 3D-branch configuration, but performance improves on underrepresented Sitting and On The Floor classes.
  • Input Components: Adding 3D pose features further increases accuracy to 82.8 3DPCK and improves AUC and MPJPE.
  • Input Components: The additional inputs improve both simpler and more difficult poses, including Standing/walking, Sitting, and On the Floor.
  • Input Components: The ablation identifies limitations of 2D-to-3D lifting and supports using joint confidences plus channel-sparse 3D pose encodings.

4 SEQUENTIAL MOTION CAPTURE (STAGE III): ADDITIONAL DETAILS

Stage III supplements motion capture with optional ground-plane calibration, camera calibration, and gradient-based tracking-error recovery. These mechanisms improve localization and help detect failures during skeletal fitting.

  • Absolute Height Calibration: Ground-plane calibration estimates a subject’s camera-relative position from foot detections and the ground-plane intersection, then derives subject height.The procedure uses a camera ray through the detected foot and measures height from the ground plane to a virtual billboard intersection.
  • Camera Calibration: Checkerboard calibration computes intrinsic and extrinsic camera parameters once before recording.The paper notes that other object-free calibration approaches could also be used.
  • Inverse Kinematics Tracking Error Recovery: Tracking failure is identified by monitoring the gradients of the 3D and limb fitting terms during gradient-descent optimization.A threshold exceeded for 30 frames indicates failures from identity-track mismatches or local minima.

5 MORE QUALITATIVE RESULTS

Qualitative evaluations show that the system remains useful across challenging multi-person scenes and supports live character control, while its main failures involve occlusion, limb association, and poses unlike the training data.

  • Limitations: The method mispredicts limb associations across subjects and can miss people when neck occlusion hides the neck reference joint.The authors suggest using multiple reference joints to address neck-occlusion failures.
  • Comparisons With Prior Work: The approach produces quality comparable to or better than prior single-person and multi-person methods, including methods running offline or at interactive rates.Prior proposal-based approaches can require filtering that suppresses valid predictions or leaves ghost predictions.
  • Live Interaction and Character Control: The system can replace typical depth-sensing game controllers by enabling subjects to interact with live virtual avatars.Temporally smooth joint-angle predictions support real-time virtual character control.
  • Diverse Pose and Scene Settings: The system handles diverse poses and scenes with varying lighting, backgrounds, and person density.Examples include challenging motions, interactions, and self-occlusion in multi-person datasets and real-world scenes.
  • Comparisons With Prior Work: VNect is limited to single-person scenarios, whereas LCRNet++ handles multiple people but operates at interactive rates and requires proposal-fusion post-processing.The comparison highlights differences in multi-person handling, speed, and proposal fusion.
Loading 1907.00837v2…