Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

1. Introduction

• Scarce and restrictive ground-truth data remains a fundamental obstacle for learning-based and traditional 3D reconstruction methods.Traditional approaches also depend on tuned expressiveness and connectivity priors,,.Source: §1 · p. 1
• MoCap provides accurate full-body 3D data, but markers and tight dark clothing make captured scenes unrepresentative of natural images.Synthetic alternatives remain non-natural and often miss deformation and ambiguity from loose clothing [24] [39] [11].Source: §1 · p. 1
• Chaining natural-image 2D detection with MoCap-trained 3D lifting leaves the method’s 3D accuracy on natural images fundamentally uncertain.Accuracy on MoCap 3D data and natural-image 2D joints does not establish accuracy for 3D natural-image reconstruction.Source: §1 · p. 1
• The multi-stage network reconstructs a coherent 3D pose from all camera views, projects it back into each image, and uses that feedback to refine joints and recover from mistakes.A robust estimator can discard a minority of incorrect 2D estimates while feeding 3D-informed joint locations back into image refinement.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The approach labels unlabelled data for training existing methods and yields overall better performance on standard benchmarks with consistent multimillimetre improvement.The authors frame dataset expansion as a possible alternative to pursuing very small architectural gains between state-of-the-art networks.Source: §1 · p. 2
• Unlike multiview bootstrapping that retrains 2D estimators, this refinement occurs at test time using one synchronized set of images rather than extensive retraining.The authors describe the two approaches as complementary.Source: §1 · p. 2

Direct image to 3D pose regression

• Monocular 3D methods regress pose directly, combine image features with 3D priors, or predict structured 3D heatmaps, but commonly require studio 3D supervision.Their dependence on MoCap images limits generalisation to images in the wild.Source: §2 · pp. 2–3

3D pose from 2D joint estimates

• Two-stage methods lift detected 2D landmarks into 3D through model fitting or regression, with prior results indicating that incorrect 2D estimates are a major reconstruction error source.Martinez et al. [16] report strong performance when ground-truth 2D landmarks are provided.Source: §2 · p. 3

Joint 2D-3D pose estimation

• Joint 2D–3D approaches share representations or prediction branches and typically combine 2D and 3D losses in end-to-end training.These methods encode interactions between image-space and 3D pose objectives.Source: §2 · p. 3

Training with 2D-only loss

• Reprojection-based methods retain an internal 3D pose while training from 2D losses, improving in-the-wild applicability and incorporating structural 3D priors.Related methods combine reprojection with supervised, adversarial, synthetic, segmentation, or optical-flow signals [35] [41] [38] [37].Source: §2 · p. 3

Multi-view human pose

• Existing multiview systems fuse per-view detections or use geometry, visual hulls, and temporal models, often estimating each view before final fusion.Examples include Elhayek et al., Pavlakos et al. [21], and Trumble.Source: §2 · p. 3
• This method generalises multi-stage approaches [35] [40] to multiple views by iteratively seeking an estimate consistent across all cameras instead of performing per-view pose estimation before fusion.Its novelty is joint multiview refinement rather than post-processing fusion.Source: §2 · p. 3

3. Our Formulation

• Each stage refines per-camera 2D heatmaps, reconstructs a 3D pose consistent across views, and reprojects it to guide subsequent predictions.The internal 3D representation enables 2D losses throughout the stages without requiring image-specific 3D annotations.Source: §3 · p. 3, §3.1 · p. 4
• The multi-view reconstruction lets the model reconsider joint locations using interim 3D pose, recover from prior mistakes, and remain trainable end-to-end through computable sub-gradients.The formulation preserves differentiability while replacing least squares with an IRLS procedure that mimics the Huber loss.Source: §3 · pp. 3–4

3.1. Details of the Network

• The network shares view-wise convolutional operations, predicts heatmaps independently for each camera, and fuses image-based and reprojected heatmaps through a final convolutional layer.Feature extraction is shared across stages, while stage-specific layers produce joint heatmaps and later stages also consume previous heatmaps.Source: §3.1 · p. 4

3.2. Studio Setup and Camera Assumptions

• The system uses Human3.6M’s multi-camera studio data under a shared-scale scaled-orthographic assumption that accommodates different body sizes but restricts camera-dependent scale changes.The dataset uses a ten-camera Vicon studio with four video cameras, while the shared scale is stronger than the prior formulation’s assumption.Source: §3.2 · p. 4

3.3. Additional data

• Because Human3.6M has limited appearance and repetitive actions, the authors add actors performing the same actions and use publicly available unlabelled images from two men and one woman.This additional corpus is captured in the same environment and addresses concerns about evaluating data augmentation on a restrictive benchmark.Source: §3.3 · pp. 4–5
• The additional 3D data is used to augment two existing networks, improving published results and testing gains in both 2D joint localisation and 3D lifting.Separate experiments hold 3D lifting fixed for 2D improvements and train a generic lifter using the additional 3D data.Source: §3.3 · p. 5

3.4.1 Rotation marginalisation for improved stability

• Rotation marginalisation addresses flickering caused by ambiguous ground-plane rotations, making reconstructions more stable for visualisation and action analysis.The method marginalises over candidate rotations after estimating the optimal reconstruction for each fixed rotation.Source: §3.4.1 · p. 5

3.4.2 Principled shape warping for multiple views

• For multiple cameras, shape warping jointly estimates one pose close to the model and consistent with all calibrated views rather than averaging independently reconstructed single-view poses.The multi-view objective can be solved as an unconstrained least-squares problem for a fixed rotation, followed by rotation marginalisation.Source: §3.4.2 · pp. 5–6

4. Refining Existing Monocular Networks

• The method labels previously unlabelled data with multi-camera 3D pose estimates, motivated by the accuracy gains from using multiple cameras.The generated labels are used to augment training data for existing methods.Source: §4 · p. 6, Figure 2 · p. 6
• The preprocessing extracts person bounding boxes and silhouettes, then crops each image so joints remain inside a centred, square region with a 25-pixel margin.The cropped views are passed to the multi-camera network for 2D prediction and 3D reconstruction.Source: §4 · p. 6
• The predicted bounding boxes, 2D landmarks, and 3D reconstructions are treated like ground-truth training data when extending existing datasets.This makes the multi-camera estimator an additional source of supervision for monocular models.Source: §4 · p. 6

5. Experiments

• Protocol 1 averages unaligned Euclidean 3D joint error over 17 joints, whereas protocol 2 reports Procrustes-aligned per-joint error.The protocols use the same subjects but differ in evaluation alignment and frame selection.Source: §5 · p. 6
• Over 10 mm lower error than monocular methods and more than 4 mm better than Pavlakos et al. [21] under protocol 1.Table 1 compares monocular and multi-camera approaches on Human3.6M using the protocol 1 metric.Source: §5 · p. 6, Table 1 · p. 7
• The Huber-loss variant improves robustness by weighting joints case by case, with the best setting treating roughly half under ℓ1 and half under squared Frobenius loss.This replaces the squared Frobenius norm and adapts the effective penalty through ϵ.Source: §5 · p. 7
• Marginalising rotations yields a small improvement that primarily stabilises reconstructions rather than reducing overall error.Ground-truth orthographic triangulation separately measures error attributable to the camera model.Source: §5 · p. 7, Table 3 · p. 7

5.1. Improving Existing Monocular 3D Pose Networks

• Training existing monocular models with automatically labelled additional data produces a substantial 3D improvement for the Martinez et al. [16] setup, even without updating its 2D inputs.Because the original stacked-hourglass inputs were unavailable, the experiment used Tome et al. [35] 2D estimates and left hyperparameters unchanged.Source: §5.1 · p. 8, Table 4 · p. 8
• Retraining Tome et al. [35] using only the newly labelled data to update its 2D pose predictor significantly improves 2D error and correspondingly reduces 3D error.The 3D shape basis remains fixed in this experiment.Source: §5.1 · p. 8
• Figure 3 sorts sampled test-set reconstructions by 3D error and contrasts unaligned protocol 1 with ground-truth-aligned protocol 2 views.Ground truth is blue and reconstructions are red in both protocol rows.Source: Figure 3 · p. 8, §5.1 · p. 8

• The approach recovers from initial joint misdetections while still using image cues in subsequent stages.Source: §6 · p. 8
• It noticeably improves over existing multi-view markerless motion-capture systems.Source: §6 · p. 8
• Its output can label otherwise unlabelled data to improve existing methods.Source: §6 · p. 8