PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross View Fusion for 3D Human Pose Estimation

Haibo Qiu, Chunyu Wang, Jingdong Wang, Naiyan Wang, Wenjun Zeng

arXiv:1909.01203v1cs.CV

TL;DR

The paper addresses absolute 3D human pose estimation from multi-view images, where monocular methods typically recover poses centered on the pelvis rather than in world coordinates. It jointly estimates multi-view 2D poses with cross-view CNN fusion and recovers 3D poses using RPSM, achieving 26mm and 29mm errors on H36M and Total Capture.

  • Problem

    Monocular 3D pose methods typically estimate relative poses centered on the pelvis, leaving their absolute locations in the world coordinate system unknown.

  • Method

    The approach combines CNN-based multi-view feature fusion for 2D pose estimation with Recursive Pictorial Structure Model recovery of 3D poses from multi-view 2D poses.

  • Results

    26mm and 29mm Mean Per Joint Position Errors are achieved on H36M and Total Capture, respectively, outperforming the reported state-of-the-art comparisons.

  • Takeaways & Limitations

    The two contributions are independent and can each be combined with existing methods, while RPSM improves over PSM by a large margin.

  • Takeaways & Limitations

    Fusion weights depend on camera configurations, so a model trained for one configuration cannot be directly applied to another.

Abstract

from arXiv · show

We present an approach to recover absolute 3D human poses from multi-view images by incorporating multi-view geometric priors in our model. It consists of two separate steps: (1) estimating the 2D poses in multi-view images and (2) recovering the 3D poses from the multi-view 2D poses. First, we introduce a cross-view fusion scheme into CNN to jointly estimate 2D poses for multiple views. Consequently, the 2D pose estimation for each view already benefits from other views. Second, we present a recursive Pictorial Structure Model to recover the 3D pose from the multi-view 2D poses. It gradually improves the accuracy of 3D pose with affordable computational cost. We test our method on two public datasets H36M and Total Capture. The Mean Per Joint Position Errors on the two datasets are 26mm and 29mm, which outperforms the state-of-the-arts remarkably (26mm vs 52mm, 29mm vs 35mm). Our code is released at \url{https://github.com/microsoft/multiview-human-pose-estimation-pytorch}.

1. Introduction

The paper targets absolute 3D human pose recovery from multiple cameras, addressing errors in monocular relative poses and inaccurate multi-view 2D estimates. It combines cross-view 2D fusion with recursive 3D refinement to improve accuracy while controlling inference cost.

  • Absolute 3D pose estimation recovers human locations in the world coordinate system, unlike monocular methods centered on the pelvis.
  • 2D estimation errors from occlusion or motion blur can substantially challenge final 3D pose estimation.
  • PSM models spatial joint dependence but suffers quantization errors and O(N^6) inference cost when increasing grid resolution.A 2000mm space with N = 32 can produce a 30mm quantization error.
  • Cross-view fusion jointly estimates multi-view 2D poses in a CNN, integrating other views and supporting end-to-end training without intermediate supervision.The fusion network addresses correspondence between views during heatmap fusion.
  • RPSM recursively refines 3D pose estimates using finer grids around previous joint locations, reducing quantization cost compared with direct discretization.Each iteration uses a small number of bins, keeping single-iteration inference fast.
  • 26mm average 3D error is reported on H36M, compared with 52mm for a state-of-the-art method.Changing PSM to RPSM reduces average error from 77mm to 26mm.

2. Related Work

Prior multi-view methods optimize body models against image features or recover 3D poses from independently estimated views. This work emphasizes cross-view feature fusion to improve 2D heatmaps before 3D recovery.

  • Multi-view 3D Pose Estimation: Multi-view pose methods commonly represent the body with simple primitives and optimize parameters to align projected models with image features.
  • Multi-view 3D Pose Estimation: PSM-based approaches have been used for 2D and 3D pose estimation, including pipelines that estimate multi-view 2D poses before direct triangulation.
  • Multi-view 3D Pose Estimation: RPSM recursively refines 3D pose estimates step by step, while prior work cited here did not perform cross-view feature fusion.
  • Multi-image Feature Fusion: The paper distinguishes feature fusion from methods that use geometric consistency to select joint locations from imperfect heatmaps or as supervision.

3. Cross View Fusion for 2D Pose Estimation

The method jointly estimates multi-view 2D pose heatmaps in one CNN by fusing features across views using learned cross-view correspondences informed by epipolar geometry.

  • Multi-view images produce initial pose heatmaps that are fused across views so each view benefits from the others.The fusion process is performed within a single CNN and supports extension from two views to multiple views.
  • Each source-view feature connects to every pixel in the other view through learnable weights that encode correspondence strength.Ideally, weights are positive for corresponding pixels and zero elsewhere; the weights can be learned end-to-end as a neural-network layer.
  • Unknown depth makes exact cross-view pixel correspondence ambiguous, so each feature is fused with all features along its epipolar line.Non-corresponding locations are expected to contribute little because the correct location should have the strongest heatmap response.
  • The fusion network integrates with CNN-based 2D pose estimators end-to-end without intermediate supervision.Two training strategies were tested, and freely learning the weights produced similar results while being simpler.
  • The learned fusion weights depend on camera configurations, preventing direct application of a model trained on one configuration to another.The weights implicitly encode the epipolar geometry of the training camera setup.
  • An annotation-free adaptation procedure trains on pseudo-labels filtered by multi-view consistency in a new camera environment.Inconsistent joints are excluded from supervision when training the cross-view fusion network.

4. RPSM for Multi-view 3D Pose Estimation

RPSM recovers 3D pose with a graphical model whose unary terms use multi-view heatmaps and whose pairwise terms encode limb-length constraints, then recursively refines joint locations on finer grids.

  • 4.1. Pictorial Structure Model: The human body is represented by joint variables in 3D world coordinates connected by edges that encode physical dependence.The experimental graphical model contains 17 variables and 16 edges.
  • 4.1. Pictorial Structure Model: The initial PSM state space is a 2000mm bounding volume around the triangulated root joint, discretized into an N × N × N grid.All joints share this discrete state space, and inference maximizes the posterior using dynamic programming on the acyclic graph.
  • 4.1. Pictorial Structure Model: Unary potentials average projected joint confidence across camera views, while pairwise potentials favor limb lengths near training-set priors.The pairwise constraint uses a tolerance of 150mm in the experiments.
  • 4.2. Recursive Pictorial Structure Model: 30mm quantization error at N = 32 motivates recursive refinement instead of using a single large grid.Increasing N reduces quantization error but raises inference cost sharply, with complexity reported as O(N 6).
  • 4.2. Recursive Pictorial Structure Model: RPSM starts with a coarse grid and recursively searches smaller, joint-specific grids centered on the previous estimates.All joints are refined simultaneously while preserving their spatial relations, and the grid becomes finer as iterations proceed.
  • 4.2. Recursive Pictorial Structure Model: RPSM explores pose space through divide-and-conquer refinement, unlike bundle adjustment's incremental exploration.The paper also notes that finite-difference gradients in bundle adjustment are unstable when most heatmap entries are zero.

5. Datasets and Metrics

The evaluation uses cross-subject splits on H36M and Total Capture, with JDR for 2D detection and unaligned MPJPE for 3D accuracy.

  • H36M Dataset: H36M training uses subjects 1, 5, 6, 7, 8, while subjects 9 and 11 are held out for testing.A single fusion model is trained across subjects because their camera parameters are similar.
  • Total Capture Dataset: Total Capture evaluates general applicability using four cameras and separate training and testing sequences across subjects 1–5.IMU sensors are not used.
  • Metrics: JDR measures the percentage of joints whose estimated 2D locations fall within half a head size of ground truth.The threshold follows the MPII evaluation convention.
  • Metrics: MPJPE measures the mean Euclidean position error between ground-truth and estimated 3D joints without aligning the estimated poses.This evaluation is referred to as protocol 1.

6. Experiments on 2D Pose Estimation

The experiments evaluate cross-view fusion for 2D pose estimation on H36M using JDR, comparing training settings and baselines. Cross-view fusion improves detection across joints, especially occluded wrists, and the resulting 2D gains reduce triangulated 3D error.

  • Experimental setup: Table 1 reports H36M 2D pose accuracy using JDR (%) for six important joints under H36M and H36M + MPII training.The Single baseline does not perform cross-view feature fusion; sum and max epipolar-line baselines are also compared.
  • Quantitative results: Cross-view fusion outperforms Single on all body joints, with the largest wrist gains from 85.72% to 95.01% and from 89.33% to 97.20%.These results correspond to training on H36M and H36M + MPII, respectively.
  • Quantitative results: The fusion method also outperforms sum and max epipolar-line baselines by learning to handle noisy heatmaps, particularly under occlusion.The authors attribute its robustness to a trained fusion network rather than direct sum/max aggregation.
  • Generalization: With H36M-only training, cross-view fusion suffers less from limited appearance variation than Single, consistent with its geometric feature-consistency prior.The authors state this prior may reduce over-fitting when training data have limited appearance variation.
  • Effect on 3D estimation: Triangulated 3D MPJPE decreases from 36.28mm to 27.90mm without alignment, while wrist error decreases from 64.18mm to 34.28mm.The largest 2D wrist improvement is accompanied by a substantial reduction in its 3D estimation error.
  • Qualitative results: Heatmap examples show warped heatmaps from other views correcting ambiguous or occluded detections in the current view.The fused heatmap combines the current detected heatmap with heatmaps warped from the other three views.

7. Experiments on 3D Pose Estimation

Experiments show that recursive refinement and cross-view fusion improve absolute 3D pose estimation, especially when 2D detections are inaccurate or joints are difficult to observe. The method achieves strong results on H36M and generalizes to Total Capture.

  • RPSM implementation: 10 iterations of RPSM refine poses while taking about 0.4 seconds, using small per-iteration grids after the initial coarse estimate.The first iteration uses 16^3 bins; later iterations use 2 × 2 × 2 bins.
  • Ablation results: 94.54mm to 26.21mm: combining stronger 2D estimation and RPSM substantially reduces H36M 3D MPJPE.With already accurate 2D poses, RPSM alone changes 27.90mm to 26.21mm.
  • Ablation results: 40.47mm to 32.12mm: RPSM still improves the difficult “sitting” action despite accurate 2D poses making direct triangulation competitive.Overall, RPSM is comparable to triangulation with accurate 2D poses and better when 2D poses are inaccurate.
  • RPSM versus PSM: 95.23mm to 47.82mm: recursive refinement substantially lowers PSM error as the number of RPSM iterations increases.The improvement is attributed to progressively refining the pose after coarse discretization.
  • Single-view versus fusion: 88.69mm to 40.47mm: cross-view feature fusion most improves H36M performance on the challenging “Sitting” action.The reported gain is attributed to better 2D poses produced by fusion.
  • State-of-the-art comparison: 31.17mm versus 52.80mm: training only on H36M already outperforms the previous state of the art without Procrustes alignment.The approach also outperforms the state of the art on Total Capture, where it is evaluated using training data from that dataset.
  • Qualitative results: 20mm, 40mm, and 120mm: projected skeletons illustrate typical, larger-error, and rare severe-error cases across four camera views.The 40mm case involves a hand visible in only one view, while the 120mm case involves consistent double counting of a leg.

8. Conclusion

The paper proposes multi-view feature fusion for more accurate 2D poses and a recursive pictorial structure model for 3D recovery. These contributions improve over PSM and can be combined with existing methods.

  • 8. Conclusion: The approach estimates 3D human poses from multiple calibrated cameras using CNN-based multi-view feature fusion and recursive pictorial structure modeling.The two contributions are independent and each can be combined with existing methods.
Loading 1909.01203v1…