Source-linked AI summary
Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views
Junting Dong, Wen Jiang, Qixing Huang, Hujun Bao, Xiaowei Zhou
TL;DR
Multi-person 3D pose estimation from a few calibrated views requires matching noisy 2D detections across cameras, while prior 3D reasoning is computationally expensive and less robust. The paper clusters cross-view 2D poses with multi-way matching, combines geometric and appearance cues, and reports improved performance with reduced state space and efficient reconstruction.
Problem
Multi-person 3D pose estimation from a few calibrated cameras requires finding correspondences among noisy, incomplete 2D detections, a challenge complicated by prior methods’ huge 3D state spaces and geometry-only matching.
Method
The approach uses cycle-consistent multi-way matching to cluster same-person 2D poses across views, combining geometric and appearance cues before separately inferring each person’s 3D pose.
Results
The approach improves multi-person 3D pose estimation performance while reducing the 3D pictorial-structure state space and enabling reliable reconstruction from clustered 2D poses.
Takeaways & Limitations
Cross-view grouping makes the problem more efficient and robust, while appearance information and cycle consistency support matching across multiple views.
Takeaways & Limitations
Prior geometry-only approaches are sensitive to camera setup and degrade significantly as the number of views decreases, motivating the paper’s appearance cues.
Abstract
from arXiv · showhide
This paper addresses the problem of 3D pose estimation for multiple people in a few calibrated camera views. The main challenge of this problem is to find the cross-view correspondences among noisy and incomplete 2D pose predictions. Most previous methods address this challenge by directly reasoning in 3D using a pictorial structure model, which is inefficient due to the huge state space. We propose a fast and robust approach to solve this problem. Our key idea is to use a multi-way matching algorithm to cluster the detected 2D poses in all views. Each resulting cluster encodes 2D poses of the same person across different views and consistent correspondences across the keypoints, from which the 3D pose of each person can be effectively inferred. The proposed convex optimization based multi-way matching algorithm is efficient and robust against missing and false detections, without knowing the number of people in the scene. Moreover, we propose to combine geometric and appearance cues for cross-view matching. The proposed approach achieves significant performance gains from the state-of-the-art (96.3% vs. 90.6% and 96.9% vs. 88% on the Campus and Shelf datasets, respectively), while being efficient for real-time applications.
1. Introduction
The paper targets multi-person 3D pose estimation from a few calibrated cameras, where noisy, incomplete 2D observations make cross-view correspondence difficult. It reduces this difficulty by grouping same-person 2D poses with multi-way matching, combining geometric and appearance cues before reconstructing poses.
- Multi-person 3D pose estimation from a few calibrated cameras is challenging because people interact in crowded scenes with significant occlusions.
- Existing 3D pictorial-structure methods are computationally expensive because they reason over a huge joint state space and are less robust with few cameras.
- Grouping detected 2D poses belonging to the same person reduces the state space and improves the efficiency and robustness of 3D pose estimation.
- The multi-way matching algorithm establishes cycle-consistent cross-view correspondences, prunes false detections, handles partial overlaps, and does not require the number of people.
- Combining geometric consistency with appearance similarity reduces matching ambiguities, and appearance information is important for linking 2D detections across views.
- The approach outperforms state-of-the-art methods by a large margin without using training data from the evaluated datasets.
2. Related work
Prior work reconstructs multi-person 3D pose mainly with 3D pictorial-structure models or pairwise 2D matching, but these approaches face large computation or inconsistent and geometry-sensitive correspondences. The paper instead uses multi-way matching within a pipeline built on detected 2D poses.
- Earlier tracking-based methods require first-frame initialization and are prone to local optima and tracking failures.
- 3D pictorial-structure methods jointly model all people and joints, creating a huge state space and heavy inference computation.
- These 3D pictorial-structure methods rely only on multi-view geometry, making performance sensitive to camera setup and fewer views.
- Pairwise epipolar matching can ignore cycle consistency and produce inconsistent correspondences across multiple views.
- The pipeline adopts a Cascaded Pyramid Network as an initial multi-person 2D pose detector, reflecting the general accuracy–speed distinction between top-down and bottom-up methods.
- Single-view 3D pose methods have reconstruction ambiguity that makes their accuracy incomparable with multi-view results.
- The approach uses person re-identification and cycle-consistency results from multi-image matching to support cross-view correspondence.
3. Technical approach
The approach first matches detected 2D bounding boxes across views using appearance and geometric affinities with globally cycle-consistent multi-way matching. It then reconstructs each person’s 3D pose separately using a reduced proposal-based 3D pictorial structure.
- Pipeline overview: The pipeline detects 2D bounding boxes and keypoints, matches boxes across views, discards unmatched false detections, and reconstructs each person’s 3D pose.The matching stage precedes 3D reconstruction and operates on noisy detections.
- Affinity matrix: Appearance descriptors from a person re-identification network are combined with epipolar geometric compatibility to score cross-view bounding-box affinities.The appearance descriptor is extracted from the cropped bounding box, while geometric consistency is measured from corresponding 2D joint locations and epipolar lines.
- Multi-way matching: The matching algorithm solves all views simultaneously under cycle consistency, avoiding inconsistent pairwise correspondences.Cycle-consistent correspondences form closed cycles across views; pairwise matching alone can violate this constraint.
- Optimization: A convex optimization formulation uses a low-rank, positive-semidefinite correspondence matrix, with nuclear-norm relaxation and symmetric box-constrained variables for tractability.The unknown rank represents the number of people best explaining the observations, while the nuclear norm serves as a convex surrogate for rank.
- Optimization: The method requires no known person count because rank minimization automatically estimates the number of people represented by the detections.Cycle consistency aggregates multi-view information and helps prune false detections.
- 3D pose reconstruction: For reconstruction, 3D joint states are triangulated from corresponding 2D joints across view pairs instead of sampled on a 3D grid.Correctly detected joints in two views ensure that the true 3D location is included among the proposals, reducing inference cost without sacrificing accuracy.
4. Empirical evaluation
The evaluation uses three public datasets and ablations to test matching cues, reconstruction choices, efficiency, and comparison with prior methods. Results show benefits from combining appearance and geometry, using matching before 3D inference, and retaining 3DPS when views are sparse or occlusion is severe.
- Datasets: The evaluation covers Campus, Shelf, and CMU Panoptic, with quantitative 3D PCP on the first two and qualitative evaluation on the third.Campus has three people and three calibrated cameras; Shelf has four people, five cameras, and heavy occlusion; CMU Panoptic lacks ground truth.
- Ablation analysis: Combining appearance and geometry improves Shelf performance when appearance alone confuses visually similar people.Appearance alone is competitive on Campus, whereas geometry alone suffers with distant cameras; their combination improves the difficult Shelf case.
- Ablation analysis: 3DPS greatly improves performance with few cameras and can help when a person is heavily occluded.The paper gives Campus with three cameras and Shelf actor 2 as examples.
- Ablation analysis: Matching before 3DPS reduces the state space and makes inference much faster than applying 3DPS directly to all detections.Without matching, performance is much worse on Campus because sparse-view geometry alone can produce false correspondences.
- Comparison with state-of-the-art: The approach outperforms prior methods without fine-tuning on the evaluated datasets, including severe-occlusion cases.Direct triangulation from robust, consistent matches also outperforms previous methods.
- Qualitative evaluation: Qualitative results show correspondences from inaccurate detections, automatic identification of the number of people, and more accurate projected 2D poses.Figure 4 displays detections, cross-view correspondences, and projections of estimated 3D poses.
- Efficiency: The implementation averages 25 ms for reID and affinity construction, 20 ms for matching, and 60 ms for 3D inference.The version without 3DPS achieves real-time performance at > 20fps on four-person, five-view Shelf sequences.
5. Summary
The paper presents a fast and robust multi-person 3D pose estimator for few-camera settings. It clusters detected 2D poses across views with multi-way matching, then reconstructs each person separately, combining geometric and appearance cues with cycle consistency.
- Summary: The approach clusters detected 2D poses belonging to the same person across views, reducing the 3DPS state space for multi-person inference.It can also reconstruct 3D poses from clustered detections by triangulation without 3DPS.
- Summary: Multi-way matching combines geometric and appearance cues with cycle consistency to establish correspondences across multiple views.The matching formulation supports globally consistent correspondences across views.
- Summary: The method is designed to recover 3D poses of crowds quickly and robustly from a few cameras.The paper reports effectiveness of the matching strategy for multi-view pose reconstruction.