Source-linked AI summary
Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo
Jiahao Lin, Gim Hee Lee
TL;DR
Single-view 3D pose estimation is ill-posed because projection loses depth, while multi-person estimation across views faces the challenge of matching corresponding poses. This paper uses plane-sweep consistency to regress depths without explicit matching, and reports better benchmark performance and efficiency than prior approaches.
Problem
Single-view 3D pose estimation is ill-posed because projection loses depth, and multi-person estimation requires cross-view matching.
Source: §1 · pp. 1–2Method
The method aggregates pose-aware cross-view consistency with plane-sweep stereo and regresses person-level then joint-level relative depths without explicit matching.
Source: §5 · p. 8Results
On three benchmark datasets, the method outperforms prior state of the arts and is more efficient than existing works.
Source: §1 · p. 2Takeaways & Limitations
Only the range of virtual depth planes needs to be predefined for each camera view, making the method more generalizable when multi-camera settings are not known in advance.
Source: §1 · p. 2Matching approximates a pose by back-projecting all its joints to one depth plane; the authors report that this usually preserves nearest-pose retrieval.
Source: §3.1.1 · p. 4
Abstract
from arXiv · showhide
Existing approaches for multi-view multi-person 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views and solve for the 3D pose estimation for each person. Establishing cross-view correspondences is challenging in multi-person scenes, and incorrect correspondences will lead to sub-optimal performance for the multi-stage pipeline. In this work, we present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot. Specifically, we propose to perform depth regression for each joint of each 2D pose in a target camera view. Cross-view consistency constraints are implicitly enforced by multiple reference camera views via the plane sweep algorithm to facilitate accurate depth regression. We adopt a coarse-to-fine scheme to first regress the person-level depth followed by a per-person joint-level relative depth estimation. 3D poses are obtained from a simple back-projection given the estimated depths. We evaluate our approach on benchmark datasets where it outperforms previous state-of-the-arts while being remarkably efficient. Our code is available at https://github.com/jiahaoLjh/PlaneSweepPose.
1. Introduction
The paper replaces explicit cross-view matching with plane-sweep depth regression for multi-view, multi-person 3D pose estimation. It reports stronger benchmark performance and greater efficiency than prior approaches.
1. Introduction
- Multi-person 3D estimation is challenging because identities of 2D poses across camera views are unknown, making cross-view correspondence establishment critical.Traditional pipelines group corresponding detections before estimating each person's 3D pose.Source: §1 · p. 1
- VoxelPose avoids explicit matching but requires a predefined common 3D volume and incurs costly voxel back-projection and 3D convolution, especially in large or sparse scenes.The proposed method instead uses plane sweeps and 1D convolutions for each 2D pose.Source: §1 · p. 2
- Plane-sweep consistency across reference views supports coarse-to-fine depth regression, followed by back-projection to recover 3D poses.The method first estimates person-level depth, then joint-level relative depth.Source: Figure 1 · p. 1, §1 · p. 2
- On Campus, Shelf, and CMU Panoptic, the method outperforms prior state of the arts and is more efficient than VoxelPose.It also requires only a predefined virtual-depth-plane range, supporting use without prior knowledge of the multi-camera settings.Source: §1 · p. 2
2. Related Work
Related work covers multi-view methods for single-person pose estimation and the harder multi-person setting, where cross-view matching is a central challenge. Prior approaches use staged matching and reconstruction or avoid matching through volumetric detection.
2.1. Multi-view Single-person
- Single-person multi-view methods fuse features using epipolar geometry or canonical 3D space, learn triangulation weights, or use volumetric, weakly supervised, and meta-learning approaches.Source: §2.1 · p. 2
2.2. Multi-view Multi-person
- Multi-person methods address unknown identities across views with shared state spaces, graphical models, or staged pipelines that match 2D poses before reconstructing 3D poses.Source: §2.2 · p. 2
- Staged methods use epipolar, appearance, or feet-joint affinities for matching; VoxelPose avoids explicit matching through volumetric 3D detection but its 3D convolution is computationally expensive.Source: §2.2 · p. 3
- This work uses plane-sweep stereo and pose-aware geometric consistency to regress pose depths without explicit correspondences, aiming for higher precision and greater efficiency.Source: §2.2 · p. 3
3. Our Method
The method replaces explicit cross-view pose matching with plane-sweep depth regression for candidate poses, first estimating person-level depth and then joint-relative depth for 3D back-projection.
3. Our Method
- Plane-sweep depth regression uses multi-view 2D poses to estimate each candidate pose’s depth, then back-projects the poses to reconstruct 3D.The person-level depth is the center hip joint’s depth; joint-level relative depth is estimated afterward.Source: §3 · p. 4, §3.1 · p. 4
3.1.1 Multi-View Score Aggregation
- At each of 64 depth planes, projected target poses are matched to nearby reference-view poses; joint alignment scores are confidence-weighted across reference views.A larger joint distance yields a lower score, indicating less support for that depth.Source: §3.1.1 · p. 4
- Projecting all joints at one shared depth is an approximation, but the authors report it usually retrieves the nearest reference pose and works sufficiently well.This treats the pose as flat during reference-view matching.Source: §3.1.1 · p. 4
3.1.2 Depth Regression
- A residual 1D-CNN regresses person-level depth from the score matrix, and local soft-argmax selects a high-response window to avoid interference from multiple modes.The implementation uses a window of 16 depth layers; the ordinary soft-argmax can fail when the response is multimodal.Source: §3.1.2 · pp. 4–5, Figure 3 · p. 5
3.2.1 Score Aggregation
- After person-level localization, joint-level score matrices use relative-depth planes spanning −1000 to +1000 mm, with 64 planes in the implementation.Training substitutes ground-truth person-level depth for the estimate to stabilize this stage.Source: §3.2.1 · p. 5
3.2.2 Depth Regression
- A 1D-CNN with dilated convolutions expands the depth receptive field for joint-depth reasoning; standard soft-argmax obtains each joint’s relative depth, and absolute joint depth is used to back-project the 2D pose.Source: §3.2.2 · pp. 5–6
- The coarse-to-fine scheme reduces computation with sparse planes in the broad first-stage range, while joint-relative estimates compensate for small person-depth offsets.The first-stage person depth therefore need not be highly precise for accurate final 3D pose estimation.Source: §3.2.2 · p. 6
3.3. Training Details
- Training uses perturbed projections of MoCap skeletons; inference estimates poses from each target view, fuses them globally, then clusters and averages nearby duplicates.Confidence scores for simulated 2D joints depend on the amount of coordinate perturbation.Source: §3.3 · p. 6
4. Experiments
Across Campus, Shelf, and Panoptic, the approach improves benchmark pose results and reaches up to 20× faster inference than VoxelPose. Ablations examine depth-plane counts, camera counts, and generalization across camera settings.
4.1. Datasets and Metrics
- Campus and Shelf use pretrained HRNet with synthesized-data depth training, while Panoptic fine-tunes HRNet; PCP is used for Campus/Shelf and AP and MPJPE for Panoptic.Shelf evaluation excludes one person occluded in most frames; Campus uses frames 350–470 and 650–750.Source: §4.1 · p. 6
4.2. Ablation Study
- Local soft-argmax improves results on both Campus and Shelf over standard soft-argmax, especially on Campus, where views typically contain two to three people.The local operation focuses on the highest-response mode without interference from other modes in a multimodal distribution.Source: §4.2 · p. 7
4.3. Inference Speed
- Inference reaches up to 20× the frame rate of VoxelPose on Campus and Shelf, with 2D pose-estimation runtime excluded.The comparison uses the same 2D pose estimates on one GTX 1080 Ti; the authors attribute the speed advantage partly to target-pose-focused 1D convolutions rather than 3D convolutions over voxels.Source: Table 3 · p. 7, §4.3 · pp. 7–8
4.2. Ablation Study
- Reducing person-level depth planes from 64 to 16 lowers pose accuracy by 0.4%, while reducing joint-level planes from 64 to 16 lowers it by 0.7%.The authors attribute the larger joint-level drop to increased quantization error; coarse person-level depth is sufficient for rough localization.Source: §4.2 · p. 7
- Reducing the number of cameras lowers performance, but accuracy remains above 92% with only two cameras.The authors note that severe occlusions in the indoor dataset can remain ambiguous even with multiple cameras.Source: §4.2 · p. 7
- Training on randomly sampled camera viewpoints performs equally well to training and evaluation on the same five dataset cameras.The method is evaluated on the five dataset cameras after training with randomly sampled viewpoints.Source: §4.2 · p. 7
4.4. Comparison to the State-of-the-arts
- Average accuracy improves from 96.7% to 97.0% on Campus and from 97.4% to 97.9% on Shelf; Panoptic error falls by ∼1 mm versus VoxelPose.The reported Campus and Shelf values are average accuracy; the Panoptic comparison is an error reduction.Source: Table 4 · p. 8, Table 5 · p. 8, §4.4 · p. 8
- The method and VoxelPose use geometric rather than photometric consistency; Figure 5 shows Shelf pose projections and 3D estimates against ground truth.The figure presents image-view projections on the left and estimated versus ground-truth 3D poses on the right.Source: §4.4 · p. 8, Figure 5 · p. 8
5. Conclusion
The conclusion presents plane-sweep depth regression as a way to estimate multi-view multi-person 3D poses without explicit cross-view matching. It uses coarse-to-fine depth estimation and is reported as more efficient and more accurate than prior state-of-the-art methods.
5. Conclusion
- Plane sweep aggregates multi-view information through pose-aware geometric consistency to regress target-view pose depths without explicit cross-view correspondences.The conclusion frames this as the paper's approach to multi-view multi-person 3D pose estimation.Source: §5 · p. 8
- Depth estimation proceeds coarse to fine, first regressing person-level depth and then per-person joint-level relative depth.Source: §5 · p. 8
- The framework is reported to be more computationally efficient and to show superior performance compared with previous state-of-the-art methods.Source: §5 · p. 8