Source-linked AI summary
Learnable Triangulation of Human Pose
Karim Iskakov, Egor Burkov, Victor Lempitsky, Yury Malkov
TL;DR
Multi-view 3D human pose estimation from few camera views remains an important but comparatively understudied challenge. The paper introduces learnable algebraic and volumetric triangulation methods, with the volumetric approach achieving state-of-the-art performance and about 30% additional error reduction over RANSAC.
Problem
Accurate multi-view 3D human pose estimation from few views remains an important, comparatively understudied challenge with direct practical applications.
Method
The paper proposes differentiable learnable triangulation using confidence-weighted algebraic triangulation and volumetric aggregation of multi-view 2D features refined into 3D pose heatmaps.
Results
About 30% additional error reduction over RANSAC was achieved by the volumetric methods, which attained state-of-the-art performance and cross-dataset generalization.
Takeaways & Limitations
Learnable triangulation can drastically reduce the number of views needed for accurate pose estimation and transfer across camera setups and datasets.
Takeaways & Limitations
The volumetric approach relies on algebraic triangulation and therefore requires at least two camera views observing the pelvis.
Abstract
from arXiv · showhide
We present two novel solutions for multi-view 3D human pose estimation based on new learnable triangulation methods that combine 3D information from multiple 2D views. The first (baseline) solution is a basic differentiable algebraic triangulation with an addition of confidence weights estimated from the input images. The second solution is based on a novel method of volumetric aggregation from intermediate 2D backbone feature maps. The aggregated volume is then refined via 3D convolutions that produce final 3D joint heatmaps and allow modelling a human pose prior. Crucially, both approaches are end-to-end differentiable, which allows us to directly optimize the target metric. We demonstrate transferability of the solutions across datasets and considerably improve the multi-view state of the art on the Human3.6M dataset. Video demonstration, annotations and additional materials will be posted on our project page (https://saic-violet.github.io/learnable-triangulation).
1. Introduction
Multi-view 3D pose estimation is motivated by its value for obtaining in-the-wild monocular-pose ground truth and for real-time tracking with practical few-camera setups. The paper addresses this underexplored problem with two learnable-triangulation methods designed to reduce the views needed for accurate 3D pose and evaluated across Human3.6M and CMU Panoptic.
- Motivation: Multi-view pose estimation can provide in-the-wild ground truth for monocular 3D pose estimation [5] [22], while marker-based and visual-inertial alternatives have limitations.Those limitations include restricted pose representations and clothing constraints, while prior multi-view dataset construction required impractically many views [5] [22].
- Motivation: Few-camera multi-view systems can directly support real-time human-pose tracking as multi-camera setups become more available in applications such as sports and computer-assisted living.Practical systems rarely use more than a few views, motivating methods suited to this regime.
- Contribution: The paper argues that multi-view pose estimation is underexplored and proposes two related learnable-triangulation methods that reduce the views needed for accurate 3D pose.Both methods are trained using marker-based motion-capture or “meta”-ground-truth supervision.
- Experiments: Experiments on Human3.6M and CMU Panoptic [5] report state-of-the-art accuracy and cross-dataset generalization for the proposed methods.The evaluation covers both accuracy and transferability across datasets.
2. Related work
Prior work separates single-view 3D pose estimation into 2D-to-3D lifting and direct image-based prediction, while multi-view methods use coordinate fusion, volumetric representations, or iterative 3D pose-prior refinement. The proposed approach differs from volumetric predecessors through learnable volume processing and end-to-end training.
- Single view 3D pose estimation: Single-view methods either lift high-quality 2D estimates to 3D with deep networks or predict 3D pose directly from images using volumetric representations [8] [16].The lifting approach is simple, fast, and can use motion-capture training with skeleton and view augmentation.
- Multi-view 3D pose estimation: Multi-view research commonly targets ground-truth annotation generation, while another approach concatenates 2D joints from all views for fully connected prediction of global 3D coordinates [14] [5] [6].The coordinate-fusion approach uses information from different views and can be trained on motion-capture data.
- Multi-view 3D pose estimation: Earlier multi-view volumetric methods unprojected pretrained 2D keypoint heatmaps into volumes and aggregated them non-learnably [12, 5].The proposed work instead processes information inside the volume learnably and trains the network end-to-end, adjusting the 2D backbone without requiring interpretable heatmaps.
- Multi-view 3D pose estimation: A separate multi-stage method used an external 3D pose prior [17] [18], repeatedly reprojecting 3D pose into camera views and fusing it with predictions to re-estimate the pose.It first extracts 2D joint heatmaps from all views, optimizes latent coordinates in 3D pose-prior space, and then iterates reprojection and convolutional fusion.
3. Method
The method estimates framewise global 3D joint positions from synchronized multi-camera video with known projection matrices using two differentiable triangulation approaches: confidence-weighted algebraic triangulation and volumetric aggregation with 3D refinement.
- Method overview: The system processes synchronized camera frames independently in time to estimate global 3D positions for a fixed set of human joints.It assumes known camera projection matrices and a single person in the scene.
- Algebraic triangulation: The algebraic baseline extracts 2D joint positions with differentiable soft-argmax and triangulates them by solving a projection-based homogeneous linear system.An inverse temperature α = 100 makes soft-argmax initially approximate heatmap maxima while preserving gradient flow.
- Algebraic triangulation: Learned camera-joint weights adapt each view’s contribution to triangulation, addressing unreliable occluded views without RANSAC’s risk of cutting off excluded-camera gradients.The weights are predicted by a convolutional branch from intermediate backbone features and learned jointly with the detector.
- Volumetric triangulation: Lifting features into 3D spatially isolates incorrect predictions, while incorporating camera parameters, modeling pose priors, and handling multimodal 2D detections.The volumetric representation is designed to overcome independent per-camera processing and support convolutional filtering of inconsistent evidence.
4. Experiments
Experiments on Human3.6M and CMU Panoptic show that learnable volumetric triangulation substantially outperforms algebraic and RANSAC baselines, remains effective with fewer views, and transfers across datasets. The approach also supports monocular estimation and benefits from its learned 3D pose prior under occlusion.
- Human3.6M: Absolute MPJPE on Human3.6M is much smaller than the 3.9 cm voxel size, highlighting the importance of subpixel-resolving soft-argmax.The absolute-coordinate evaluation excludes scenes with erroneous ground-truth annotations and is reported in Table 2.
- Human3.6M: With a single camera, the volumetric method achieves 49.9 mm error without tuning after training with a random number of cameras from one to four.For monocular estimation, training uses L1 loss on joint positions relative to the pelvis.
- CMU Panoptic: Using only four cameras, volumetric triangulation surpasses the RANSAC accuracy obtained with 28 cameras on CMU’s meta-ground-truth.The error-versus-camera-count experiment samples random camera subsets and demonstrates reduced camera requirements for real-world setups.
- CMU Panoptic: On CMU Panoptic, volumetric triangulation has a dramatic advantage over algebraic triangulation, especially under strong occlusions and missing body parts.The authors attribute this advantage to the learned 3D pose prior; comparisons use absolute MPJPE on four-camera validation scenes.
- Cross-dataset transfer: A CMU-trained model transfers to Human3.6M with 36 mm MPJPE for learnable triangulation and 34 mm for volumetric triangulation.These errors are higher than the 16–18 mm obtained by methods trained directly on Human3.6M, but are described as reasonable for cross-dataset generalization.
5. Conclusion
The paper presents two learnable-triangulation methods achieving state-of-the-art multi-view 3D human pose estimation on Human3.6M. They reduce the views needed for high accuracy and produce smooth CMU Panoptic pose sequences without temporal processing, but support only single-person scenes and require pelvis visibility in at least two views for volumetric triangulation.
- Conclusion: The two proposed learnable-triangulation methods achieve state-of-the-art performance on the Human3.6M dataset.The methods are presented as novel solutions for multi-view 3D human pose estimation.
- Conclusion: The methods drastically reduce the views needed for high accuracy and produce smooth CMU Panoptic pose sequences without temporal processing.The authors suggest this smooth output could potentially improve CMU Panoptic ground-truth annotation.
- Conclusion: The approach supports only a single person, while volumetric triangulation requires at least two camera views observing the pelvis.The single-person limitation may be mitigated with ReID applied to 2D human detections, but the volumetric method's dependence on algebraic-triangulation predictions remains a constraint.