Source-linked AI summary
Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation
Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, Robert Wang
TL;DR
Multi-view 3D pose estimation can be computationally costly when methods use volumetric grids, limiting its fit for low-compute headsets. This paper fuses views in a camera-disentangled latent representation and lifts camera-conditioned 2D detections with efficient differentiable DLT. Across two datasets, it reports better or comparable accuracy than state-of-the-art methods while running in real time.
Problem
Volumetric multi-view pose methods are computationally expensive, while low-compute headsets need lightweight approaches to use integrated cameras.
Source: §1 · p. 1Method
The method fuses camera-transformed image features into a camera-disentangled latent pose representation, decodes per-view 2D detections, and lifts them with differentiable DLT.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach combines real-time multi-view pose estimation with flexibility to unseen camera settings and negligible computational overhead over monocular methods.
Source: §1 · p. 2, §5 · p. 8The method assumes spatially calibrated and temporally synchronized cameras capturing one individual.
Source: §3 · p. 2
Abstract
from arXiv · showhide
We present a lightweight solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. Building upon recent advances in interpretable representation learning, we exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. This allows us to reason effectively about 3D pose across different views without using compute-intensive volumetric grids. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections, that can be simply lifted to 3D via a differentiable Direct Linear Transform (DLT) layer. In order to do it efficiently, we propose a novel implementation of DLT that is orders of magnitude faster on GPU architectures than standard SVD-based triangulation methods. We evaluate our approach on two large-scale human pose datasets (H36M and Total Capture): our method outperforms or performs comparably to the state-of-the-art volumetric methods, while, unlike them, yielding real-time performance.
1. Introduction
Multi-view pose could suit multi-camera VR/AR headsets, but current volumetric approaches are computationally expensive. The paper instead fuses views in a camera-independent latent representation and uses efficient differentiable triangulation.
1. Introduction
- Multi-view pose is relevant to VR/AR headsets with integrated cameras, but existing volumetric networks and pictorial structures are computationally expensive and may not fit low-compute headsets.The paper targets this lightweight-computing constraint.Source: §1 · p. 1
- The method maps image features into canonical coordinates, fuses them into a camera-disentangled pose representation, then conditions that representation on camera transforms to decode per-view 2D locations.The resulting 2D detections are lifted to 3D with differentiable DLT; the GPU-friendly implementation uses Shifted Iterations instead of standard SVD.Source: §1 · pp. 1–2
- The proposed GPU-friendly DLT implementation is hundreds of times faster than standard implementations.Standard DLT implementations relying on SVD are rarely efficient on GPUs.Source: §1 · p. 2
- On Human3.6M and TotalCapture, the method outperforms state of the art without extra training data and is comparable with extra 2D annotations, while running in real time and handling unseen viewpoints.Figure 1 contrasts the volumetric approaches with the proposed lightweight architecture.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
Earlier multi-view pose work ranged from hand-crafted body models to deep systems that either estimate poses per view or aggregate views volumetrically. The paper situates its approach among lightweight fusion methods and triangulation-based 3D recovery.
2. Related Work
- Early multi-view methods matched hand-crafted image features with parametric body models, while deep-learning approaches shifted toward separate 2D pose estimation per view.The early methods had limited success outside controlled settings.Source: §2 · p. 2
- Volumetric methods aggregate 2D predictions in 3D grids, including Pictorial Structures and learned 3D convolutional networks, but remain computationally demanding.Simple triangulation remains the de-facto standard when real-time performance is sought.Source: §2 · p. 2
- Lightweight methods that fuse views without volumetric grids concatenate 2D detections or refine heatmaps, but lack camera information and overfit to a specific camera setting.The paper contrasts this with its ability to handle different and unseen camera settings.Source: §2 · p. 2
- Triangulation recovers a 3D point from its images across calibrated views; the paper uses differentiable DLT and proposes a GPU-friendly implementation up to two orders of magnitude faster than SVD-based versions.The method is simple and differentiable.Source: §2 · p. 2
- Other methods back-propagate through SVD-based triangulation, but the cited approaches do not explicitly reason about multi-view inputs and therefore struggle with large self-occlusions.These methods lift 2D detections to 3D keypoints.Source: §2 · p. 2
3. Method
The method assumes synchronized, calibrated multi-view images and estimates absolute 3D joint coordinates. It maps image features into a shared canonical frame, fuses them, decodes per-view 2D detections, and triangulates them with DLT.
3.1. Lightweight pose estimation from multi-view inputs
- The baseline decodes each view independently, while Canonical Fusion uses camera transforms to align and fuse features into a view-independent pose representation.The shared representation lets the convolutional block focus on reasoning about occlusion and supports flexibility across cameras, including unseen ones.Source: §3.1 · p. 3, §3.2 · pp. 3–4
3.2. Learning a view-independent representation
- Camera projection matrices map features into a common canonical representation, where shallow 1D convolutions fuse them into a unified 3D-pose representation.Feature Transform Layers are parameter-free and computationally efficient; they reshape, transform, and restore feature maps.Source: §3.2 · pp. 3–4
3.3. Decoding latent codes to 2D detections
- A shallow convolutional decoder maps camera-conditioned features to per-view joint heatmaps, which are integrated across spatial axes into 2D locations.The heatmap integration is differentiable, allowing gradients to flow through the 2D detections.Source: §3.3 · p. 4
3.4. Efficient Direct Linear Transformation
- DLT combines 2n linear equations from n calibrated views to estimate a 3D point; under noisy observations, it seeks an approximate solution.The authors use a bound on the smallest singular value under Gaussian observation noise to guide shifted inverse iterations.Source: §3.4 · pp. 4–5
- For 2D errors up to 70 pixels, two iterations match full SVD accuracy while running 10/100 times faster on CPU/GPU.The comparison uses the proposed shifted-inverse-iteration DLT implementation and an SVD-based one.Source: §3.4 · p. 5
3.5. Loss function
- In their experiments, models are pre-trained by minimizing L2D-MPJPE for 20 epochs, then fine-tuned by minimizing 3D MPJPE, which is also the test metric.The DLT implementation is differentiable with respect to 2D joint locations, allowing gradients with respect to 3D landmarks to flow back to the input images.Source: §3.5 · p. 5
4. Experiments
The experiments evaluate the method on TotalCapture and Human3.6M, including tests of camera disentanglement, unseen-view generalization, accuracy, and inference efficiency. Results show strong benchmark accuracy and real-time performance, including under an additional-data setup.
4. Experiments
- The evaluation uses ground-truth performer crops resized to 256×256, a ResNet152 encoder, and 50 training epochs.Training also uses random ±30-degree rotations and color augmentation.Source: §4 · p. 5
4.1. Datasets specifications
- TotalCapture uses four cameras for training and testing, with an additional experiment training on cameras (1,3,5,7) and testing on (2,4,6,8); Human3.6M tests on unseen subjects 9 and 11.The datasets provide marker-based 3D joint locations; TotalCapture’s test performances are unseen during training.Source: §4.1 · p. 6
4.2. Qualitative evaluation of disentanglement
- Decoded poses remain consistent when synthesized from both seen z-axis rotations and unseen camera projections, supporting camera-viewpoint disentanglement.The unseen projections are produced by rotating a camera around the plane through two consecutive camera views.Source: §4.2 · p. 6, Figure 5 · p. 6
4.3. Quantitative evaluation on TotalCapture
- On TotalCapture, Canonical Fusion beats volumetric [25] by ~5%, with 19% improvement from Fusion, a further 10% from Canonical Fusion (no DLT), and 3% from DLT training.Table 1 reports MPJPE (mm) without rigid alignment or offline post-processing for the authors’ methods.Source: Table 1 · p. 7, §4.3 · p. 7
4.4. Generalization to unseen cameras
- For unseen TotalCapture cameras, Fusion loses 47.1mm in accuracy, while Canonical Fusion’s drop is similar to the Baseline’s ~10mm.The evaluation discards frames where the performer is out of view for the Baseline, favoring that comparison method.Source: Table 2 · p. 7, §4.4 · p. 7
4.5. Quantitative evaluation on Human 3.6M
- On Human3.6M without additional data, Canonical Fusion (no DLT) outperforms [25] by ~15%, and the best model outperforms all other compared multi-view approaches.The authors note that [17] cannot be compared in this setup because it reports no results without additional data.Source: Table 3 · p. 8, §4.5 · p. 8
4.6. Exploiting additional data
- With additional 2D training data, Canonical Fusion reaches accuracy comparable to volumetric [17], beats [25] by ~20%, and runs ~57 times faster than [17] Volumetric.The comparison reports performance, inference time, and model size on Human3.6M; the speed comparison is for the unoptimized models.Source: Table 4 · p. 8, §4.6 · p. 8
5. Conclusions
The paper concludes that geometry-aware multi-view fusion can support real-time 3D pose estimation without volumetric grids. It reports strong benchmark performance without additional data and comparable accuracy to more computationally intensive volumetric methods when additional 2D annotations are used.
5. Conclusions
- The proposed fusion reasons across multi-view geometry with negligible computational overhead over monocular methods.Combined with the DLT formulation, it yields a real-time multi-camera 3D pose approach.Source: §5 · p. 8
- The approach reports state-of-the-art performance on standard benchmarks without additional data and flexibility to unseen camera settings.Source: §5 · p. 8
- With additional 2D annotations, its accuracy is comparable to more computationally intensive volumetric methods.Source: §5 · p. 8
7. Supplementary Material
The supplementary material details the architectures, efficient DLT implementation, and feature transform layers, then provides additional prediction visualizations.
7.1. Architectures
- Canonical fusion transforms each view’s features into a shared world-coordinate representation, fuses them, then projects the unified features back to each view; unlike Fusion, all views are reconstructed from one latent representation.The compared networks share a ResNet152 encoder and convolutional decoder; their feature-fusion modules differ.Source: §7.1 · p. 10
7.2. Efficient Direct Linear Transformation
- The DLT implementation uses shifted inverse iterations to find the needed singular vector; two iterations sufficed, and µ=0.001 avoids numerical instability near singular matrices.The stated perturbation analysis assumes i.i.d. Gaussian noise in 2D observations; the supplementary discussion attributes GPU inefficiency of SVD to two-sided transformations and their data dependencies.Source: §7.2 · pp. 10–12
7.3. Feature Transform Layer
- Feature transform layers linearly transform latent features using a transform matrix, preserving the input transform’s structure; here they explicitly encode camera transformations for camera-disentangled pose representations.The supplementary text describes rotations as invertible, norm-preserving choices for the transform matrix.Source: §7.3 · p. 12
7.4. Additional results
- Additional figures compare baseline, Fusion, and Canonical Fusion architectures and show uncurated Human3.6M and TotalCapture predictions, including TotalCapture’s seen- and unseen-camera settings.Figure 7 illustrates mapping a factor of variation onto a circle in feature space.Source: §7.4 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14