PaperlayerView exact revision on arXiv

Source-linked AI summary

MEOM: Multi-View Expected-OKS Maximization for Human Pose Triangulation

Ziliang Xiong, Henglin Shi, Per-Erik Forssen

arXiv:2608.30521v1cs.CV

TL;DR

Multi-view 3D pose triangulation can fail when multimodal heatmaps are collapsed to single peaks, motivating methods that preserve spatial probability mass and assess its reliability. The paper introduces MEOM for mass-based multi-view fusion and HDR calibration for diagnostics, supporting both label-free refinement and supervised end-to-end training. It reports competitive or improved performance, including 19.11 mm absolute MPJPE on Human3.6M at roughly half the inference cost of a volumetric approach.

  • Problem

    Collapsed heatmap peaks make triangulation unreliable under multimodality and occlusion, while robust fusion and reliability assessment of full heatmap distributions remain needed.

  • Method

    MEOM aggregates expected-OKS responses across camera views to select 3D joints agreeing in probability mass, while HDR calibration diagnoses heatmap reliability.

  • Results

    19.11 mm absolute MPJPE is achieved on Human3.6M, outperforming the state-of-the-art volumetric approach at half the inference cost; label-free MEOM is especially advantageous on ambiguous and occluded frames.

  • Takeaways & Limitations

    MEOM provides a mass-based algebraic alternative that remains competitive with methods using larger backbones, temporal fusion, simulated 3D data, or volumetric triangulation.

  • Takeaways & Limitations

    Pure MEOM supervision can under-constrain absolute 3D scale and geometry, so the method retains a 3D MSE term.

Abstract

from arXiv · show

Conventional algebraic triangulation solves 3D human pose estimation (HPE) from multi-view 2D keypoints. The typical approach, decoding 2D keypoints from predicted heatmaps, is unreliable as heatmaps can be multimodal under occlusion, and collapsing them into single peaks discards their spatial distribution. We seek to use the entire heatmap to estimate 3D poses more accurately, which requires solving two problems: how to robustly fuse heatmaps across views, and how to assess the reliability of heatmaps. For the former, we introduce a novel objective, Multi-viewExpected-OKS Maximization (MEOM), that locates a 3D joint where the views agree in probability mass. For the latter, we adopt highest-density-region (HDR) calibration as a diagnostic of that mass, independently of distance-based metrics. The proposed framework covers two settings, with and without 3D supervision. Without 3D supervision, we optimize 3D poses from pretrained heatmap predictors by maximizing MEOM, achieving comparable performance with state-of-the-art methods that rely on larger backbones, temporal fusion, and simulated 3D data. On ambiguous Human3.6M (H36MA) and occluded CMU Panoptic frames, the advantage is substantial. When 3D labels are available, we train the model end-to-end with a combined MEOM and MSE loss, achieving 19.11 mm absolute MPJPE on Human3.6M outperforming the state-of-the-art volumetric approach on absolute MPJPE at half the inference cost.

1. Introduction

The paper introduces MEOM to triangulate 3D joints from multi-view heatmap probability mass rather than collapsed peaks, and pairs it with HDR calibration to assess reliability. It supports optimization with frozen predictors or end-to-end training, outperforming point-based refinements and matching volumetric accuracy at lower inference cost.

  • Motivation: Collapsed heatmaps can select sharp, low-mass intersections under multimodality and occlusion, while volumetric methods retain mass at cubic memory cost.Algebraic triangulation is fast and geometrically explicit, but reduces each heatmap to one point.
  • MEOM: MEOM scores 3D candidates by aggregating expected-OKS responses across cameras, selecting locations where views agree in probability mass.The fixed OKS kernel acts as a sliding window that favors mass rather than the highest peak.
  • Calibration: HDR calibration diagnoses whether heatmap probability mass is reliable, complementing distance-based metrics such as OKS and MPJPE.HDR-ECE measures how often ground-truth keypoints fall within predicted highest-density regions.
  • Training settings: Without 3D labels, frozen heatmap predictors are refined by maximizing MEOM after view-weighted DLT initialization; with labels, training uses differentiable expected-OKS decoding and combined MEOM+MSE loss.The supervised setting does not require subsequent MEOM 3D refinement.
  • Results: On H36MA and occluded CMU Panoptic, MEOM gains are larger, while supervised training surpasses volumetric triangulation in absolute MPJPE at roughly half the inference cost.The framework is evaluated on Human3.6M, H36MA, and CMU Panoptic.

2. Related Work

The related work contrasts fast algebraic triangulation, which collapses heatmaps, with volumetric and probabilistic approaches that preserve uncertainty or represent ambiguity. MEOM retains algebraic triangulation while operating directly on heatmap distributions without a voxel grid or learned 3D prior.

  • 3D multiview HPE: Algebraic multi-view HPE uses view-weighted DLT and is fast, but collapses each heatmap to one point and at most a scalar confidence.Differentiable soft-argmax can support end-to-end training, while volumetric alternatives aggregate heatmap scores over 3D voxel grids.
  • 3D multiview HPE: Volumetric methods preserve heatmap mass by discretizing 3D space, whereas MEOM retains triangulation without constructing a voxel grid or learning a 3D human prior.This preserves explicit projective geometry while avoiding the volumetric representation described in the related work.
  • 2D human pose estimation: Modern 2D HPE represents keypoint uncertainty with dense heatmaps, while standard MSE training typically uses isotropic Gaussian targets and OKS evaluates scale- and joint-normalized similarity.ProbPose instead predicts calibrated non-parametric heatmaps with temperature scaling and trains with OKS loss.
  • Uncertainty and calibration: Probabilistic 3D HPE models ambiguity with structured Gaussians, normalizing flows, diffusion, or flow matching, but these methods do not directly evaluate predicted 3D pose distributions.In 2D HPE, standard confidences such as heatmap maxima are described as poorly calibrated.

3. Method

MEOM retains heatmap probability mass during multi-view triangulation by optimizing expected-OKS agreement, with separate formulations for inference without labels and end-to-end training with labels. HDR calibration diagnoses the reliability of that mass, while temperature scaling provides a practical recalibration approach.

  • Motivation: Heatmaps represent spatial uncertainty, but direct likelihood optimization can degrade because predicted distributions are sparse.
  • Expected-OKS fusion: Expected-OKS responses convolve each heatmap with a fixed kernel, replacing peak selection with a search for larger probability mass.The response is formed as S(v) = H(v) ∗ K.
  • Triangulation without labels: Without 3D labels, a frozen heatmap predictor supplies view-weighted DLT initialization, followed by iterative refinement that maximizes MEOM over the 3D joint.The heatmaps remain fixed while the 3D estimate is optimized.
  • End-to-end training: With 3D supervision, the model fixes the ground-truth 3D pose, trains heatmaps through differentiable expected-OKS decoding, and combines MEOM with 3D MSE.The joint objective retains MSE because pure MEOM supervision can under-constrain absolute 3D scale and geometry.
  • Calibration: HDR calibration evaluates whether ground-truth keypoints fall within predicted highest-density regions at their nominal coverage levels, and temperature scaling recalibrates heatmap mass.On discrete heatmaps, HDR regions are obtained by sorting pixels by descending density and taking the shortest prefix reaching the target mass.

4. Experiments

Experiments evaluate MEOM without and with 3D supervision across Human3.6M, ambiguous H36MA, and occluded CMU Panoptic, using absolute MPJPE as the primary metric. MEOM remains accurate with fewer views and frozen heatmaps, while end-to-end training reaches 19.11 mm absolute MPJPE at roughly half the volumetric inference cost.

  • Datasets and evaluation metrics: Experiments use absolute MPJPE in global coordinates as the primary metric, with pelvis-relative MPJPE and PA-MPJPE reported secondarily.Expected-OKS decoding and MEOM use predefined COCO per-joint kernel constants σk; retuning them for these datasets is outside scope.
  • MEOM triangulation without 3D labels: MEOM achieves the lowest absolute MPJPE on Human3.6M and H36MA among three refinement objectives, reaching 36.04 mm and 37.72 mm, respectively.On H36MA, MEOM improves absolute error by 4.0 mm over reprojection-error refinement, while heatmap-likelihood refinement reaches 92.25 mm.
  • MEOM triangulation without 3D labels: Expected OKS remains the most accurate objective across 4, 3, and 2 camera views on H36MA, although absolute error increases as views are removed.The comparison uses a fixed setting per objective across the tested view counts.
  • MEOM triangulation without 3D labels: On CMU Panoptic, frozen COCO ProbPose with no target-domain adaptation yields 23.99 mm absolute MPJPE, close to the 21.30 mm end-to-end algebraic baseline and below 39.50 mm RANSAC.The result remains competitive despite self-occlusion, multi-person occlusion, and mismatched frozen 2D heatmaps.
  • End-to-end triangulation with MEOM: End-to-end MEOM training reaches 19.11 mm absolute and 21.86 mm relative MPJPE on Human3.6M, with absolute error 0.06 mm below the official volumetric model.Both stages use 158.2 GFLOPs at test time, versus 301.4 GFLOPs for volumetric triangulation.
  • Temperature Scaling and HDR calibration: HDR-ECE is minimized at T=0.5, while MEOM peaks at T=1.0 but remains within 0.17 mm of its optimum at T=0.5.Over-sharp maps can cost up to 9.6 mm, and HDR-ECE tracks the reprojection-error minimum more sharply than NLL.

5. Conclusion

MEOM unifies 2D Expected-OKS decoding with multi-view algebraic fusion by locating joints where views agree in probability mass. HDR calibration complements this objective by diagnosing mass reliability alongside 3D error.

  • MEOM infers 3D joints where views agree in probability mass without collapsing heatmaps to peaks or constructing a voxel grid.
  • With frozen 2D networks, MEOM outperforms point-based and likelihood refinement, especially under ambiguity and occlusion.
  • With 3D labels, the dual MEOM score trains an algebraic triangulator matching volumetric-method accuracy at substantially lower inference cost.
  • HDR calibration should be reported alongside MPJPE because poorly calibrated heatmaps degrade mass-based triangulation.

A. Multi-view OKS as a factorized score

The paper interprets multi-view Expected-OKS fusion as a factorized score over calibrated views. Expected-OKS responses preserve heatmap mass while enabling continuous, differentiable refinement instead of dense voxelized search.

  • A candidate 3D joint is scored by aggregating per-view Expected-OKS responses under a factorized multi-view model.
  • The score is evaluated over 3D candidates and optimized through 2D OKS decoding with algebraic DLT initialization and refinement.
  • Each heatmap is convolved with an OKS kernel so Expected-OKS decoding selects probability mass rather than a single highest peak.
  • Temperature soft-argmax on the OKS response approaches hard arg max as τ decreases, providing a differentiable mode tracker.
  • Operating on the OKS response attenuates single-pixel spikes and mitigates, but does not eliminate, expectation collapse under multimodality.

B. Temperature Scaling as a Discrete HDR Recalibrator

Temperature scaling recalibrates discrete heatmap distributions for HDR coverage, with monotonic temperature effects enabling one-dimensional optimization. Its correction remains limited when competing modes create incompatible per-sample requirements.

  • Temperature scaling adjusts discrete heatmap sharpness, and increasing temperature monotonically increases HDR coverage across levels.
  • Hard OKS decoding can be approximated differentiably by temperature soft-argmax on the OKS response, unlike soft-argmax on the raw heatmap.
  • HDR-ECE has a V-shaped temperature objective, making an interior optimum reachable by one-dimensional sweep; the reported data optimum is T≈0.5.
  • HDR-ECE is reported alongside 3D error because it evaluates mass reliability relevant to heatmap-based refinement.
  • Exact HDR calibration generally cannot be achieved by one shared temperature when per-sample competing modes are inconsistent.

D. Additional experimental tables

Additional experiments examine triangulation weighting and implementation details. Expected-OKS remains strongest under OKS view weighting, while presence weighting has little effect on reprojection error.

  • The appendix reports triangulation tables, end-to-end compute protocols, and temperature-sweep results deferred from the main experiments.
  • The Human3.6M S9/S11 ablation evaluates presence and OKS view weights with DLT and refinement toggles over 2,181 frames.
  • Presence weighting barely changes reprojection error, while Expected OKS remains best under OKS weights in both splits.

D.2. CMU Panoptic triangulation ablation

The CMU Panoptic ablation identifies the Expected-OKS configuration with OKS weighting at both DLT and refinement as the source of the reported 23.99 mm absolute MPJPE.

  • 23.99 mm absolute MPJPE is reported for the Expected-OKS row with OKS weighting at both DLT and refinement.
  • The expanded comparison varies the full objective and view-weighting grid under MRCNN boxes.
  • The main-text CMU result corresponds specifically to the Expected-OKS configuration, rather than an unspecified ablation setting.

D.3. End-to-end MSE weight ablation

The end-to-end Stage 2 ablation varies the MSE weight while fixing the MEOM weight at 1.0, with relative and absolute MPJPE favoring different settings.

  • Relative MPJPE is best at λMSE=100.
  • Absolute MPJPE is marginally best at λMSE=500.
  • The ablation tests λMSE ∈{10, 50, 100, 200, 500} under MEOM weight 1.0.

D.4. Computational cost of 2D-to-3D lifting

Expected-OKS lifting adds modest computation compared with algebraic soft-argmax and DLT while remaining far cheaper than volumetric unprojection and V2V processing. The reported sweep also shows that expected-OKS decoding retains a substantial accuracy advantage over heatmap likelihood under stronger softening.

  • Lift computation: 0.162 GFLOPs is added by Expected-OKS decoding per joint, versus 1.3 MFLOPs for algebraic soft-argmax and DLT.
  • Lift computation: 919× cheaper than volumetric unprojection and V2V, the Expected-OKS lift keeps the full MSE+MEOM pipeline at 158 GFLOPs versus 301 GFLOPs volumetric.
  • Runtime and memory: 1.9 ms and 12 MB peak allocation characterize the Expected-OKS lift, compared with 23 ms and 547 MB for the volumetric lift.
  • Measurement setup: The cost comparison uses four 384 × 384 views and excludes the shared ResNet-152 backbone from the lift measurements.
  • Temperature sweep: Expected-OKS achieves 37.5 mm best absolute MPJPE at T=1.0, while likelihood plateaus near 56.4 mm at T=20.
  • Temperature sweep: Likelihood refinement improves with softening but saturates well above expected OKS, whose best expected-OKS absolute MPJPE is marked at 37.5 mm.
Loading 2608.30521v1…