PaperlayerView exact revision on arXiv

Source-linked AI summary

Epipolar Transformers

Yihui He, Rui Yan, Katerina Fragkiadaki, Shoou-I Yu

arXiv:2005.04551v1cs.CV

TL;DR

Multi-view 2D detectors traditionally operate independently before triangulation, limiting their use of 3D information in difficult cases. The paper introduces a differentiable epipolar transformer that fuses cross-view features along epipolar lines, improving results on InterHand and Human3.6M, including 26.9 mm MPJPE and a 4.23 mm gain over state-of-the-art on Human3.6M without external data.

  • Problem

    Independent 2D detectors handle occlusions and oblique viewing angles without leveraging 3D information, despite multi-view geometry being available.

  • Method

    The differentiable epipolar transformer uses epipolar constraints and feature matching to augment intermediate 2D-detector features with 3D-aware features from neighboring views.

  • Results

    The method consistently improves over baselines on InterHand and Human3.6M; on Human3.6M it achieves 26.9 mm MPJPE and outperforms state-of-the-art by 4.23 mm without external data.

  • Takeaways & Limitations

    The epipolar transformer can be integrated into existing 2D pose networks and is especially beneficial for multi-view pose estimation with few cameras.

  • Takeaways & Limitations

    The method relies on precise camera calibration and neighboring views whose viewing-angle differences are not too large.

Abstract

from arXiv · show

A common approach to localize 3D human joints in a synchronized and calibrated multi-view setup consists of two-steps: (1) apply a 2D detector separately on each view to localize joints in 2D, and (2) perform robust triangulation on 2D detections from each view to acquire the 3D joint locations. However, in step 1, the 2D detector is limited to solving challenging cases which could potentially be better resolved in 3D, such as occlusions and oblique viewing angles, purely in 2D without leveraging any 3D information. Therefore, we propose the differentiable "epipolar transformer", which enables the 2D detector to leverage 3D-aware features to improve 2D pose estimation. The intuition is: given a 2D location p in the current view, we would like to first find its corresponding point p' in a neighboring view, and then combine the features at p' with the features at p, thus leading to a 3D-aware feature at p. Inspired by stereo matching, the epipolar transformer leverages epipolar constraints and feature matching to approximate the features at p'. Experiments on InterHand and Human3.6M show that our approach has consistent improvements over the baselines. Specifically, in the condition where no external data is used, our Human3.6M model trained with ResNet-50 backbone and image size 256 x 256 outperforms state-of-the-art by 4.23 mm and achieves MPJPE 26.9 mm.

1. Introduction

The paper addresses multi-view 3D pose estimation by giving 2D detectors access to cross-view 3D-aware features before triangulation. The proposed epipolar transformer improves pose estimation on Human3.6M and InterHand, including a 26.9 mm MPJPE result on Human3.6M.

  • Motivation: Multi-view pose estimation resolves depth ambiguity using synchronized, calibrated views, but commonly detects 2D joints independently before robust triangulation.The two-step pipeline first detects 2D joints per view and then triangulates them into 3D.
  • Motivation: Independent 2D detection handles occlusions and oblique viewing angles without 3D information, potentially producing inaccurate or 3D-inconsistent detections.The detector may instead require more capacity and training data for these difficult cases.
  • Method: The differentiable epipolar transformer augments intermediate reference-view features with features from neighboring views to make them 3D-aware.It searches along the source-view epipolar line, computes feature similarities, and forms a weighted approximation of the corresponding feature.
  • Method: The network takes synchronized, geometrically calibrated multi-view images, outputs 2D joints, and can subsequently support 3D triangulation.Although its output is 2D, the network uses both 2D and 3D features for more accurate predictions.
  • Results: 26.9 mm MPJPE is achieved on Human3.6M with ResNet-50, 256×256 inputs, and no external data, outperforming Qiu et al. by 4.23 mm.Experiments were conducted on Human3.6M and InterHand, where the approach consistently outperformed baselines.
  • Contributions: The method is designed for easy integration, uses few learnable parameters, supports feature-matching analysis, and can generalize to new calibrated multi-camera setups.Its output feature dimensions match the input dimensions, and its parameter size is C-by-C.

2. Related Work

Prior work addresses multi-view pose estimation through temporal modeling, weak supervision, feature fusion, differentiable triangulation, epipolar constraints, and attention mechanisms. The paper distinguishes its approach by using epipolar geometry specifically for deep feature fusion.

  • Multi-view 3D Human Pose Estimation: Multi-view human pose methods include temporal convolution over 2D keypoints, multi-view constraints as weak supervision, and differentiable triangulation.The paper identifies Qiu et al. and Iskakov et al. as its closest comparisons.
  • Multi-view 3D Human Pose Estimation: Qiu et al. fuse views with fixed pixel-pair attention, avoiding calibration but requiring more data and weights while failing to adapt when camera setups change.The learned attention becomes inapplicable when the multi-camera configuration changes at test time.
  • Multi-view 3D Human Pose Estimation: Iskakov et al. fuse features with 3D voxel maps, whereas this paper fuses 3D-aware features in 2D maps to reduce computational and memory demands.The paper presents this as a key difference between learnable triangulation and its method.
  • Multi-view Hand Pose Estimation: Multi-view hand-pose research is less common than monocular hand-pose work because obtaining annotated multi-view hand data is difficult.Prior efforts include multi-view bootstrapping for 2D hand keypoints and an RGB-D two-hand action dataset.
  • Epipolar Geometry in Deep Neural Networks: Other neural methods apply epipolar constraints to depth regression, interest points, or semi-supervised keypoint detection, while this paper applies epipolar geometry to deep feature fusion.This distinction defines the paper’s positioning within epipolar-geometry methods.
  • Attention Mechanism: The method is called epipolar attention because it computes feature-similarity attention weights along epipolar lines for feature fusion.This connects the paper’s terminology to transformer and non-local attention mechanisms.

3. The Epipolar Transformer

The epipolar transformer samples source-view features along geometrically determined epipolar lines, matches them to reference-view features, and fuses the result into 3D-aware features. Its fusion designs preserve the reference feature shape for integration into existing networks.

  • Overview: The epipolar transformer has an epipolar sampler and a feature fusion module operating densely across intermediate feature-map locations.The sampler gathers source-view features, while fusion combines them with the corresponding reference-view feature.
  • The Epipolar Sampler: For a reference-view query pixel, camera projection matrices and centers determine the corresponding source-view epipolar line.The corresponding source point must satisfy the epipolar constraint l^T p′ = 0.
  • The Epipolar Sampler: The sampler uniformly selects K source-view locations along the visible epipolar line and uses bilinear interpolation for sub-pixel coordinates.In the experiments, K equals 64; query points whose lines miss the source image are skipped.
  • Feature Fusion Module: Because the true corresponding source point is unknown, the method approximates its feature by a similarity-weighted sum of features sampled along the epipolar line.Similarity uses a dot product followed by softmax.
  • Feature Fusion Module: A residual-style fusion transforms the source feature with Wz and adds it to the reference feature, producing a fused feature with the same shape as the input.The implementation uses 1 × 1 convolutions for Wz, enabling insertion into existing network stages.
  • Feature Fusion Module: The Bottleneck Embedded Gaussian alternative downsamples channels by two before upsampling, preserving the input shape after fusion.Figure 2 compares this architecture with the Identity Gaussian design and depicts tensor shapes and fusion operations.

4. Experiments

Experiments on InterHand and Human3.6M evaluate the epipolar transformer across datasets, architectural choices, viewing angles, view counts, and comparisons with prior methods. The method improves pose estimation, while relying on accurate calibration and suitably separated neighboring views.

  • Experimental setup: Experiments use InterHand and Human3.6M, with MPJPE computed as the average L2 distance between predicted and ground-truth joints.InterHand uses synchronized multi-view studio imagery; Human3.6M has four cameras with larger viewing-angle differences.
  • Experimental setup: The epipolar transformer is trained in a single-stage Hourglass network and paired with RANSAC triangulation to obtain 3D joint positions.During prediction, the source view is randomly selected from cameras used during training.
  • Ablation studies: Identity Gaussian performs slightly better than Bottleneck Embedded Gaussian, while softmax weighting performs slightly better than max along the epipolar line.The authors hypothesize that preserving resolution benefits accurate correspondences and that softmax provides gradients for all sampled locations.
  • Ablation studies: 24° is the most effective viewing-angle difference tested for InterHand, balancing feature similarity against matching difficulty and occlusion risk.The tested settings were 6°, 12°, 24° and 42°, with 24° used by default afterward.
  • Human3.6M results: 97.01% JDR is achieved on Human3.6M with the epipolar transformer, compared with 95.9% JDR for cross-view fusion using the same ResNet-50 and 256×256 input.Data augmentation further raises JDR to 98.25%.
  • Human3.6M results: 26.9 mm MPJPE is achieved without external data using RPSM, approximately 14 mm better than the equivalent cross-view result; triangulation achieves 33.1 mm, about 12 mm better.The cited comparison uses the same ResNet-50 backbone and 256×256 input size.

5. Conclusion

The epipolar transformer fuses features along neighboring views’ epipolar lines, improving multi-view pose estimation, including settings with few cameras and occlusions.

  • The epipolar transformer enables 2D pose detectors to use 3D-aware features by fusing neighboring-view features along epipolar lines.
  • Experiments show improvements over baselines on Human3.6M and InterHand, especially when few cameras are available.
  • Feature matching along epipolar lines provides more accurate matches in difficult occlusion scenarios.
  • The module is easy to add to existing 2D pose networks because it has few learnable parameters and preserves feature dimensions.

Supplementary Materials: Epipolar Transformer for Multi-view Pose Estimation

The supplementary material studies where the epipolar transformer can be inserted in backbone pose networks.

  • The epipolar transformer can be added at either an early or late stage of Hourglass and ResNet-50 backbone models.

A. Dealing with Image Transformations

Image transformations remain compatible with the epipolar transformer when corresponding camera projection parameters are updated consistently.

  • Spatial image transformations must be reflected in the camera calibration parameters used by the epipolar transformer.
  • Rotation, scaling, and cropping can be applied through affine transformations if the projection matrix is updated accordingly.
  • Reference and source views may use different scaling and cropping parameters.
  • Projection matrices require special scaling when images are resized or pooled.
  • Feature coordinates are aligned to pixel centers rather than top-left corners for precise epipolar-transformer sampling.

B. 2D Prediction Visualization in Video

The supplementary videos visualize predicted 2D skeletons and hands from baseline and epipolar-transformer models on Human3.6M and InterHand test sets.

  • The Human3.6M video visualizes 2D predicted skeletons from three ResNet-50, 256×256 configurations corresponding to Table 6 entries.
  • The InterHand video compares ground truth with a 5.46 mm baseline and a 4.91 mm epipolar-transformer model.

C. Stages to Add Epipolar Transformer

The experiments compare inserting the epipolar transformer at early versus late stages. Figure 8 shows the precise insertion locations for each setting and network.

  • C. Stages to Add Epipolar Transformer: The study compares early- and late-stage insertion of the epipolar transformer.The comparison is reported in Table 2 under the section on which stage to insert the transformer.
  • C. Stages to Add Epipolar Transformer: Figure 8 illustrates the insertion locations for the early and late settings.The figure covers an one-stage Hourglass network on InterHand and a ResNet-50 simple baseline on Human3.6M.
  • C. Stages to Add Epipolar Transformer: The early setting uses an one-stage Hourglass network on InterHand, while the late setting uses a ResNet-50 simple baseline on Human3.6M.

D. Epipolar Transformer Visualization on Human3.6M [13]

The Human3.6M visualizations examine feature-matching similarity along epipolar lines in easy and more difficult cases. In the easy case, both the proposed and baseline deep features identify an accurate match.

  • D. Epipolar Transformer Visualization on Human3.6M [13]: The visualizations analyze feature-matching similarity for Human3.6M images across Figures 9, 10, and 11.
  • D. Epipolar Transformer Visualization on Human3.6M [13]: In an easy Human3.6M case, the epipolar-transformer prediction aligns well with the ground truth.
  • D. Epipolar Transformer Visualization on Human3.6M [13]: In that easy case, the baseline deep features without the epipolar transformer also match the point accurately.

E. Epipolar Transformer Visualization on InterHand

InterHand visualizations show that epipolar-transformer deep features can distinguish the correct finger when similarly colored fingers create multiple color-feature matches. The figures compare feature matches along epipolar lines using RGB, baseline deep features, and epipolar-transformer features.

  • E. Epipolar Transformer Visualization on InterHand: InterHand color features produce multiple similarity peaks because different fingers can have similar colors.
  • E. Epipolar Transformer Visualization on InterHand: For most InterHand cases, epipolar-transformer deep features discriminate the correct finger from similarly looking fingers.
  • E. Epipolar Transformer Visualization on InterHand: Figures 9–11 compare epipolar-transformer deep features, baseline deep features without the transformer, and RGB features along Human3.6M epipolar lines.
  • E. Epipolar Transformer Visualization on InterHand: Figure 12 compares color features with epipolar-transformer deep features using best matches and similarity distributions along epipolar lines.
Loading 2005.04551v1…