Source-linked AI summary
Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, Mohammad Norouzi
TL;DR
Geometric vision commonly separates keypoint detection from downstream reasoning, while obtaining category-specific 3D keypoints can require strong supervision. KeypointNet learns latent 3D keypoints and detectors end-to-end for relative pose estimation, using differentiable pose and multi-view consistency objectives; it discovers consistent keypoints and outperforms a supervised baseline without ground-truth keypoint annotations.
Problem
The paper addresses the need for category-specific 3D keypoints without relying on manual annotations or separate post-process geometric reasoning.
Method
KeypointNet jointly learns latent 3D keypoints and detectors, optimizing differentiable relative-pose and multi-view-consistency objectives.
Results
KeypointNet discovers geometrically and semantically consistent keypoints and outperforms supervised baselines on relative 3D pose estimation without keypoint annotations.
Takeaways & Limitations
Task-driven optimization produces meaningful, spatially varied parts such as car wheels, plane wings, and chair legs for geometric estimation.
Takeaways & Limitations
Orientation errors can flip predicted keypoints for visually ambiguous cars or unusual wing shapes, and real-world transfer remains preliminary.
Abstract
from arXiv · showhide
This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet are visualized at http://keypointnet.github.io/.
1 Introduction
KeypointNet replaces separately trained keypoint detection and post-process geometric reasoning with end-to-end task-driven optimization. For relative 3D pose estimation, it learns category-specific 3D keypoints without manual keypoint annotations and outperforms a supervised baseline.
- Motivation: Existing geometric-vision pipelines typically detect keypoints first and apply geometric reasoning afterward.The paper investigates whether jointly optimizing both stages can improve geometric tasks.
- Motivation: Manual keypoint annotations or offline model fitting provide costly supervision for category-specific 3D pose estimation.Offline fitting is described as expensive and error prone.
- Approach: KeypointNet jointly learns category-specific 3D keypoints and their detectors for a differentiable downstream objective.The framework includes a differentiable pose objective and a multi-view consistency loss.
- Approach: During training, two views and their known rigid transformation supervise keypoints that remain consistent and support transformation recovery.At inference, the model extracts 3D keypoints from a single image.
- Results: KeypointNet discovers geometrically and semantically consistent keypoints across viewing angles and object instances, then significantly outperforms a strongly supervised pose-estimation baseline.The evaluation uses ShapeNet car, chair, and plane categories and does not use ground-truth keypoint annotations.
2 Related Work
Prior work studies supervised keypoint detection, latent representations, and correspondence learning with varying structural assumptions. KeypointNet instead learns task-directed latent 3D keypoints whose physical structure supports correspondence across large pose changes.
- Keypoint detection: Monocular 3D human-keypoint methods generally infer predefined joints using supervised objectives or 3D structural priors.Other approaches lift inferred 2D keypoints into 3D or match image data to models.
- Latent keypoints: KeypointNet differs from supervised landmark methods because its keypoint set is latent rather than defined a priori.The latent points are optimized end-to-end for geometric estimation.
- Latent representations and correspondence: Latent feature approaches include capsule-style representations and methods that learn correspondences with appearance, geometry, or CAD-model supervision.These lines of work vary in how much supervision and structural prior they use.
- KeypointNet: KeypointNet combines relative pose estimation with multi-view consistency to select latent keypoints for a downstream task.Its output can be viewed as a latent 3D feature representation shaped by these objectives.
- KeypointNet: Representing correspondences in physical 3D enables the method to find occluded matches across images with large out-of-plane rotations.This contrasts with approaches using only view-consistency supervision on depth maps or other correspondence formulations.
3 End-to-end Optimization of 3D Keypoints
KeypointNet jointly learns category-specific 3D keypoints and detectors by optimizing them for relative pose estimation. Multi-view consistency and differentiable pose losses encourage keypoints that remain consistent across views while supporting geometric recovery.
- End-to-end objective: KeypointNet predicts an ordered list of 3D keypoints from each image and jointly optimizes them for a downstream task.Each keypoint combines pixel coordinates with an associated depth value.
- Multi-view consistency: Multi-view consistency penalizes discrepancies between corresponding keypoints after applying the known rigid transformation between two views.The loss measures error in observable pixel coordinates, while predicted depth enables projection between views.
- Relative pose estimation: Consistency alone can produce collapsed or otherwise non-unique keypoint configurations, so pose estimation supplies task-specific optimality and encourages keypoint separation.The pose task is used because it naturally favors well-posed geometric estimation problems.
- Relative pose estimation: The pose loss measures angular error between the ground-truth rotation R and the rotation estimated from the two keypoint sets using orthogonal Procrustes.The rotation estimate is obtained from unprojected, mean-subtracted 3D keypoint coordinates and differentiable SVD operations.
- Discovered keypoints: The learned keypoints are geometrically and semantically consistent across views and object instances, including meaningful parts such as car wheels.The pose objective is reported to discover useful selections such as car wheels, plane wings and cockpit, and chair legs and back.
4 KeypointNet Architecture
KeypointNet uses spatial probability maps and depth predictions within an equivariant fully convolutional architecture. A coarse pose-orientation flag helps resolve ambiguities caused by object symmetries.
- Translation equivariance: Translation equivariance requires image shifts to produce the same shifts in predicted keypoint locations.Without this property, a standard CNN would need examples covering objects at every possible image location.
- Keypoint prediction: The network predicts a probability distribution map gi(u, v) for each keypoint and uses spatial softmax expectations to recover pixel coordinates.Depth is predicted separately at every pixel and combined with the spatial distribution.
- Network architecture: Strided-one fully convolutional layers preserve the probability-map resolution and equivariance, while dilated convolutions expand the receptive field.The architecture stacks multiple dilated-convolution layers for broader contextual processing.
- Translation equivariance: The equivariant design reduces training examples needed for generalization and removes coordinate-conversion burden from the network.This lets the network focus on other tasks, including depth inference.
- Network architecture: The architecture stacks 13 dilated-convolution layers and outputs 2N channels divided between keypoint probability maps and depth maps.All layers use 3×3 kernels; the final outputs are converted into 3D coordinates.
- Breaking symmetry: A binary left-right orientation flag conditions keypoint prediction to break ambiguities from visually symmetric object parts.The flag is inferred from projected front and back points and then fed into the keypoint prediction network.
5 Additional Keypoint Characteristics
KeypointNet adds regularizers that encourage distinct, object-contained 3D keypoints while preventing degenerate solutions. These objectives use silhouette information during training and support concentrated spatial predictions.
- Separation loss: Separation loss penalizes keypoints closer than a 3D threshold δ.It is computed in 3D, allowing keypoints to share pixels when their depths differ.
- Separation loss: The separation objective primarily prevents two latent keypoints from collapsing to one local minimum.
- Silhouette consistency: Silhouette consistency constrains keypoint probability mass to the object silhouette and encourages concentrated, low-variance spatial distributions.
- Silhouette consistency: The binary segmentation mask is used to compute the silhouette loss during training, but not at inference time.The loss is zero when all probability mass lies within the silhouette.
- Silhouette consistency: A variance-minimization term makes distributions peaky, helping keep their means inside non-convex silhouettes.
6 Experiments
Experiments on ShapeNet cars, chairs, and planes compare KeypointNet with supervised keypoint baselines and assess consistency across views, instances, and occlusions. The method achieves lower reported rotation errors and produces semantically consistent predictions, while orientation mistakes create flipped keypoints.
- Training data: ShapeNet training data uses approximately 51K models across 270 categories, with 200 rendered views per model forming 100 training pairs.
- Comparison with a supervised approach: The supervised comparison uses human landmark labels for cars, chairs, and planes and evaluates angular distance error on held-out test models.The supervised network uses the same architecture and an L2 loss to human labels.
- Comparison with a supervised approach: KeypointNet produces lower mean and median rotation errors than both supervised techniques.
- Generalization across views and instances: Predicted keypoints remain consistent across views, including completely occluded parts such as a chair’s back right leg.
- Generalization across views and instances: Across ShapeNet cars, planes, and chairs, the network generalizes to unseen appearances and shape variations while predicting occluded wheels and chair legs.
- Generalization across views and instances: Using physical 3D transformations, the model predicts semantically consistent keypoints across instances and from any viewing angle.The experiments use 10 keypoints by default and include examples such as plane cockpit tracking.
- Failure cases: When the orientation network predicts incorrectly, keypoints can be flipped, especially for visually ambiguous car fronts and backs or unusual wing shapes.
7 Discussion & Future work
KeypointNet learns consistent latent keypoints without annotations and outperforms direct supervised keypoints for rigid 3D pose estimation, while real-image transfer remains preliminary.
- 7 Discussion & Future work: KeypointNet discovers consistent keypoints across multiple views and object instances without ground-truth keypoint annotations.Its objectives combine relative pose estimation with multi-view consistency.
- 7 Discussion & Future work: The discovered keypoints outperform a direct supervised learning baseline on rigid 3D pose estimation.
- 7 Discussion & Future work: Transfer to real-world images is preliminary, using ShapeNet training images with random backgrounds.The paper suggests domain adaptation or direct training on real images with relative pose labels as future directions.
- 7 Discussion & Future work: The framework can extend to arbitrary keypoint counts using confidence prediction and unordered-set losses.Visual descriptors could also be added through post-processing or joint end-to-end optimization.
B Ablation study
The ablation study shows that pose estimation, silhouette, and multi-view consistency losses shape where keypoints are placed and whether they remain stable across views.
- B Ablation study: Removing multi-view consistency causes keypoints to move with viewing angle instead of tracking particular object parts.
- B Ablation study: The pose-estimation loss alone can strongly optimize only some keypoints and does not explicitly force every point to be consistent.
- B Ablation study: Removing the pose estimation loss places keypoints near object centers, where rotation causes the least pixel displacement across views.
- B Ablation study: Increasing pose-estimation noise encourages keypoints to spread apart from the object center.The comparison uses twice the noise, 0.2, versus the baseline setting.
- B Ablation study: Removing silhouette consistency causes keypoints to lie outside the object while still satisfying multi-view consistency.They occupy a virtual 3D space that rotates with the object.
C Results on deformed object
On a non-rigidly deformed car, KeypointNet predicts local semantic parts and captures the deformation of the overall car structure.
- C Results on deformed object: The network predicts wheel locations and the overall deformation of a non-rigidly deformed car.This tests robustness to shape variation and whether detections rely on local features rather than a regular rigid structure.
D Results using different numbers of keypoints
With more predicted keypoints, the network progresses from prominent components to a broader set of object parts.
- D Results using different numbers of keypoints: As the number of keypoints increases, detections begin with prominent components and gradually track more parts.For planes, the first components include the head and wings.
E Proof-of-concept results on real-world images
Adding random SUN397 backgrounds to rendered training examples lets KeypointNet predict keypoints on some real ImageNet cars, while large perspective distortions and strong visual patterns remain challenging.
- Random SUN397 backgrounds added to rendered training examples enable keypoint prediction on some ImageNet cars.The paper presents both hand-picked successes and failure cases in Figure 10.
- Large perspective distortion and strong patterns or specular highlights cause particular difficulties for real-image keypoint prediction.