Source-linked AI summary
Viewpoints and Keypoints
Shubham Tulsiani, Jitendra Malik
TL;DR
The paper addresses rigid-object pose estimation through global viewpoint and local keypoints, including settings with known and unknown bounding boxes. It introduces CNN-based viewpoint prediction and viewpoint-conditioned, multiscale keypoint estimation, reporting significant improvements over the state of the art across both tasks.
Problem
Rigid-object pose estimation requires predicting both coarse viewpoint and finer keypoint locations, including the challenging setting without annotated bounding boxes.
Method
The method predicts viewpoint with CNNs and combines multiscale appearance likelihoods with viewpoint-conditioned likelihoods for keypoint prediction.
Results
The approach significantly improves state-of-the-art performance for viewpoint and keypoint prediction with and without annotated bounding boxes.
Takeaways & Limitations
Viewpoint estimates can substantially improve local appearance-based keypoint predictions, while error-mode and object-characteristic analyses identify directions for future work.
Takeaways & Limitations
The analysis excludes diningtable and bottle because of few unoccluded instances or insignificant variation.
Abstract
from arXiv · showhide
We characterize the problem of pose estimation for rigid objects in terms of determining viewpoint to explain coarse pose and keypoint prediction to capture the finer details. We address both these tasks in two different settings - the constrained setting with known bounding boxes and the more challenging detection setting where the aim is to simultaneously detect and correctly estimate pose of objects. We present Convolutional Neural Network based architectures for these and demonstrate that leveraging viewpoint estimates can substantially improve local appearance based keypoint predictions. In addition to achieving significant improvements over state-of-the-art in the above tasks, we analyze the error modes and effect of object characteristics on performance to guide future efforts towards this goal.
1. Introduction
The paper represents rigid-object pose through global viewpoint and local keypoint locations, predicting both with CNN-based models. It uses viewpoint estimates to condition multiscale keypoint predictions and evaluates both boxed and detection settings.
- Pose is characterized by viewpoint for global structure and keypoint locations for local detail.
- The method first predicts viewpoint, then uses it to improve local appearance-based keypoint predictions.The approach is motivated by global precedence and viewpoint-dependent appearance.
- Multiscale appearance models combine fine localization with coarser contextual robustness for keypoint prediction.Fine-scale responses can localize accurately but produce false positives, while coarser responses are more robust to mis-localization.
- Viewpoint-conditioned keypoint likelihoods are combined with appearance likelihoods to produce final keypoint predictions.Figure 2 summarizes the combination of multiscale convolutional response maps and viewpoint-conditioned likelihoods.
- The work evaluates pose estimation with known boxes and in detection settings where objects must be detected and posed simultaneously.It emphasizes that conventional keypoint evaluation assumes ground-truth boxes, unlike deployment with imprecise detections.
- The authors analyze algorithmic failure modes and the effect of object characteristics on performance.
2. Related Work
Prior work covers CNN and part-based viewpoint estimation, keypoint localization, and human keypoint detection, but generic-object keypoint detection remains comparatively unaddressed.
- CNNs implicitly capture hierarchical relations, contrasting with DPMs that explicitly model part appearances and deformations.
- Viewpoint methods range from instance-based and 3D part-based models to fixed global descriptors and viewpoint-specific DPM components.
- Keypoint prediction distinguishes localization with known bounding boxes from detection when the bounding box is unknown.
- Human keypoint research includes convolutional approaches evaluated both with known boxes and without ground-truth box annotations.
- For generic objects, keypoint annotations exist, but keypoint detection without assumed boxes has been largely unaddressed.
- Closely related methods jointly estimate keypoints and pose but use explicit local-evidence aggregation, category restrictions, or instance-model matching.
3. Viewpoint Estimation
The viewpoint estimator represents rigid-object pose relative to a canonical pose using three Euler angles and predicts discretized angular classes with a shared CNN architecture.
- Viewpoint is defined by azimuth, elevation, and cyclorotation relative to a canonical pose.These Euler angles are equivalent to a rotation-matrix representation.
- Each angle is formulated as classification over Nθ disjoint angular bins.
- The network produces Nc∗Na∗Nθ outputs across object classes, angles, and angular bins.
- A CNN initialized from ImageNet pretraining is fine-tuned with an added fully connected output layer.The evaluated architectures are based on the Krizhevsky and Simonyan networks.
- A selective loss trains one CNN jointly across classes while using only outputs corresponding to each instance’s class.
4. Viewpoint Informed Keypoint Prediction
Keypoint prediction combines multiscale convolutional appearance responses with a non-parametric likelihood conditioned on predicted viewpoint. This integrates local evidence with global pose context to estimate keypoint locations.
- 4.1. Multiscale Convolutional Response Maps: The method uses a fully convolutional network whose output channels correspond to class-specific keypoints.
- 4.1. Multiscale Convolutional Response Maps: Target response maps train each keypoint channel to activate near its annotated spatial location.The output receptive-field centers are aligned to positions in the input image.
- 4.1. Multiscale Convolutional Response Maps: A second network at 192 × 192 input and 6×6×Nkp output supplies coarser context, with upsampled outputs combined with the larger network.
- 4.2. Viewpoint Conditioned Keypoint Likelihood: The viewpoint-conditioned prior uses training instances near the predicted rotation to model likely keypoint positions.Nearby instances are selected using geodesic distance between rotation matrices.
- 4.2. Viewpoint Conditioned Keypoint Likelihood: The global pose-conditioned likelihood is a Gaussian mixture that is combined with local appearance likelihood to obtain keypoint locations.
- 4.2. Viewpoint Conditioned Keypoint Likelihood: Coordinates are normalized by warping each instance bounding box to a fixed 12 × 12 size, with σ = 2.
5. Experiments : Viewpoint Prediction
The experiments evaluate viewpoint estimation with known object bounds and in the detection setting, using rotation-based metrics and comparisons against prior methods. Results show strong viewpoint predictions and significant improvement over state-of-the-art methods across categories.
- Viewpoint Estimation with Ground Truth box: The known-bounds experiment isolates viewpoint estimation by evaluating unoccluded, nontruncated objects independently of localization errors.The authors defer analysis of occlusion and truncation effects in this setting to a later section.
- Evaluation protocol: Viewpoint error is measured using geodesic distance between ground-truth and predicted rotation matrices.The evaluation uses median error and accuracy within a fixed angular threshold as complementary metrics.
- Viewpoint Estimation with Ground Truth box: Figure 3 sorts unoccluded ground-truth instances by viewpoint error and visualizes predictions by rendering a 3D model from each predicted viewpoint.Columns represent the 15th through 90th error percentiles.
- Viewpoint Estimation with Ground Truth box: End-to-end CNN training improves results, while the proposed method with TNet significantly outperforms the Pool5-TNet baseline.The authors also report a significant improvement using the ONet architecture and use it for subsequent experiments.
- Detection and Viewpoint Estimation: ARPθ is stricter than azimuth-only AVP because it requires correct prediction of all Euler angles.The detection evaluation considers both localization and viewpoint correctness, while the cited AVP metric evaluates only azimuth among viewpoint angles.
- Detection and Viewpoint Estimation: The CNN-based approach significantly outperforms state-of-the-art methods across all categories in the detection and viewpoint-estimation setting.This setting augments RCNN detections from MCG proposals with pose predictions from each detection bounding box.
6. Experiments : Keypoint Prediction
The experiments evaluate keypoint localization with known boxes and keypoint detection with object proposals, using PCK and APK metrics. Multiscale CNN responses and viewpoint-conditioned likelihood improve performance, with the approach also extending to articulated pose estimation.
- Evaluation settings and metrics: Keypoint localization evaluates predicted keypoints against annotated instances using PCK, while keypoint detection evaluates scored hypotheses using APK.Both metrics use a distance threshold based on α and the object’s maximum dimension.
- Evaluation settings and metrics: The evaluation uses keypoint annotations from, with PASCAL VOC training images for training and validation images for evaluation.
- Keypoint localization: The multiscale convolutional baseline outperforms, and adding viewpoint-conditioned likelihood produces a significant further performance gain.The compared systems are conv6, conv12, conv6+conv12, and the full conv6+conv12+pLikelihood system.
- Keypoint detection: In the detection setting, object proposals provide candidate locations, and keypoint hypotheses are scored by combining keypoint log-likelihood with the object detection score.The reported detection results show consistent improvement from the pose-conditioned likelihood over local appearance predictions.
- Keypoint detection: The results support viewpoint-conditioned priors, end-to-end CNNs, and multiscale response maps as effective components for keypoint prediction.
- Generalization to articulated pose: APK = 0.22 for PASCAL VOC 2012 person keypoints, compared with APK = 0.15 for state-of-the-art method.This demonstrates applicability of the multiscale convolutional response approach to articulated pose estimation.
7. Analysis
The analysis examines viewpoint and keypoint prediction under known bounding boxes, focusing on object characteristics, error modes, and component-level evaluation. Performance is weaker for occluded and small objects, while inaccurate localization is the main keypoint error source and most viewpoint errors fall into analyzed modes.
- Analysis: The analysis evaluates viewpoint and keypoint prediction with known bounding boxes, separating pose-estimation behavior from detection-system effects.The study adapts diagnostics for pose estimation and reports full-method keypoint results using PCK.
- Object Characteristics: Viewpoint prediction performs worse for occluded objects and shows a significant gap between small and large objects.The authors note that additional context may be needed for strong performance without higher-resolution input.
- Error Modes: Only about 3% of viewpoint-prediction errors are not explained by the analyzed azimuth error modes, which occur with similar frequency.The analysis restricts generic-rotation error characterization to predicted azimuth.
- Object Characteristics: The analysis excludes diningtable and bottle from its object-characteristics evaluation because of few unoccluded instances and insignificant variation, respectively.
- Object Characteristics: Keypoint prediction shows the same object-characteristic pattern: occluded objects are handled poorly, with a significant performance gap between small and large objects.
- Error Modes: Laterally symmetric keypoints are not a significant error mode, while inaccurate localizations are the main source of keypoint-prediction error.The conclusion follows from similar performance in the left/right setting and the difference between base performance and PCK[α = 0.2].
8. Conclusion
The paper presents CNN-based viewpoint prediction and keypoint prediction that combines multiscale appearance with viewpoint-conditioned likelihood. It reports improvements over state-of-the-art in settings with and without annotated bounding boxes and includes evaluations, ablations, and diagnostics.
- The algorithm predicts viewpoint with CNN architectures and combines multiscale appearance with a viewpoint-conditioned likelihood to predict keypoints.
- The approach improves state-of-the-art performance for viewpoint and keypoint prediction with and without annotated bounding boxes.
- The paper evaluates keypoint detection, analyzes performance through ablation studies, and examines error modes and object characteristics.