PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild

Yang Xiao, Vincent Lepetit, Renaud Marlet

arXiv:2007.12107v2cs.CV

TL;DR

The paper tackles the limited generalization of object detection and viewpoint estimation to novel categories with few labeled samples. It conditions predictions on class-representative image or 3D features, and also develops a category-agnostic viewpoint method when 3D models are unavailable. Across few-shot detection, viewpoint estimation, and their joint evaluation, the approach achieves state-of-the-art or promising results on several benchmarks.

  • Problem

    Few-shot object detection and viewpoint estimation must generalize to novel object categories despite limited annotated samples and costly 3D supervision.

  • Method

    The approach conditions query predictions on class-informative image features for detection and optional exemplar 3D-model embeddings for viewpoint estimation, while also using a category-agnostic image-embedding method.

  • Results

    The framework achieves state-of-the-art performance across several few-shot detection and viewpoint-estimation benchmarks and promising results for the joint task using predicted classes and bounding boxes.

  • Takeaways & Limitations

    Few-shot viewpoint estimation can remain effective with predicted detections, and exemplar 3D models provide viewpoint-relevant information for novel classes.

  • Takeaways & Limitations

    Performance is constrained for small or nearly symmetrical objects, and the category-agnostic method relies on cross-class pose consistency that is not guaranteed.

Abstract

from arXiv · show

Detecting objects and estimating their viewpoints in images are key tasks of 3D scene understanding. Recent approaches have achieved excellent results on very large benchmarks for object detection and viewpoint estimation. However, performances are still lagging behind for novel object categories with few samples. In this paper, we tackle the problems of few-shot object detection and few-shot viewpoint estimation. We demonstrate on both tasks the benefits of guiding the network prediction with class-representative features extracted from data in different modalities: image patches for object detection, and aligned 3D models for viewpoint estimation. Despite its simplicity, our method outperforms state-of-the-art methods by a large margin on a range of datasets, including PASCAL and COCO for few-shot object detection, and Pascal3D+ and ObjectNet3D for few-shot viewpoint estimation. Furthermore, when the 3D model is not available, we introduce a simple category-agnostic viewpoint estimation method by exploiting geometrical similarities and consistent pose labelling across different classes. While it moderately reduces performance, this approach still obtains better results than previous methods in this setting. Last, for the first time, we tackle the combination of both few-shot tasks, on three challenging benchmarks for viewpoint estimation in the wild, ObjectNet3D, Pascal3D+ and Pix3D, showing very promising results.

1 INTRODUCTION

The paper addresses few-shot object detection and viewpoint estimation by conditioning predictions on class-representative information, including image features and optional exemplar 3D models. It also introduces category-agnostic viewpoint estimation and evaluates the joint detection-and-viewpoint task in the few-shot regime.

  • Motivation: Few-shot methods often fail to generalize to novel object categories because they rely on rich labeled data, while 3D annotation is costly.These constraints motivate transferring knowledge from base categories to novel categories with few annotated samples.
  • Approach: The framework combines query-image features with class-informative features for few-shot object detection and viewpoint estimation.The aggregation module can use support-image information for detection and exemplar 3D-model information for viewpoint estimation.
  • Approach: Category-agnostic viewpoint estimation predicts three Euler angles from image embeddings by exploiting consistent canonical orientations and geometrical similarities across related classes.The method shares network weights across categories and can operate without 3D models during training and testing.
  • Approach: Exemplar 3D models condition viewpoint prediction together with image embeddings and improve performance on novel classes in the few-shot regime.The models need not exactly match the objects in the input image and require viewpoint supervision rather than additional keypoint annotations.
  • Contributions: The paper reports state-of-the-art performance across few-shot detection and viewpoint-estimation benchmarks and evaluates their joint application to novel objects.The joint task predicts both object locations and 3D poses from few-shot data, including evaluations on Pascal3D+ and Pix3D.

2 RELATED WORK

Prior work studies few-shot learning through transferable embeddings, reweighting, keypoints, and category-agnostic viewpoint estimation, while this paper targets the combined detection and viewpoint setting.

  • Few-shot learning transfers knowledge from data-rich base categories to novel categories with few annotated samples.
  • Meta-learning methods include metric-learning approaches that construct transferable embedding spaces from sparse annotations.
  • Object detection with limited annotations: Recent few-shot detection methods generate class-specific coefficient vectors from support samples to reweight query features.
  • Object detection with limited annotations: This paper replaces feature reweighting with feature aggregation and evaluates randomly selected supports to reduce sampling bias.
  • Viewpoint estimation with limited annotations: Existing viewpoint methods may require similar testing categories, exact CAD models, or category-specific keypoints and branches.
  • Viewpoint estimation with limited annotations: The paper instead extracts canonical shape features from few labeled samples using random object models and shares one architecture across base and novel classes.

3 METHOD

The method transfers knowledge from abundant base classes to few-shot novel classes using task-specific class data for detection and optional 3D models for viewpoint estimation.

  • Few-shot learning setup: Viewpoint estimation represents each object with an image crop and three Euler angles: azimuth, elevation, and in-plane rotation.
  • Few-shot learning setup: Object detection class data consists of images augmented with binary masks for objects’ ground-truth bounding boxes.
  • Few-shot learning setup: Viewpoint class data may additionally contain 3D models, sampled as point clouds in normalized canonical object space.
  • Few-shot learning setup: At each training iteration, class data is randomly sampled for every training class.
  • The few-shot setup partitions classes into base and novel categories, with abundant base samples and few novel samples.

3.2 Few-shot Learning with Class Data

The network encodes query inputs and class data separately, aggregates their features, and applies task-specific predictors for detection or viewpoint estimation.

  • Query and class encoders produce feature vectors that an aggregation module combines before task-specific prediction.
  • Few-shot object detection: For detection, the predictor estimates classification scores and bounding boxes for each region of interest and class.
  • Few-shot viewpoint estimation: For viewpoint estimation, the predictor classifies quantized angles and refines them with regressed angular offsets.
  • Few-shot object detection: The detection network uses Faster R-CNN with masked RGB class inputs and shared query and class-encoder weights except for the first layer.
  • Few-shot object detection: The cosine-similarity box classifier uses instance-level feature normalization, which empirically reduces intra-class variance and improves novel-class detection accuracy.
  • Few-shot viewpoint estimation: For viewpoint estimation, image features from ResNet-18 are aggregated with PointNet features from canonical point clouds.
  • Feature aggregation: The aggregation concatenates element-wise products, feature differences, and the original query embedding.
  • Feature aggregation: This partial disentanglement improves few-shot detection and reduces variation caused by random support samples.

3.3 Category-agnostic Viewpoint Estimation

Without 3D models, the method predicts viewpoints from category-agnostic image embeddings, sharing one predictor across classes and fine-tuning with balanced base and novel data.

  • The no-model variant estimates three Euler angles using only an embedding extracted from the object-centered image crop.
  • All object classes share the same category-agnostic feature extraction and viewpoint prediction modules.
  • Training first uses abundant base-class images, followed by fine-tuning on a balanced dataset containing base and novel classes.
  • The approach removes the 3D branch and aggregation module when no 3D model is available.
  • The simple no-model approach outperforms previous methods on few-shot viewpoint estimation.
  • Fine-tuning on both base and novel categories supports learning without forgetting by preventing exclusive focus on novel-category performance.

3.4 Learning Procedure

The learning procedure trains detection in two stages and viewpoint estimation with discretized-angle losses, using class data or exemplar 3D models to construct class features for conditioning.

  • Object detection: Detection uses the same loss as Meta R-CNN, combining proposal, classification, localization, and class-feature diversity objectives.Lrpn handles foreground-background distinction and proposal refinement; Lcls, Lloc, and Lmeta supervise box classification, regression, and feature diversity.
  • Viewpoint estimation: Viewpoint estimation discretizes each Euler angle into 15-degree bins and combines cross-entropy bin classification with Huber offset regression.The viewpoint meta loss is removed so the network learns inter-class similarities rather than box-classification differences.
  • Training data: Training uses all available class 3D models for viewpoint estimation, while detection class data is abundant for base classes and limited to K shots during fine-tuning.Detection uses typically |Zc| = 200 for base training and |Zc| = K during balanced few-shot fine-tuning.
  • Inference: At inference, class features are computed once by averaging all class features used during few-shot fine-tuning.These averaged features are stored as offline class representations rather than randomly sampled during inference.

4 EXPERIMENTS

The experiments evaluate few-shot object detection and viewpoint estimation under established protocols, averaging performance across varied few-shot training samples for fair comparison.

  • Experimental design: The evaluation covers few-shot object detection and viewpoint estimation benchmarks using the same base-novel class splits as prior work.Performance is averaged over multiple runs with different groups of few-shot training examples to obtain a sensible accuracy estimate.

4.1 Few-shot Object Detection

Few-shot detection is evaluated on PASCAL VOC and MS-COCO, where the method generally performs best or improves results across the reported metrics, with aggregation reducing support-sample variation.

  • Datasets and protocol: PASCAL VOC evaluation uses three five-novel-class splits and K = 1, 2, 3, 5, or 10 annotated bounding boxes per novel class.VOC contains 20 categories, with 15 base and 5 novel classes in each split.
  • Datasets and protocol: MS-COCO evaluation uses 20 novel classes and tests K = 10 or 30 annotated bounding boxes per novel class.The remaining 60 COCO categories are treated as base classes.
  • Evaluation metrics: Detection performance is measured with AP0.5 on PASCAL VOC and standard MS-COCO metrics including mAP, AP0.5, and AP0.75.The COCO evaluation also reports size-specific average precision and average recall metrics.
  • Ablation studies: Cosine-similarity classification slightly improves detection results compared with the standard fully connected classifier, although gains are not systematic on novel classes.The comparison is reported for PASCAL VOC and MS-COCO.
  • Ablation studies: The aggregation [fqry ⊙ fcls, fqry − fcls, fqry] yields the best precision and significantly reduces variance from random support-data sampling.Explicitly providing the query-class feature difference improves performance beyond leaving that relation to be learned implicitly.
  • Results: The method achieves the best performance in most PASCAL VOC cases and significantly improves MS-COCO results across all evaluation metrics.The authors report especially strong PASCAL performance as the number of shots increases.

4.2 Few-shot Viewpoint Estimation

Few-shot viewpoint estimation is tested within and across datasets using exemplar 3D models or image-only features, with strong gains over prior methods but remaining errors under ambiguity, clutter, and shape variation.

  • Evaluation setup: Intra-dataset evaluation splits ObjectNet3D into 80 base and 20 novel classes, reporting Acc30 and median rotational error.All models are trained and tested on ObjectNet3D with ground-truth classes and bounding boxes provided at test time.
  • Intra-dataset results: +16 points in Acc30 and a median-error reduction from 31.5° to 15.6° are reported over MetaView on novel ObjectNet3D classes.The method relies only on viewpoint annotations and uses one architecture across object classes, unlike category-specific keypoint detectors.
  • Inter-dataset results: Inter-dataset evaluation trains on 88 ObjectNet3D base classes and tests on 12 Pascal3D+ novel classes to assess cross-domain generalization.This setting uses disjoint source and target categories.
  • Inter-dataset results: +19 points in Acc30 and a median-error reduction from 51.3° to 28.3° are reported over prior methods in inter-dataset evaluation.The authors attribute the improvement to leveraging viewpoint-relevant 3D information for novel-class and domain-shift generalization.
  • 3D representation ablation: Point-cloud 3D representations provide the best overall performance, while 3D models improve novel-class accuracy and reduce support-sample variance.Even one 3D exemplar per class clearly improves performance over image-only estimation.
  • Qualitative analysis: Failures commonly arise from pose-appearance ambiguity, heavy clutter, and large shape variation between training and testing objects.Keypoint-based methods perform somewhat better on some difficult categories, but their predictions can still be grossly wrong.
  • 3D representation ablation: The gap between one and multiple 3D exemplars per class is negligible compared with the gap between using and not using 3D models.Additional exemplars may require more sophisticated feature combination than simple feature averaging to become useful.

4.3 Joint Detection and Viewpoint Estimation

The paper evaluates joint few-shot detection and viewpoint estimation using predicted detections across ObjectNet3D, Pascal3D+, and Pix3D, including realistic cross-dataset settings. Performance remains strong despite imperfect boxes and classes, though failures arise from missed detections, truncation, and viewpoint ambiguity.

  • Evaluation setup: The networks are trained separately, and viewpoint estimation uses predicted classes and bounding boxes rather than ground-truth detections.This setting is more realistic and challenging than evaluation with perfect detections.
  • Evaluation setup: Joint evaluation counts objects with IoU above 0.5 and rotational error below 30° using the correct predicted class.The metric combines detection and viewpoint-estimation correctness.
  • ObjectNet3D: 50% accuracy with predicted classes and boxes exceeds MetaView’s 48% with ground-truth classes and boxes on ObjectNet3D.The comparison is made in the 10-shot setting, where performance generally drops for predicted detections because truncation makes viewpoint estimation harder.
  • Pascal3D+: 35% accuracy with predicted detections exceeds 33% for competing methods on Pascal3D+ despite training and testing on different datasets.All 12 Pascal3D+ categories are treated as novel, making this an inter-dataset evaluation.
  • Qualitative results: Qualitative failures mainly result from missed detections, wrong classifications, truncated objects, and appearance ambiguities, especially for small or symmetric objects.Successes and failures are visualized on ObjectNet3D and Pascal3D+ using projected CAD models.
  • Pix3D: On Pix3D, performance drops from 91% to 72%, yet the viewpoint network achieves 72% after training on only 10 annotated samples per testing category.The evaluation covers bed, chair, sofa, and table, using predicted 2D boxes from GP2C.

4.4 Limitations

The method remains limited on small and nearly symmetric objects, and its category-agnostic viewpoint approach assumes consistent orientation across related classes. That orientation consistency is common in available datasets but is not guaranteed.

  • Object appearance: Small objects provide fewer visible cues, while nearly symmetric objects can produce large viewpoint errors from front-back orientation mistakes.Knives are given as an example, and rendered views may still resemble the actual image despite the orientation error.
  • Object appearance: Preventing failures on almost-symmetric objects may require specific treatment of their symmetries.This limitation is shared with other viewpoint-estimation work and is more pronounced in few-shot settings.
  • Category-agnostic viewpoint estimation: The category-agnostic approach relies on related classes having consistently oriented, geometrically aligned objects, but this assumption is not guaranteed.Datasets can be normalized with systematic rotations of ground-truth viewpoints when orientation discrepancies occur.

5 CONCLUSION AND PERSPECTIVES

The paper presents a coherent framework for few-shot object detection and viewpoint estimation and reports improved accuracy across standard benchmarks. It also demonstrates promising joint-task performance on novel objects detected in realistic, imperfect-detection settings.

  • Conclusion: The framework addresses few-shot object detection and viewpoint estimation together in a coherent and efficient approach.The authors report accuracy improvements over the state of the art on several standard benchmarks.
  • Conclusion: Joint viewpoint estimation remains promising when applied to novel objects detected by the few-shot detector rather than perfect ground-truth detections.The comparison is described as adversarial because competing methods are evaluated with ground-truth classes and bounding boxes.
  • Perspectives: Future work will pursue category-agnostic detection and pose estimation, plus 3D model retrieval and refinement using visual agreement.These directions target arbitrary unseen objects and improved 3D candidate selection.
Loading 2007.12107v2…