PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias

Stefan Stojanov, Anh Thai, James M. Rehg

arXiv:2101.07296v2cs.CVcs.LG

TL;DR

The paper asks how 3D shape can support image-based low-shot categorization, a setting where its use had not been investigated and suitable large-category shape datasets were lacking. It learns image embeddings aligned to a discriminative shape space and reports improved generalization over image-only training, while introducing Toys4K. The conclusions are bounded by reliance on synthetic object data.

  • Problem

    How to use 3D shape to improve image-based low-shot learning was unclear, and existing 3D datasets lacked enough categories for effective class-disjoint evaluation.

    Source: §3 · p. 3

  • Method

    The method trains a discriminative point-cloud embedding and learns an image encoder that maps images into that shape embedding space.

    Source: §3.1 · p. 4

  • Results

    The shape-biased approach improves generalization accuracy over image-based training alone, and the paper introduces Toys4K as a diverse dataset with the largest number of categories available to date.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The work frames natural visual biases such as object shape as a direction for designing few-shot architectures.

    Source: §5 · p. 8

    Because the findings depend on synthetic object data, their generality to other conditions is limited.

    Source: §5 · p. 8

Abstract

from arXiv · show

It is widely accepted that reasoning about object shape is important for object recognition. However, the most powerful object recognition methods today do not explicitly make use of object shape during learning. In this work, motivated by recent developments in low-shot learning, findings in developmental psychology, and the increased use of synthetic data in computer vision research, we investigate how reasoning about 3D shape can be used to improve low-shot learning methods' generalization performance. We propose a new way to improve existing low-shot learning approaches by learning a discriminative embedding space using 3D object shape, and using this embedding by learning how to map images into it. Our new approach improves the performance of image-only low-shot learning approaches on multiple datasets. We also introduce Toys4K, a 3D object dataset with the largest number of object categories currently available, which supports low-shot learning.

1. Introduction

The introduction asks whether 3D shape can inform image-based few-shot categorization, given evidence of shape bias in rapid learning and a texture bias in image-trained CNNs. It proposes shape-guided image embeddings and Toys4K to investigate that question.

1. Introduction

  • Infant-learning studies suggest shape cues dominate color and texture when people categorize novel objects from a few examples, but this use of shape for image-based low-shot learning had not been investigated.The cited CNN study found a texture bias, but the authors caution that it does not establish whether shape could be used more effectively or how much the bias reflects ImageNet’s composition.Source: §1 · p. 1
  • Toys4K adds 4,179 3D objects across 105 categories, addressing the limited category scale of existing datasets for testing low-shot categorization.The categories were selected to include objects commonly encountered by infants and children.Source: §1 · p. 1
  • The approach trains a discriminative shape embedding from 3D objects, then learns an image encoder to map images into that embedding space.The introduction describes the shape space as more discriminative than image-based spaces.Source: Figure 1 · p. 2, §1 · p. 2
  • The proposed shape-biased image embedding improves generalization accuracy over image-based training alone across a representative set of state-of-the-art few-shot architectures.The paper also reports that purely shape-based few-shot learning outperforms image-based approaches, establishing an empirical upper bound on shape bias.Source: §1 · p. 2

2. Related Work

Related work covers synthetic-object recognition, low-shot learning, 3D shape datasets, and multimodal representation learning. The paper distinguishes its focus on combining image appearance and 3D shape for generalization to novel categories.

2. Related Work

  • Prior comparisons of point clouds, depth maps, voxels, and images focused on standard supervised classification, not low-shot classification of novel categories or combining shape and appearance.Existing work separately studied appearance-, point-cloud-, and voxel-based recognition of synthetic objects.Source: §2 · p. 2
  • Optimization-based methods learn representations for rapid gradient adaptation, while metric-based methods learn similarity measures intended to discriminate novel classes.The paper places its shape-bias approach in the metric-learning family and describes it as the first to combine appearance and shape for low-shot learning.Source: §2 · p. 2
  • Toys4K is curated for generalization to new classes and has more object categories than the compared datasets despite its smaller total size.ModelNet40 has 40 categories and ShapeNetCore.v2 has 55; ShapeNetSem is unsuitable for recognition because instances can have multiple categories, while ABC and Thingi10k lack category annotations.Source: §2 · pp. 2–3
  • Unlike prior low-shot work combining image appearance with language vectors at query time, this approach uses image and shape information for the support set and image-only queries.Image–mesh embedding work instead focuses on retrieval within categories seen during training and testing.Source: §2 · p. 3

3. Using Shape for Low-Shot Classification

The paper combines 3D shape and image information to improve low-shot image categorization, and introduces Toys4K to support experiments across many object categories. Its method maps images into a shape-trained embedding space, while low-shot queries remain image-only.

3. Using Shape for Low-Shot Classification

  • Low-shot learning with shape is constrained by the need to generalize across disjoint classes, while existing shape datasets support only 10-way or 20-way testing.The authors identify both the difficulty of using shape to improve image-based learning and the limited category counts of existing datasets as barriers.Source: §3 · p. 3

3.2. Toys4K Dataset

  • Toys4K contains 4,179 instances in 105 categories, with at least 15 instances per category, and supports up to 50-way classification.Its categories were developed with developmental-psychology experts; objects were manually selected and aligned within each category. The table caption describes Toys4K as having the most categories among available 3D-object datasets.Source: §3 · p. 3, §3.2 · p. 5, Table 2 · p. 5

3.1. Low-Shot Learning with Shape Bias

  • Point-cloud-based low-shot learning generalizes better than image-based learning, and shape-biased training improves image-based low-shot generalization.On ModelNet40-LS, the reported comparison favors point-cloud learning; the table caption identifies the comparison but supplies no numerical values.Source: Table 1 · p. 4, §3.1 · p. 4
  • The method trains an image mapping to match point-cloud embeddings and their pairwise distances, then averages image and shape embeddings for support prototypes while classifying image-only queries by nearest centroid.The two training losses are minimized with equal weight.Source: §3.1 · pp. 4–5
  • When images from base, validation, and test classes are available, the Image + Point Cloud Oracle closely matches shape-only performance and significantly outperforms the image-based approach.This comparison suggests that extrapolating shape-embedding metric properties to novel categories is a key challenge.Source: §3.1 · p. 5

4. Experiments

The experiments evaluate shape-biased low-shot learning on three datasets against image-only methods and other baselines. Across the benchmarks, shape bias improves one-shot results, while pairwise loss helps the image-to-shape mapping better approximate the shape embedding space.

4.1. Datasets

  • Evaluation uses rendered images and sampled surface point clouds from Toys4K, ModelNet40-LS, and ShapeNet-LS.Toys4K uses 40 base, 10 validation, and 55 test classes; the corresponding splits are 20/10/10 for ModelNet40-LS and 25/10/20 for ShapeNet-LS.Source: §4.1 · p. 6

4.2. Baselines

  • The baselines include Prototypical Networks, FEAT, RFS, SimpleShot, and a joint image–shape triplet-loss method.Image encoders use ResNet18 and point-cloud encoders use DGCNN; the low-shot baselines were validated against their original results.Source: §4.2 · p. 6

4.3. Explicit Shape-Bias Improves Image-Based Generalization

  • Shape bias improved one-shot SimpleShot and FEAT performance by up to 6%-points in low-data settings.Tables 3–5 cover ModelNet40-LS, Toys4K, and ShapeNet55-LS; ShapeNet55-LS shows consistent improvement for both methods.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
  • Shape-biased models without point clouds at low-shot inference also improve one-shot performance, with good five-shot performance.One-shot gains increase with the number of ways; SimpleShot maps images directly for nearest-class-mean classification, while FEAT adds its Transformer module.Source: §4.3 · p. 7

4.4. Analysis of Pairwise Loss

  • Adding pairwise loss L2 improves the learned mapping's approximation of the point-cloud embedding on training and novel classes.The analysis reports that this term significantly improves SimpleShot with shape bias.Source: §4.4 · pp. 7–8, Figure 5 · p. 7

4.5. Shape Bias and Failure Analysis

  • Shape-biased low-shot accuracy correlates more strongly with the point-cloud model than image-only accuracy does.The analysis also reports room to improve image-to-shape mappings, as shape-biased SimpleShot errors overlap with point-cloud SimpleShot errors.Source: §4.5 · p. 8, Table 6 · p. 8

5. Discussion and Conclusion

The paper presents shape bias as a first step toward improving low-shot object categorization and introduces Toys4K, a dataset with the largest number of categories available. Its findings depend on synthetic object data, which limits conclusions about shape bias in more general conditions.

5. Discussion and Conclusion

  • The authors characterize the work as a first step in investigating shape bias for low-shot object categorization.They report improved generalization and introduce Toys4K as a diverse, challenging dataset with the largest number of categories available to date.Source: §5 · p. 8
  • Shape-biased classifications correlate more strongly with point-cloud classifications than image-only classifications do.This comparison is reported over the same 5K low-shot episodes.Source: Figure 6 · p. 8
  • Dependence on synthetic object data limits conclusions about shape bias under more general conditions.The authors call synthetic data essential for obtaining matched 2D and 3D data at sufficient scale.Source: §5 · p. 8
  • The paper frames exploiting visual-world biases such as object shape as a direction for designing few-shot architectures.It identifies exploiting other sources of bias as a future direction.Source: §5 · p. 8

A. Further Dataset Details

The supplementary dataset details describe image and point-cloud inputs, augmentation, and the class splits used for three low-shot datasets.

A. Further Dataset Details

  • Images use 224 × 224 RGB inputs, while point clouds use 1,024 randomly sampled 3D coordinates.Image augmentations include geometric transformations and color jitter; point-cloud augmentations include translation, jittering, and dropout.Source: A. Further Dataset Details · p. 12

A.1. Toys4K

  • Toys4K uses 40 training, 10 validation, and 55 test classes, with episodes of up to 5 shots and 10 queries.Its validation and test classes are randomly selected from the categories outside the training split.Source: §A.1 · p. 12

A.2. ModelNet40-LS

  • ModelNet40-LS uses 20 training, 10 validation, and 10 test classes, with up to 5 shots and 15 queries per episode.Source: §A.2 · p. 12

A.3. ShapeNet55-LS

  • ShapeNet55-LS uses a split of 25 training, 10 validation, and 20 test classes; validation and testing generate low-shot episodes with up to 5 shots and 15 queries.Source: §A.3 · p. 12

B. Further Low-Shot Analysis

The supplementary analysis examines low-shot confusion matrices for ModelNet40-LS and ShapeNet55-LS, reporting that they reflect the main-text improvement from adding shape bias.

B. Further Low-Shot Analysis

B.1. Confusion Matrices

  • Confusion matrices aggregate 5K episodes: 10-way for ModelNet40-LS and 20-way for ShapeNet55-LS.The matrices reflect the main-text finding that adding shape bias improves overall low-shot classification performance.Source: §B.1 · p. 12

B.2. Per-episode Analysis

Per-episode examples show shape bias correcting image-only classification errors, while the section also documents baseline implementations and SimpleShot training choices.

B.2. Per-episode Analysis

  • In some episodes without view ambiguities, the image-only model misclassifies while the shape-biased model correctly classifies, including a bicycle confused with a sheep.The analysis is presented in Figure 10.Source: §B.2 · p. 12

C. Baseline Algorithm Details

C.1. SimpleShot

  • 50.60(0.34) versus 49.69(0.19) for 1-shot 5-way accuracy, and 68.06(0.23) versus 66.92(0.17) for 5-shot 5-way accuracy, are the reimplementation and author-reported SimpleShot results.SimpleShot uses SGD with learning-rate decay at epochs 300 and 360 over 400 epochs; the reported result selects the best of three feature-normalization strategies.Source: §C.1 · p. 12

C.2. RFS

This section reports RFS and FEAT implementation details and reimplementation results, alongside illustrations of rendered samples and Toys4K category composition.

C.2. RFS

  • 54.59(0.86) versus 53.73(0.81) for 1-shot 5-way miniImageNet accuracy are the RFS reimplementation and original-code results.RFS trains an embedding with cross-entropy and fits a logistic-regression classifier for each low-shot episode.Source: §C.2 · pp. 12–13
  • Rendered samples from ShapeNet, ModelNet, and Toys4K illustrate high image quality from ray-tracing-based rendering.Source: Figure 7 · p. 13
  • The Toys4K category-composition table presents the dataset’s category composition.Source: Table 7 · p. 13

C.3. FEAT

  • 54.85(0.20) versus 54.85(0.20) for 1-shot 5-way accuracy, and 71.45(0.73) versus 71.61 for 5-shot 5-way accuracy, are the FEAT reimplementation and original-code results.The authors train separate FEAT models by way and shot setting; for shape-biased FEAT they omit learning-rate scheduling and momentum because these hurt performance.Source: §C.3 · p. 13

C.4. Protoypical Networks

The supplied confusion-matrix captions report improved shape-biased SimpleShot performance on two low-shot datasets; the section also details the Prototypical Networks reimplementation.

C.4. Protoypical Networks

  • On ModelNet-LS, the shape-biased image embedding improves performance without point clouds at test time, and adding point-cloud support information improves it further.The caption directs readers to Table 3 for aggregate results.Source: Figure 8 · p. 14
  • On ShapeNet-LS, the shape-biased image embedding improves performance without point clouds at test time, and point-cloud support information improves it further.The caption directs readers to Table 5 for aggregate results.Source: Figure 9 · p. 14
  • 66.94(0.71) versus 66.68(0.68) for 5-shot 5-way miniImageNet accuracy are the Prototypical Networks reimplementation and reported reimplementation results.The implementation trains separate 1-shot and 5-shot models and uses 20-way training.Source: §C.4 · pp. 13–14

C.5. Triplet Model

The triplet model learns a joint image-and-shape embedding, while the point-cloud embedding algorithm follows SimpleShot and classifies queries by nearest-neighbor cosine similarity.

C.5. Triplet Model

  • The joint triplet model uses image and point-cloud encoders, with the same-instance point-cloud encoding as the positive and a different-instance encoding as the negative.Category-based pair construction performed worse in the authors’ experiments.Source: §C.5 · pp. 14–15
  • Figure 10 shows Toys4K low-shot episodes where shape bias allows improved performance.The figure displays image and point-cloud shots, image queries, and model predictions.Source: Figure 10 · p. 15

D.1. Algorithm

  • The point-cloud embedding algorithm is based on SimpleShot and classifies each query by nearest-neighbor cosine similarity using support features and labels.AccAccumulator collects validation accuracy across episodes; the model uses pooled architecture features.Source: §D.1 · p. 15

D.2. Architecture Study

DGCNN with random SO3 point-cloud rotations performs best in the ModelNet40-LS architecture study, and the selected rotation strategy is used in shape-embedding experiments. The section also specifies the training setup for the shape-biased image embedding, SimpleShot, and FEAT.

D.2. Architecture Study

  • DGCNN performs best among the tested point-cloud architectures, and random SO3 rotations improve low-shot generalization on ModelNet40-LS.The selected SO3 strategy is used for all shape-embedding space learning experiments.Source: §D.2 · p. 15, Table 8 · p. 16

E. Details for Learning a Shape Biased Image Embedding

  • The shape-biased image embedding is trained for 400 epochs with Adam, batch size 256, and learning-rate decay at epochs 300 and 360.The initial learning rate is 0.001, and the L2 weight penalty is 0.0001.Source: E. Details for Learning a Shape Biased… · p. 16

E.1. SimpleShot with Shape Bias

  • SimpleShot uses nearest-centroid classification in the image embedding space and requires no parameter updates during low-shot testing.Its image embedding function is trained using Algorithm 2.Source: §E.1 · p. 16

E.2. FEAT with Shape Bias

  • For shape-biased FEAT, the image embedding is fine-tuned from the Algorithm 2 mapping while the set-to-set function is trained from scratch.The experiment uses FEAT's recommended default hyperparameters.Source: §E.2 · p. 16
Loading 2101.07296v2…