PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch-based 3D Shape Retrieval using Convolutional Neural Networks

Fang Wang, Le Kang, Yi Li

arXiv:1504.03504v1cs.CV

TL;DR

The paper addresses ambiguous best-view selection and manually crafted features in sketch-based 3D retrieval. It uses two predefined views and two Siamese CNNs with within- and cross-domain similarity learning, significantly outperforming state-of-the-art methods on three large datasets.

  • Problem

    Existing retrieval methods rely on subjective best views and hand-crafted features, making sketch-to-view matching inputs ambiguous.

  • Method

    The method uses two predefined directions per object, separate Siamese CNNs for sketches and views, and an alignment loss over within- and cross-domain similarities.

  • Results

    The method significantly outperforms state-of-the-art approaches across multiple metrics on three large datasets.

  • Takeaways & Limitations

    Learning cross-domain similarities removes the need for the view-similarity assumption and supports direct comparison of sketch and view features.

  • Takeaways & Limitations

    The paper identifies semantic ambiguity between some shapes as beyond its scope, and reports noticeable overfitting before training stops.

Abstract

from arXiv · show

Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of "best views" are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection -- matching) is pragmatic but also problematic because the "best views" are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of "best views" and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics.

1. Introduction

Sketch-based 3D retrieval avoids ambiguous best-view selection by using two predefined viewpoints and learned representations for sketches and views. Two Siamese CNNs align the domains, yielding significantly stronger retrieval results on three large datasets.

  • Sketch queries are attractive because hand-drawn input is easy while remaining rich enough to specify shapes.
  • Existing methods project 3D models into multiple views and manually define features to match sketches with projections.
  • Best-view selection is ambiguous because viewpoint similarity is not guaranteed, and training may require manually selected viewpoints.
  • The proposed minimalism approach uses two significantly different predefined viewpoints per upright object instead of selecting many best views.
  • Two domain-specific Siamese CNNs learn representations for sketches and views, while an alignment loss maps both sources into a shared target space.
  • The method significantly outperforms state-of-the-art approaches across multiple metrics, including precision-recall and nearest-neighbor retrieval, on three large datasets.

2. Related work

Related work covers datasets, handcrafted features, and similarity learning for sketch-based 3D retrieval. It motivates learned CNN and Siamese representations as alternatives to feature engineering and shallow learning.

  • Datasets: Research datasets include the Princeton Shape Benchmark, SHREC’14 Benchmark, Bonn Architecture Benchmark, and large sketch collections organized for retrieval challenges.
  • Features: 3D retrieval has used global shape descriptors, distance functions, local features, and bag-of-words representations.
  • Features: Sketch matching commonly combines boundary information and internal structures from line drawings, with methods including shape context, BF-DSIFT, and GALIF.
  • Features: Other approaches use geometry, graph-based features, or semantic labeling, whereas this paper focuses on view-based methods using 2D shape features.
  • CNN and Siamese network: CNNs learn complicated mappings from raw images with less domain knowledge than handcrafted features and shallow learning frameworks.
  • CNN and Siamese network: Siamese networks learn similar output vectors for labeled similar pairs and dissimilar vectors for labeled dissimilar pairs in weakly supervised metric learning.

3. Learning feature representations for sketch based 3D shape retrieval

The paper learns separate sketch and view representations with Siamese CNNs, aligning within-domain and cross-domain similarities without requiring viewpoint similarity. It uses two predefined viewpoints and renders line drawings from closed boundaries and Suggestive Contours.

  • Two networks, one loss: The pairwise loss brings similar outputs closer and pushes dissimilar outputs apart using a binary similarity label and feature distance.The loss is applied to sketch pairs, view pairs, and sketch-view pairs; category labels are not used in the framework.
  • An illustrated example: Cross-domain metric learning aligns sketch and view features using within-domain and cross-domain pair similarities.After alignment, matching supports sketch-sketch, view-view, and sketch-view retrieval.
  • Two networks, one loss: Two Siamese networks separately process sketches and views because the domains have distinctive intrinsic properties.The networks share the same design but are learned separately.
  • View definitions and line drawing rendering: Two viewpoints are generated for the whole dataset, separated by more than 45 degrees, under the assumption that most models are upright.The method argues that two sufficiently different views are unlikely both to be degenerate and reduce matching cost.
  • View definitions and line drawing rendering: The approach does not rule out using more best views, but comparing view-selection methods is outside the paper’s scope.
  • View definitions and line drawing rendering: 3D models are rendered as 2D line drawings using closed boundaries and Suggestive Contours.Both outer and internal edges are treated as important in the rendering process.

4. Experiments

Experiments on three large datasets evaluate cross-domain retrieval and additionally report within-domain retrieval, with the method outperforming state-of-the-art approaches across recognized metrics.

  • Experiments on three large datasets show significant outperformance over state-of-the-art approaches across precision-recall and nearest-neighbor metrics.The study also reports effective retrieval within each domain and notes fast filtering-based computation.

4.1. Datasets

The experiments use PSB/SBSR and reorganized SHREC datasets, whose differing sketch counts, class distributions, model orientations, and scales create distinct evaluation challenges.

  • PSB / SBSR dataset: PSB contains 1814 3D models divided equally into training and testing sets, while SBSR provides 1814 hand-drawn sketches collected from category names.SBSR participants received no visual clues from the 3D models.
  • SHREC’13 & ’14 dataset: The original SBSR sketch distribution is uneven, with 27 of 90 training classes having no more than five instances and some classes having 50.This imbalance motivates reorganized evaluation datasets.
  • SHREC’13 & ’14 dataset: SHREC’13 balances each class at 80 sketches, split into 50 training and 30 testing instances, while model counts remain uneven.The largest class has 184 models, and 23 classes contain no more than 5 models.
  • SHREC’13 & ’14 dataset: SHREC’14 expands the benchmark to 8987 models and twice as many classes, making retrieval more challenging because models come from varied sources and orientations.The best reported algorithm reaches only 0.16 top-1 accuracy.
  • Evaluation criteria: Performance is measured using precision-recall, mAP, nearest neighbor, E-Measure, first and second tier, and DCG.Nearest neighbor measures top-1 retrieval accuracy, while mAP is the area under the precision-recall curve.

4.2. Experimental settings

The experimental setup uses sampled Siamese training pairs, fixed epoch limits without validation data, two views, and measurements of retrieval and computational cost.

  • Stopping criteria: No validation set is specified, so training ends after 50 epochs for PSB/SBSR and 20 for SHREC’13, or upon convergence.Multiple runs are performed and mean values are reported.
  • Generating pairs for Siamese network: Training uses two matched views and 20 unmatched views per sketch, making dissimilar pairs approximately ten times more frequent.The pairing parameters are kp = 2 and kn = 20.
  • PSB/SBSR retrieval examples: Retrieval examples show eight queries from different classes, with each row listing the top 15 results and cyan marking correct retrievals.
  • Computational cost: On SHREC’13, retrieval takes approximately 0.002 sec per query on average when view features are preprocessed.Timing uses a PC with a 2.8GHz CPU and GTX 780 GPU.
  • Computational cost: Training takes approximately 2.5 hours for PSB/SBSR and 6 hours for SHREC’13.Training time depends on the total number of pairs and epochs.
  • Using more than two views yields no significant performance gain through ten views but substantially increases computational cost and GPU-memory use.This result motivates selecting only two views.

4.3. Shape retrieval on PSB/SBSR dataset

On PSB/SBSR, the method retrieves effectively across both seen and unseen classes, while semantic ambiguity remains outside its scope.

  • Examples: Retrieval examples show effective performance across queries, with cyan indicating correct retrievals in the reported examples.The section also examines retrieval on unseen classes.
  • Examples: The method performs exceptionally well on popular classes such as human, face, and plane, but fine-grained distinctions like shelf versus box remain difficult.These classes differ only in small parts of their models.
  • Limitations: Semantic ambiguity between classes such as barn and house is beyond the scope of the paper.The distinction between these classes is functional rather than strongly visual.
  • Statistics: 10% higher precision-recall values are reported than Fig. 9 in on this dataset.The comparison is based on fixed-point precision-recall values.
  • Unseen classes: Unseen classes comprise 71 testing-only classes, making PSB/SBSR useful for investigating whether similarity learning is biased.Only 21 classes appear in both training and testing sets.
  • Unseen classes: The method retrieves similar shapes even in a flower failure case, where a potting plant is returned.The authors present this as evidence that the method learns similarity effectively.

4.4. Shape retrieval on SHREC’13 dataset

On SHREC’13, the learned features organize related sketches and views while outperforming comparison methods across retrieval metrics. The results also indicate that sketch inconsistency is the main challenge, and training shows noticeable overfitting.

  • A visualization of the learned features: PCA visualization shows sketch and view features, represented by green and yellow points, respectively, in a two-dimensional feature space.The learned features are reduced to two dimensions for visualization.
  • A visualization of the learned features: Similar shapes are grouped into neighboring regions, including animals, vehicles, houses, and churches, supporting within-domain and cross-domain retrieval.The visualization is coarse and the method is weakly supervised.
  • Statistical results: 10% performance gain is observed at small recall, increasing to 30% when recall reaches 1 on SHREC’13.The method’s curve also decreases more slowly than other methods as recall increases.
  • Statistical results: Noticeable overfitting occurs during training when the stopping criterion is reached.The authors suggest performance could improve through further tuning of network structure and training.
  • Statistical results: The method performs better in every reported metric on both SHREC’13 and SHREC’14 benchmarks.The comparison is summarized in Table 3.
  • Statistical results: Using identical Siamese networks for sketches and views is inferior to the proposed separate-network configuration, though it remains better than other methods.This supports the hypothesis that the two domains have different variations.
  • Statistical results: Within-domain retrieval results demonstrate learned features for both domains, with the view domain more consistent than the sketch domain.The comparison identifies sketch inconsistency as the most challenging issue.

5. Conclusion

The paper proposes learned feature representations for sketch-based 3D shape retrieval, replacing best-view matching with predefined viewpoints and Siamese CNNs. Experiments on three large datasets found the method superior.

  • The method learns feature representations for sketch-based 3D shape retrieval.
  • Predefined viewpoints and two Siamese CNNs replace best-view computation and matching.
  • Experiments on three large datasets demonstrated that the method is superior.
Loading 1504.03504v1…