PaperlayerView exact revision on arXiv

Source-linked AI summary

SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

Li Yi, Hao Su, Xingwen Guo, Leonidas Guibas

arXiv:1612.00606v1cs.CV

TL;DR

Semantic annotation on irregular, non-isomorphic 3D shape graphs requires both within-shape multiscale processing and cross-shape information sharing. SyncSpecCNN addresses these needs with spectral dilated kernels and a spectral transformer that synchronizes graph spectral domains, achieving state-of-the-art performance on segmentation and keypoint tasks.

  • Problem

    Irregular, non-isomorphic shape graphs make convolutional weight sharing and information sharing across related shapes difficult, limiting semantic annotation of new 3D models.

  • Method

    SyncSpecCNN uses spectral dilated convolution kernels for multiscale analysis and a Spectral Transformer Network to synchronize spectral domains across shapes.

  • Results

    SyncSpecCNN achieves state-of-the-art performance across evaluated semantic annotation tasks, including 3D part segmentation and 3D keypoint prediction.

  • Takeaways & Limitations

    The framework supports general vertex-function prediction beyond part segmentation, with experiments demonstrating applications to keypoints and other shape annotations.

  • Takeaways & Limitations

    Aligning diverse shapes to one average shape can cause unwanted distortion, motivating the use of multiple average shapes.

Abstract

from arXiv · show

In this paper, we study the problem of semantic annotation on 3D models that are represented as shape graphs. A functional view is taken to represent localized information on graphs, so that annotations such as part segment or keypoint are nothing but 0-1 indicator vertex functions. Compared with images that are 2D grids, shape graphs are irregular and non-isomorphic data structures. To enable the prediction of vertex functions on them by convolutional neural networks, we resort to spectral CNN method that enables weight sharing by parameterizing kernels in the spectral domain spanned by graph laplacian eigenbases. Under this setting, our network, named SyncSpecCNN, strive to overcome two key challenges: how to share coefficients and conduct multi-scale analysis in different parts of the graph for a single shape, and how to share information across related but different shapes that may be represented by very different graphs. Towards these goals, we introduce a spectral parameterization of dilated convolutional kernels and a spectral transformer network. Experimentally we tested our SyncSpecCNN on various tasks, including 3D shape part segmentation and 3D keypoint prediction. State-of-the-art performance has been achieved on all benchmark datasets.

1. Introduction

The paper frames semantic annotation on irregular 3D shape graphs as a function-learning problem and introduces SyncSpecCNN to address multiscale processing within shapes and information sharing across different shapes.

  • 3D part understanding supports geometric, stylistic, and functional interpretation, motivating inference of parts and keypoints on new models.
  • Mesh irregularity prevents direct use of grid-based deep-learning optimizations such as convolutional weight sharing.
  • SyncSpecCNN learns vertex functions from annotated training shapes and predicts corresponding functions on new 3D models using spectral graph representations.
  • The architecture addresses within-shape coefficient sharing and multiscale analysis, plus cross-shape sharing across very different graphs.
  • SyncSpecCNN achieves state-of-the-art performance on 3D part segmentation and 3D keypoint prediction tasks.
  • The method introduces spectral dilated kernels for multiscale analysis and a Spectral Transformer Network for sharing coefficients across non-isometric shapes.

2. Background

Graph CNNs extend convolutional learning to irregular graphs, while spectral representations and functional maps provide the basis for multiscale processing and synchronization across shape graphs.

  • Supervised 3D segmentation methods commonly classify geometric features, while spectral analysis has also been used for unsupervised shape segmentation.
  • A 3D shape is modeled as a graph whose vertices lie in R3 and whose edges connect nearby points; vertex functions encode information such as segments.
  • Graph Laplacian eigenvectors form orthogonal bases for vertex-function spaces, with eigenvalues serving as frequencies related to function smoothness.
  • Spectral convolution extends the Fourier convolution theorem: graph convolution becomes pointwise multiplication of spectral representations.
  • Functional maps align functions between shape-specific spectral domains because varied graph bases make vertex functions otherwise incomparable.
  • SyncSpecCNN explicitly aggregates multiscale information and synchronizes spectral domains to share parameters among very different shape graphs.

3. Problem

The problem is to predict per-vertex labels on a graph representation of a 3D shape, using vertex functions and geometric features as inputs.

  • Given a shape graph G = (V, E), the task is to predict per-vertex labels such as segmentation or keypoints, represented as functions f: V → R^K.
  • Precomputed vertex features capture location, curvature, and local context, while the graph Laplacian supplies spectral frequencies for the functional space.
  • The framework uses one graph vertex function as input and predicts another as output, supporting multiple annotation goals.

4.1. Overview

SyncSpecCNN combines spatial-domain neural-network operations with spectral graph convolutions, using dilated spectral kernels to enlarge effective context without increasing parameter count.

  • SyncSpecCNN repeats convolution of vertex functions followed by nonlinear transformations, similar to a fully convolutional segmentation network.
  • Graph convolution is performed by modulation in the spectral domain rather than by conventional spatial convolution.
  • Spectral kernels use a dilated parameterization that enlarges effective kernel size for large-context capture without increasing the number of parameters.

4.2. Network Architecture

SyncSpecCNN combines spatial-domain nonlinear processing with spectral graph convolution, using SpecTN to synchronize spectral domains and enable parameter sharing across shapes.

  • Network components: The architecture includes Forward Transform, Backward Transform, Spectral Multiplication, and Spectral Transformer Network modules alongside standard CNN layers.ReLU, DropOut, 1×1 Convolution, and BatchNormalization operate on graph vertex functions in the spatial domain.
  • Layer configuration: Table 1 specifies each layer’s dilation parameter, SpecTN usage, kernel parameters, and output-channel count.The dilation parameter controls convolution kernel size.
  • Convolution block: Each convolution block transforms a vertex function into spectral coefficients, synchronizes them with a functional map, modulates them pointwise, and transforms the result back.Skip links are added to facilitate information flow between earlier and later layers.
  • Spectral synchronization: SpecTN is omitted for small kernels because their smoothly varying spectral multipliers are less sensitive to basis misalignment across shape graphs.The transformer is mainly needed when spectral domains differ substantially, particularly for larger kernels.

4.3. Spectral Dilated Kernel Parameterization

The paper parameterizes spectral convolution kernels with dilation-controlled modulated exponential windows, enabling multi-scale aggregation and larger receptive fields with fewer parameters.

  • Motivation: Spectral dilated kernels adapt the image-segmentation idea of enlarging kernel size without increasing parameters to graph convolutions in the spectral domain.The approach is intended to capture context at multiple scales.
  • Spectral representation: Spectral coefficients are indexed by Laplacian frequencies λ_i, and the kernel multipliers are treated as a function of those frequencies.Each λ_i corresponds to an eigenbasis capturing intrinsic shape geometry.
  • Kernel smoothness: The multiplier function is bandwidth-constrained because low-frequency concentration produces smooth spatial kernels, while multiplier smoothness controls spatial kernel size.This constraint reduces the number of learnable parameters.
  • Dilation parameterization: Each layer uses a dilation parameter γ and modulated exponential windows with sine/cosine terms to increase kernel expressiveness.The parameterization uses 2n + 1 learnable parameters, while n controls their number.
  • Dilation effects: Large γ yields rapidly changing, narrow-band multipliers and smooth kernels with large spatial support, whereas small γ yields compact-support kernels.The dilation parameter therefore controls the spectral-to-spatial scale trade-off.
  • Cross-shape behavior: The same spectral kernel representation can produce different spatial kernels on different spectral domains, especially when the kernel is large.This motivates synchronizing spectral domains before sharing large kernels across shape graphs.
  • Advantages: The parameterization aggregates multiple scales, enlarges receptive fields compactly to mitigate overfitting, and reduces computation through fewer parameters.These benefits arise from varying kernel sizes across layers while keeping the parameterization compact.

4.4. Spectral Transformer Network

SpecTN synchronizes spectral domains by predicting functional maps to a canonical space, enabling comparable representations and shared spectral kernels across shapes. Its optimization uses reduced low-frequency bases, regularization, and pretraining with externally computed maps.

  • Basic idea: Different spectral domains can produce different vertex functions from the same kernel parametrization, especially for large kernels.Synchronization is therefore necessary for sharing large-kernel parameters across different shape graphs.
  • Basic idea: SpecTN predicts a matrix C that transforms spectral coefficients α into synchronized coefficients α′ = Cα.The transformed coefficients are passed to subsequent network modules.
  • Input to SpecTN: The network uses voxelized Laplacian eigenbases as SpecTN input because all voxel functions occupy a common 3D volumetric space.Rough shape graph correspondences can further facilitate transform prediction.
  • Optimization of SpecTN: SpecTN optimization is difficult because its transformation has quadratically many parameters in the number of spectral bases.The authors address this with reduced bases, orthogonality regularization, and initialization from precomputed functional maps.
  • Optimization of SpecTN: Pretraining SpecTN against externally computed functional maps is crucial before task-specific fine-tuning.The pretrained network is plugged into SyncSpecCNN and then fine-tuned for tasks such as shape segmentation.
  • Optimization of SpecTN: A single average shape may distort alignment for diverse shapes, so the method uses multiple average shapes and assigns each input to its closest one.The canonical domain combines the spectral domains of the selected average shapes.

4.5. Implementation Details

The implementation constructs k-nearest-neighbor graphs from point clouds, computes normalized graph Laplacians, and retains low-frequency eigenbases. SpecTN synchronizes the first 15 bases into a 45-dimensional canonical domain.

  • Graph construction: Input point clouds contain around 2000–3000 points, and the graph is built with k = 6 nearest neighbors.Edge weights use inverse squared distances for connected points.
  • Spectral representation: The normalized graph Laplacian is L = I − D^-1/2WD^-1/2, and the smallest 100 eigenvalues and corresponding eigenbases are used.The implementation retains low-frequency spectral components for each graph.
  • Spectral synchronization: SpecTN predicts a functional map C ∈ R^15×45 that maps each shape’s first 15 eigenbases into a canonical domain.The experiments therefore synchronize only the low-frequency end of each spectral domain.

5. Experiment

Experiments evaluate SyncSpecCNN on segmentation, keypoint, normal prediction, and ablation tasks. The framework generally outperforms baselines, benefits from spectral synchronization and multiscale kernels, and remains effective on varied or incomplete shapes.

  • Experimental setup: The evaluation covers shape segmentation, keypoint prediction, vertex normal prediction, and component ablations.The segmentation experiments include per-category, cross-category, and partial-data settings.
  • Part segmentation: On 16-category part segmentation, the full pipeline achieves the highest IoU on most categories and outperforms traditional and deep-learning baselines by a large margin.Ours1 omits SpecTN, whereas Ours2 is the full pipeline with SpecTN.
  • Cross-category segmentation: SpecTN improves generalization across shapes with diverse topology and geometry, including cross-category segmentation and classification.The cross-category setting removes the category label at test time and introduces greater geometric and topological variation.
  • Partial-data segmentation: On partial-shape segmentation, SyncSpecCNN outperforms ACNN by a large margin and experiences a smaller performance drop from complete to partial shapes.The partial-data experiments use simulated single-view scans.
  • Keypoint prediction: For keypoint prediction, SyncSpecCNN outperforms the previous method, with more precise predictions near the low Euclidean-error region.Performance is evaluated using five-fold cross-validation and PCK curves.
  • Normal prediction: Normal predictions are generally close to ground truth, including on thin structures, but boundaries become smoother than the sharper ground-truth normals.The paper attributes this regression limitation to using few eigenbases for high-frequency targets.
  • Ablation studies: Multiscale kernels outperform small- or large-scale kernels alone, while SpecTN improves robustness to sampling-density variation.The kernel ablation attributes improved segmentation performance to multiscale information aggregation.
  • Qualitative results: Large-scale context helps distinguish parts with similar local geometry across substantial shape variation, such as lamp bases and lampshades.The qualitative examples include wide benches versus ordinary chairs and pendant lamps versus table lamps.

6. Conclusion

SyncSpecCNN combines spectral dilated kernels for multiscale analysis within shape graphs with a spectral transformer for sharing parameters across different graphs. Experiments report state-of-the-art performance across semantic annotation tasks, including part segmentation and keypoint prediction.

  • Conclusion: SyncSpecCNN targets semantic annotation on 3D shape graphs using spectral dilated kernels and a spectral transformer network.The kernels support multiscale analysis within a shape, while the transformer synchronizes spectral domains across shapes.
  • Conclusion: The combined contributions achieve state-of-the-art performance on 3D shape part segmentation and 3D keypoint prediction.The paper reports validation through extensive experiments across semantic annotation tasks.
Loading 1612.00606v1…