PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas Funkhouser, Silvio Savarese

arXiv:1803.08495v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Text-to-3D shape retrieval and generation lack a direct natural-language bridge without fine-grained labels. This paper learns joint text-shape embeddings and combines them with a conditional Wasserstein GAN, outperforming retrieval baselines and generating colored shapes.

  • Problem

    Text-to-3D shape embedding was previously unexplored, while existing text-image methods relied on costly, ill-defined fine-grained labels.

  • Method

    The paper jointly learns end-to-end instance-level text-shape embeddings from natural descriptions and uses them to condition a Wasserstein GAN for colored shape generation.

  • Results

    The learned embeddings outperform prior baselines for text-to-shape retrieval and support the proposed task of generating colored 3D shapes from text.

  • Takeaways & Limitations

    The work establishes a first step toward connecting natural language with realistic colored 3D objects exhibiting variation in color, texture, and shape detail.

  • Takeaways & Limitations

    The approach assumes each shape forms its own instance-level class with corresponding descriptions, and the authors characterize generation as only a first step.

Abstract

from arXiv · show

We present a method for generating colored 3D shapes from natural language. To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes. Our model combines and extends learning by association and metric learning approaches to learn implicit cross-modal connections, and produces a joint representation that captures the many-to-many relations between language and physical properties of 3D shapes such as color and shape. To evaluate our approach, we collect a large dataset of natural language descriptions for physical 3D objects in the ShapeNet dataset. With this learned joint embedding we demonstrate text-to-shape retrieval that outperforms baseline approaches. Using our embeddings with a novel conditional Wasserstein GAN framework, we generate colored 3D shapes from text. Our method is the first to connect natural language text with realistic 3D objects exhibiting rich variations in color, texture, and shape detail. See video at https://youtu.be/zraPvRdl13Q

1 Introduction

The paper introduces joint text–3D-shape embeddings for text-to-shape retrieval and colored shape generation. It combines instance-level association and metric learning, evaluates on two new datasets, and uses a conditional Wasserstein GAN for generation.

  • Introduction: The work targets text-to-shape retrieval and generation, enabling a described colored 3D object to be retrieved or generated.The motivating example is a round glass coffee table with four wooden legs.
  • Introduction: The paper addresses the absence of prior text-to-3D-shape embedding by learning a joint representation directly from natural-language descriptions of 3D instances.Unlike related text-to-image methods, it does not require fine-grained category labels or pre-training on large datasets.
  • Introduction: The method combines learning by association with metric learning to connect similar descriptions and shape instances while strengthening within-modality similarities.It uses only instance-level text–shape correspondences and avoids expensive fine-grained category or attribute annotations.
  • Introduction: The learned embedding supports text-to-shape retrieval and the proposed colored shape generation task, addressed with a novel Conditional Wasserstein GAN.Colored generation is emphasized because shape descriptions commonly involve color or material properties.
  • Introduction: 75K natural-language descriptions cover 15K chair and table shapes in ShapeNet, alongside a procedurally generated colored-primitives dataset with synthetic captions.Experiments on both datasets reportedly outperform baselines by a large margin for retrieval and generation.

2 Related Work

The paper builds on prior work in multimodal representation learning, cross-modal metric learning, text-to-image synthesis, and 3D voxel generation. Its framework combines these directions for colored voxel generation from freeform text.

  • Visual description representations: Prior visual-description methods learn text embeddings for image synthesis but rely on pre-training and expensive, fine-grained category labels.
  • Multimodal representations: Multimodal representation methods learn joint audio-video, image-text, or multimodal generative representations, while CCA-based approaches have scalability limitations.
  • Metric learning: The approach extends Song et al.’s method and relates to cross-modal metric learning using contrastive, triplet, and N-pair losses.
  • Generative models and GANs: Earlier GAN research addressed text-to-image synthesis by separately training text embeddings before generating images.
  • 3D voxel generation: Existing 3D voxel methods use voxel-wise cross entropy or GANs, whereas this framework generates colored voxels from freeform text with a Wasserstein GAN.

3 Datasets

The paper introduces two datasets for learning joint representations between freeform text and 3D shapes: human-designed ShapeNet CAD objects with natural-language descriptions and procedurally generated geometric primitives. Each example pairs a text description with a colored voxel-grid representation of a shape.

  • Dataset overview: The datasets support training and evaluation of joint representations for freeform text descriptions and 3D CAD shapes.They include ShapeNet objects augmented with natural-language descriptions and a controlled dataset of procedurally generated 3D geometric primitives.
  • Data representation: Each data example is a text description–3D shape voxel-grid pair (x, s), with x represented as an arbitrary-length vector of word indices.The shape tensor uses v for the maximum voxel dimension and c for the number of channels per voxel.
  • ShapeNet objects and natural language descriptions: The realistic ShapeNet dataset contains 8,447 table instances and 6,591 chair instances created by human designers to represent real objects.These categories provide 3D CAD shapes paired with natural-language descriptions.

4 Joint Text–3D Shape Representation Learning

The method learns a joint text–shape embedding that clusters similar items, links descriptions to associated shapes, and separates dissimilar cross-modal pairs. It combines cross-modal association learning, instance-level supervision, metric constraints, and norm regularization.

  • Learning objectives: The joint embedding is designed to cluster similar texts and shapes, keep descriptions near associated shapes, and separate dissimilar text–shape pairs.These objectives motivate the association and metric-learning components.
  • Cross-modal association: Text and shape encoders produce embeddings whose dot-product similarities define softmax cross-modal association probabilities.The similarity matrix is Mij = Ti · Sj, followed by softmax normalization.
  • Cross-modal association: Text–shape–text round trips are trained toward uniform associations among similar descriptions, while an entropy loss encourages each shape to associate with all matching descriptions.The approach also uses shape–text–shape round trips to provide additional supervision.
  • Instance-level supervision: Instance-level classes assign each shape its corresponding descriptions, avoiding expensive and ambiguous fine-grained category labels.This relaxed supervision does not explicitly separate descriptions and shapes without correspondence labels.
  • Metric learning: Metric learning pushes positive same-class pairs above negative different-class pairs by margin α and extends these constraints across text and shape embeddings.The authors report that combining metric and association learning improves cross-modal associations over either alone.
  • Optimization: The final training objective combines association and metric-learning losses, while thresholded L2 regularization prevents embedding degeneracy without enforcing unit norms.Norm regularization is applied only when embedding vectors exceed a threshold.

5 Generating Colored 3D Shapes from Text

The paper applies joint text–shape representations to colored 3D shape generation with a conditional GAN comprising text encoding, generation, and criticism. Its conditional Wasserstein GAN formulation uses matching and mismatching text–shape pairs to encourage realistic, attribute-consistent, and diverse outputs.

  • Model architecture: The generation model combines a text encoder, generator, and critic: text and noise are mapped into latent representations that condition generated shapes, while the critic evaluates realism and text–shape consistency.The text encoder maps descriptions to latent vectors, which are concatenated with noise before generation.
  • Model motivation: GANs support one-to-many generation because they encourage specified attributes without penalizing valid shapes for differing from one particular model instance.Per-voxel constraints would incorrectly force generated shapes to match a specific instance.
  • Conditional Wasserstein GAN: The formulation is based on Wasserstein GANs to improve output diversity and avoid mode collapse, and presents the first conditional Wasserstein GAN to the authors’ knowledge.Unlike a traditional conditional GAN, the method samples matching and mismatching text–shape pairs alongside text for the generator.
  • Training objective: The gradient penalty samples from real matching pairs with probability 0.5 or generated fake voxelizations, while text embeddings are concatenated with randomly sampled noise vectors.The critic is D, the generator is G, and pmat and pmis denote matching and mismatching text–shape pairs.

6 Experiments

Experiments evaluate the learned joint representation through text-to-shape retrieval, ShapeNet generation, and embedding visualizations. The model outperforms retrieval and generation baselines while enabling text-controlled manipulation of shape attributes.

  • Text-to-shape retrieval: The full model achieves the best text-to-shape retrieval performance on the primitives dataset, outperforming DS-SJE, LBA-TST, LBA-MM, ML, and Full-TST.Retrieval is measured with normalized discounted cumulative gain (NDCG) and recall rate (RR@k).
  • Text-to-shape generation: Our CGAN outperforms GAN-INT-CLS on all generation metrics, while CWGAN further improves every metric except class accuracy.The evaluation uses occupancy IoU, inception score, color Earth Mover’s Distance, and color/occupancy classification accuracy.
  • Text-to-shape generation: CWGAN generates the most realistic shapes, more natural color distributions, and better text conditioning than the baselines, including a chair with red seat and backrest padding.GAN-INT-CLS struggles with color and occupancy distributions, while CGAN remains affected by structural and color artifacts.
  • Shape manipulation using text: Changing text attributes often changes the corresponding generated properties: “white” alters color, while “rectangular” alters shape relative to “wooden” and “round.”Each output is independently generated, so consistency between outputs is not enforced.
  • Embedding analysis: Embedding dimensions correlate with category, physical shape, and color, and vector arithmetic lets the generator produce shapes with manipulated desired properties.Visualized dimensions distinguish chairs from tables, rectangular from round tables, and white, brown, and red colors.

7 Conclusion · Appendix

The paper presents an end-to-end method for learning joint text–3D-shape embeddings from instance-level descriptions, enabling cross-modal retrieval and text-to-shape generation with a conditional Wasserstein GAN. The authors characterize text-to-shape generation as challenging and their approach as a first step.

  • 7 Conclusion: The method learns joint embeddings of text and 3D shapes end-to-end using only instance-level natural-language descriptions.The embeddings connect the text and shape modalities within a shared representation.
  • 7 Conclusion: The learned embeddings enable retrieval between text and shape modalities, outperforming approaches from prior work.The conclusion identifies cross-modal retrieval as a demonstrated capability.
  • 7 Conclusion: The authors combine the learned embedding with a conditional Wasserstein GAN formulation for text-to-shape generation.This combination addresses generation of 3D shapes conditioned on natural-language text.
  • 7 Conclusion: Text-to-shape generation is described as a challenging problem, and the presented approach is characterized as only a first step.The conclusion explicitly frames the method as an initial contribution to this task.
  • Appendix: The work acknowledges support from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1147470.The acknowledgment states that the material is based upon this support.
  • Appendix: The work also acknowledges support from Google, Intel, and the Technical University of Munich–Institute for Advanced Study.The supplied acknowledgment passage lists these additional supporters.

A Model Details · B Dataset Details

The method combines CNN/RNN text encoding, 3D convolutional shape encoding, and conditional GAN components to learn joint text–shape representations and generate voxelized shapes. Evaluation uses paired text descriptions and colored 3D voxel grids from both procedural primitives and ShapeNet CAD objects.

  • A Model Details: The text encoder uses trainable word vectors with a CNN/RNN architecture containing a 256-unit GRU, ReLU activations, and L2 regularization weighted 0.0005.All layers except the last use ReLU operators.
  • A Model Details: The shape encoder is a standard 3D-CNN, while the generator uses fractionally-strided convolutions for upsampling and a final sigmoid activation.ReLU follows every layer except the last in both architectures.
  • A Model Details: The critic applies leaky ReLUs with leak rate 0.2, concatenates text embeddings from two 256-dimensional fully connected layers with conv5 features, and omits batch normalization for Wasserstein GANs.Non-Wasserstein implementations use a sigmoid activation in the final critic layer.
  • A Model Details: Joint representation training combines learning by association and metric learning, using batches of 100 unique shapes with two matching captions per shape and a norm-threshold loss at 10.The additional loss applies when the embedding norm exceeds the threshold.
  • A Model Details: GAN training uses a learning rate of 5 × 10−5, decay rate 0.95 every 10000 steps, an initial 100-to-1 critic-to-generator schedule, and then 5 critic steps per generator iteration.The gradient penalty coefficient is 10.
  • B Dataset Details: The model represents each object as a colored RGB 3D voxel grid paired with a natural-language description, forming training examples (x, s).The paper uses text description–3D voxel shape pairs as training data.
  • B Dataset Details: Two paired datasets evaluate the method: a controlled procedurally generated collection of colored geometric primitives and a real ShapeNet CAD-object collection augmented with human-provided descriptions.Both datasets pair text descriptions with 3D voxel shapes.

B.1 Primitive Shapes Dataset · B.2 ShapeNetCore Descriptions and Voxelizations · B.3 Preprocessing of Natural Language Descriptions

The paper constructs synthetic primitive data and a large ShapeNetCore dataset pairing realistic colored 3D objects with freeform descriptions, then voxelizes and preprocesses these multimodal inputs for learning.

  • B.1 Primitive Shapes Dataset: B.1 uses six primitive types, 14 colors, and nine sizes with random perturbations to create 7,560 voxelized shapes.The perturbations generate 10 samples from each of 756 primitive configurations.
  • B.1 Primitive Shapes Dataset: B.1 generates synthetic descriptions with 10 templates that vary shape, size, and color attributes and their ordering.Example templates place attributes before the shape or describe attributes after naming the shape.
  • B.2 ShapeNetCore Descriptions and Voxelizations: B.2 uses 8,447 tables and 6,591 chairs from ShapeNetCore to study detailed intra-category variation in realistic designer-made CAD models.The dataset focuses on table and chair categories because they contain many object instances suitable for collecting descriptions.
  • B.2 ShapeNetCore Descriptions and Voxelizations: B.2 augments ShapeNetCore with 75,344 natural-language descriptions, providing five crowdworker descriptions per CAD model.Descriptions were collected through Amazon Mechanical Turk.
  • B.2 ShapeNetCore Descriptions and Voxelizations: B.2 collects freeform appearance descriptions from workers viewing slowly rotating rendered objects, after finding manipulable full 3D views slower to use.Workers were prompted to describe color, shape, material, and physical appearance without restrictions on length or sentence count.
  • B.2 ShapeNetCore Descriptions and Voxelizations: B.2’s sample shows that 95% of descriptions identify the object type, 86% describe parts and their fit, 83% mention color, and 63% mention material.Additionally, 47% describe the geometric shape of some part.
  • B.2 ShapeNetCore Descriptions and Voxelizations: B.2 computes surface and solid color voxelizations using hybrid surface sampling and views, voxelizing at 256^3 before low-pass-filtered downsampling.The hybrid method targets surfaces in concavities, while filtering reduces artifacts from high-frequency texture colors.
  • B.3 Preprocessing of Natural Language Descriptions: B.3 lowercases, tokenizes, lemmatizes, and spell-corrects descriptions, discarding only 16 of 75,360 entries exceeding 96 tokens.Terms occurring ≤2 times are mapped to the UNK token.

C Additional Retrieval Results

The additional retrieval experiments assess whether the learned joint embedding supports both within-modality and cross-modality retrieval. They report quantitative results across all tasks and qualitative analyses for each retrieval type.

  • Retrieval Tasks: The embedding supports within-modality retrieval by querying text or shapes and retrieving nearest neighbors in the same modality.The within-modality tasks are text-to-text and shape-to-shape retrieval.
  • Retrieval Tasks: The embedding also supports cross-modality retrieval through text-to-shape and shape-to-text tasks.These tasks query one modality and retrieve instances in the other.
  • Results Organization: The appendix presents quantitative results for all retrieval tasks, followed by qualitative results for text-to-text, shape-to-shape, and cross-modal retrieval.The additional experiments are organized across Appendices C.1–C.4.

C.1 Quantitative retrieval results … D Additional Generation Results

The model achieves the strongest quantitative retrieval performance through jointly trained association and metric learning, with additional round trips improving ShapeNet retrieval. Qualitative results show semantically and visually attribute-matched retrievals, superior cross-modal results over baselines, and further generation and vector-arithmetic analyses.

  • C.1 Quantitative retrieval results: Jointly trained association and metric learning produces the best primitives-dataset retrieval, outperforming baselines across text-to-text, shape-to-shape, and text-to-shape tasks.Association learning or metric learning alone underperforms, while their joint training generalizes better to unseen configurations.
  • C.1 Quantitative retrieval results: On ShapeNet, retrieval is counted correct only when the retrieved text or shape belongs to the same object instance as the query.Semantically similar descriptions from different models are marked incorrect under this instance-level evaluation.
  • C.1 Quantitative retrieval results: Additional round trips significantly improve text-to-text retrieval, while Full-TST slightly outperforms Full-MM on shape-to-text retrieval and Full-MM leads text-to-shape retrieval.The two full models remain comparable on shape-to-text retrieval.
  • C.2 Text-to-text Retrieval: Text-to-text retrieval returns descriptions from different shapes that remain semantically similar, including correspondences such as “oval” and “circular.”Retrieved examples also preserve attributes such as red chairs and glass coffee tables.
  • C.3 Shape-to-shape Retrieval: Shape-to-shape retrieval commonly finds neighbors matching the query’s style, shape, material, color, and attributes such as table geometry.For example, brown armless chairs retrieve wooden armless chairs, sofas retrieve sofas, and tables retrieve similarly shaped tables.
  • C.3 Shape-to-shape Retrieval: Compared with LBA (TST) and DS-SJE, the model retrieves more reasonable shape neighbors, whereas the baselines may return tables for chairs or chairs with unrelated attributes.The model successfully retrieves chairs with similar material, color, and style.
  • C.4 Cross-modal Retrieval: For text-to-shape retrieval, the model matches category, color, and shape, while DS-SJE often returns the same category-level results without capturing described attributes.The additional comparison identifies the model as the only method producing reasonable text-to-shape retrieval results.

D.1 GAN generation results · D.2 Additional Vector Arithmetic Results

CWGAN generates the most realistic, diverse, and text-consistent colored shapes, outperforming the baselines while still missing some fine-grained geometric details. Additional vector arithmetic shows that the learned embeddings support attribute composition and transfer for controlling novel shape generation.

  • D.1 GAN generation results: GAN-INT-CLS struggles to condition on text, producing incorrect colors for listed glass or white tables and highly similar chairs and tables.The passage attributes these poor generations primarily to the quality of the learned text embeddings.
  • D.1 GAN generation results: The CGAN performs better than GAN-INT-CLS, usually generating models with the correct category and color when conditioned on text.CGAN still exhibits mode collapse, including groups of highly similar brown tables and chairs.
  • D.1 GAN generation results: CWGAN produces the most realistic generations, matching ground-truth color and category while providing substantially greater diversity than the other approaches.In one example, CWGAN alone generates a white table when other methods produce a chair or incorrectly colored table.
  • D.1 GAN generation results: Although the model significantly outperforms the baselines, it misses fine-grained geometric details such as table shape, leg count, and desk form.The authors identify these description details as difficult for the model to capture and note substantial room for improvement.
  • D.2 Additional Vector Arithmetic Results: Vector arithmetic on text and shape embeddings enables the trained generator to produce novel shapes with specific attributes.The additional results use simple operations on learned representations to control generated shapes.
  • D.2 Additional Vector Arithmetic Results: The joint embeddings support composition and transfer of attributes, allowing shape generation to be controlled through vector arithmetic.This capability is illustrated in the additional vector arithmetic results.
Loading 1803.08495v1…