Source-linked AI summary
CvxNet: Learnable Convex Decomposition
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, Andrea Tagliasacchi
TL;DR
Learning 3D geometry requires a representation that is compact, topology-agnostic, and usable by downstream mesh-based applications. CvxNet learns a low-dimensional family of convex decompositions through an auto-encoding pipeline, using differentiable hyperplane-based convex indicators. The method yields detailed reconstructions, outperforms competing part-based methods, and consistently outperforms OccNet on multi-view input, while relying on an approximate signed-distance formulation.
Problem
CvxNet addresses the need for a compact, learnable 3D representation that abstracts away topology during training while remaining usable as polygonal meshes at test time.
Method
An encoder predicts parameters for K convex primitives, whose differentiable indicators are decoded from unordered half-space constraints and trained by matching sampled occupancy values.
Results
CvxNet provides more detailed reconstructions than SIF, consistently outperforms OccNet on multi-view input, and is better than other evaluated part-decomposition methods.
Takeaways & Limitations
The learned convexes capture semantic structure and produce representations directly usable in graphics and physics pipelines.
Takeaways & Limitations
The signed-distance function is approximate because its gradient norm is not necessarily one everywhere.
Abstract
from arXiv · showhide
Any solid object can be decomposed into a collection of convex polytopes (in short, convexes). When a small number of convexes are used, such a decomposition can be thought of as a piece-wise approximation of the geometry. This decomposition is fundamental in computer graphics, where it provides one of the most common ways to approximate geometry, for example, in real-time physics simulation. A convex object also has the property of being simultaneously an explicit and implicit representation: one can interpret it explicitly as a mesh derived by computing the vertices of a convex hull, or implicitly as the collection of half-space constraints or support functions. Their implicit representation makes them particularly well suited for neural network training, as they abstract away from the topology of the geometry they need to represent. However, at testing time, convexes can also generate explicit representations -- polygonal meshes -- which can then be used in any downstream application. We introduce a network architecture to represent a low dimensional family of convexes. This family is automatically derived via an auto-encoding process. We investigate the applications of this architecture including automatic convex decomposition, image to 3D reconstruction, and part-based shape retrieval.
1. Introduction
CvxNet motivates convex decomposition as a compact, learnable representation that combines part-based structure with explicit and implicit geometric forms. It targets limitations of volumetric and surface representations while supporting downstream polygonal-mesh applications.
- Representation choices: Voxel representations face a curse of dimensionality, while higher-resolution hierarchical alternatives require complex implementations.Voxels show promise at low resolution, but hierarchical representations reduce memory at the cost of implementation complexity.
- CvxNet: CvxNet reconstructs 3D objects from images as collections of convex hulls and outputs polygonal meshes without computationally expensive iso-surfacing.The resulting meshes can be used in physics simulation and other polygonal-mesh applications.
- Representation choices: Different 3D representations trade off memory, topology handling, interpretability, collision efficiency, and semantic part structure.Surface, volumetric, primitive, and part-based models serve different needs in graphics, computer vision, and robotics.
- CvxNet: CvxNet approximates geometry with a small number of convex elements whose parameters are inferred from data without human supervision.Training predicts primitive configurations and parameters by matching reconstructed geometry to the target.
- CvxNet: The representation is locally supported and can associate corresponding convex elements with semantic parts across a shape collection.For example, the same element may represent chair backs across the chair dataset.
2. Related works
Prior 3D representations include voxels, meshes, point clouds, analytic implicit surfaces, and volumetric primitives, each with distinct computational or modeling limitations. Convex decomposition offers compact shape approximation, while CvxNet’s differentiable hyperplane-based formulation supports learning.
- Voxels: Voxel methods are memory-intensive beyond 64^3 resolution, while octrees extend resolution but introduce additional complexity.Octrees exploit voxel sparsity and can reach 512^3 resolution in the cited example.
- Surfaces: Mesh and point-cloud methods can require non-trivial post-processing or produce self-intersecting meshes.Meshes are standard in graphics, whereas point clouds are memory-efficient sensor representations but need further processing for final meshes.
- Primitives: Constructive Solid Geometry represents shapes through programs of Boolean operations on primitives, but this is generally difficult.Volumetric primitives include unit cubes in a lattice as a special case corresponding to voxel representations.
- Implicit surfaces: Implicit-surface methods learn occupancy or signed-distance functions, but their implicit descriptions have limited interpretability.The supplied passage identifies interpretability as a disadvantage but does not provide the remainder of the discussion.
- CvxNet: CvxNet maps unordered hyperplane parameters to a differentiable convex indicator using a LogSumExp smooth maximum.The smooth maximum lets gradients propagate through all hyperplanes while approximating convex signed distance functions.
- Convex decomposition: Convex decomposition approximates input shapes with a small number of convexes instead of producing exactly convex pieces that may be too numerous.The related-work discussion frames small approximate decompositions as more practical than exact convex splitting.
3. Method – CvxNet
CvxNet learns differentiable convex representations from sampled geometry, using auto-encoding to obtain low-dimensional convex families and compositions of convexes. Its representation supports both neural training through implicit indicators and explicit polygonal-mesh extraction.
- 3. Method – CvxNet: The model represents objects with an indicator function and predicts K convex primitive parameters from inputs such as images, point clouds, or voxel grids.Random samples are used to enforce agreement between the predicted and target indicator functions.
- 3.1. Differentiable convex indicator: A convex is defined by half-space constraints whose signed distances are combined with a smooth LogSumExp approximation and converted into a differentiable indicator.The smooth maximum and sigmoid make the representation suitable for gradient-based learning.
- 3.2. Convex encoder/decoder: An auto-encoder maps inputs to a low-dimensional latent code that decodes into hyperplane parameters, implicitly correlating hyperplanes with orientations.The learned latent space supports smooth interpolation between synthesized convex shapes.
- 3.3. Explicit interpretation: The implicit convex representation can be converted into polygonal meshes through duality transforms and two convex-hull computations without the curse of dimensionality.This provides an explicit mesh representation for downstream applications.
- 3.4. Multi-convex decomposition: The network composes K decoded convexes using shape codes and transformations, while outputting their shapes and poses rather than prescribing how a final image is generated.The composition uses the maximum of convex indicators to form a union.
- 3.5. Training losses: Auxiliary losses encourage parsimonious, uniquely parameterized, and trainable decompositions by discouraging overlap, removing null spaces, and preventing convex collapse.The unique-parameterization loss also centers convex bodies at the origin and helps reactivate inactive hyperplanes.
4. Experiments
Experiments evaluate CvxNet for abstraction, semantic correspondence, and depth- or RGB-based reconstruction on ShapeNet, with PartNet labels for semantic decomposition. CvxNet provides interpretable convex elements, dominates SIF in abstraction trade-offs, and compares favorably with reconstruction baselines.
- Experimental setup: ShapeNet experiments use the established voxelization, renderings, and data split, with multi-view depth from dodecahedron cameras and watertight meshes for inside/outside supervision.PartNet labels overlapping ShapeNet categories support quantitative semantic-decomposition evaluation.
- Experimental setup: CvxNet represents geometry with a small number of convex elements and is compared against self-supervised methods including VP, SIF, and OccNet.The comparison spans interpretable primitive-based approaches and implicit reconstruction methods.
- Abstraction: The learned convexes form natural correspondence across a shape collection, allowing component-wise semantic labeling and revealing semantic structure in PartNet.Each convex can be assigned its most common PartNet label for point-cloud segmentation.
- Abstraction: CvxNet’s abstraction curve dominates SIF’s across the tested numbers of primitives, indicating a stronger parsimony–reconstruction-accuracy trade-off.The comparison is visualized with Pareto-optimal curves in Figure 9.
- Reconstruction: CvxNet is consistently better than SIF, VP, and SQ for part decomposition, generally comparable to state-of-the-art reconstruction methods, and better than OccNet on F-score with multi-view depth.The reconstruction evaluation covers both multiple depth maps and a single RGB image.
- Reconstruction: The RGB-to-3D setup trains both CvxNet encoder and decoder from scratch, unlike SIF’s staged template and image-encoder training.This distinguishes the training protocols used in the comparison.
- Ablations: Ablations find relative insensitivity to bottleneck dimensionality, worse results with signed-distance supervision for Lapprox, and slightly worse performance when guidance and localization losses are merged.The studies also vary convex and hyperplane counts and verify that each loss benefits learning.
5. Conclusions
CvxNet offers a differentiable, learnable representation of convex primitives whose inferred shapes can be used directly in graphics and physics pipelines. The authors report more detailed abstraction than SIF and consistent multi-view reconstruction improvements over OccNet, while identifying several directions for future extension.
- CvxNet provides a differentiable representation of convex primitives that is amenable to learning.
- Its inferred representations are directly usable in graphics and physics pipelines.
- The self-supervised method produces more detailed reconstructions than SIF and consistently outperforms OccNet on multi-view input.
- Future work includes variable part counts, symmetry and hierarchy modeling, rotations, permutation-invariant encoders, and auto-decoder architectures.
Supplementary Material
Supplementary analyses examine input quality, loss simplification, and model complexity. They show that RGB inputs lose high-frequency details relative to depth, while merged losses simplify training with only a slight performance cost.
- Depth vs. Color: RGB-to-3D reconstruction loses many high-frequency surface details compared with depth-to-3D reconstruction, whereas low-frequency surfaces remain accurately approximated by both inputs.
- Original Losses vs. Merged Losses: Collapsing guidance and localization losses simplifies the training objective at the cost of slightly worse performance.
6. Union of smooth indicator functions
The method constructs smooth indicator functions for individual convexes and combines them into a union using a maximum operation compatible with the sigmoid formulation. Additional analyses examine class count and model complexity.
- Smooth convex indicators: The k-th convex is represented by a smooth indicator function derived from its signed distance function.
- Union construction: The union of convex indicator functions is formed using the maximum operator, reflecting the minimum-based union of signed distance functions.
- Union construction: Because monotonic functions preserve the max operation under the sigmoid, the maximum can be extracted from the sigmoid expression.
- Single-class vs. multi-class: Multi-class training improves depth-to-3D reconstruction across the benchmark, whereas single-class training benefits RGB-to-3D reconstruction in most cases.
- Model complexity: Model complexity is analyzed by varying the number of hyperplanes H and convexes K against mIoU, Chamfer-L1, and inference time.
7. Merged guidance loss and localization loss
The paper merges guidance and localization losses because both encourage convex elements to remain close to the ground truth, then evaluates the effects of losses, training targets, latent size, and interpolation.
- Merged guidance loss and localization loss: Guidance and localization losses share the objective of keeping convex elements near the ground-truth geometry.The paper proposes merging them into a single training loss because their motivations are different but their effects are consistent.
- Merged guidance loss and localization loss: Removing any individual loss term reduces overall performance on ShapeNet multi with RGB input.The ablation removes one loss term at a time in columns 2 through 5.
- Merged guidance loss and localization loss: Indicator-function training substantially outperforms signed-distance training, whose performance degenerates significantly.The comparison replaces O(x) with signed distance and removes the Sigmoid for signed-distance learning.
- Merged guidance loss and localization loss: Reconstruction performance remains relatively stable as the latent dimensionality varies.The latent-size ablation changes the number of latent dimensions used by the architecture.
- Merged guidance loss and localization loss: Linear interpolation between four encoded corner models generates intermediate shape results.The supplementary video provides denser interpolation between the corner examples.
- Merged guidance loss and localization loss: Each convex is trained to explain the N closest samples of O using weak gradients from the approximate Φ_k(x).This provides the mechanism for assigning local geometric responsibility during training.
8. Proof of auxiliary null-space loss
The auxiliary null-space loss rewrites hyperplane parameters in matrix form and has a unique minimizer that geometrically centers the convex body at the origin.
- Proof of auxiliary null-space loss: The hyperplane collection is represented by offsets d_h and a matrix N whose columns are the hyperplane normals n_h.This parameterization converts the minimization into a matrix expression involving the offsets and normals.
- Proof of auxiliary null-space loss: The auxiliary minimization has a unique solution computable explicitly, for example with the Moore-Penrose inverse of N.The passage states that uniqueness holds for the minimization over translations.
- Proof of auxiliary null-space loss: Geometrically, minimizing L_unique centers the convex body at the origin by minimizing the squared distances to its hyperplanes.For normalized normals, |H_h(x)| equals the distance to each hyperplane, so the sum measures squared distances to the polytope sides.
- Proof of auxiliary null-space loss: The t-SNE visualization compares CvxNet and OccNet latent spaces, where nearby same-class samples have similar geometric structure.The overlap between cabinets and speakers is meaningful because both exhibit cuboid geometry.