PaperlayerView exact revision on arXiv

Source-linked AI summary

I2CD: Direct Image-to-Convex Decomposition for Simulation-Ready Collision Geometry

Qian Wang, Liam Merz Hoffmeister, Brian Scassellati, Daniel Rakita

arXiv:2610.03453v1cs.ROcs.CV

TL;DR

Image-to-3D models produce dense visual meshes, while simulation and planning require convex collision geometry and current conversion pipelines are slow and brittle. I2CD freezes a pretrained image-to-3D model and trains a lightweight head to predict convex decompositions directly from RGB images. It reports higher volumetric IoU at lower latency than eight pipelines, along with faster planner-ready scene geometry and comparable pick-and-place success.

  • Problem

    Image-to-3D generators produce dense meshes with topological defects, while obtaining collision geometry through repair and approximate decomposition is a slow, fragile process.

    Source: §I · p. 1

  • Method

    I2CD freezes a pretrained image-to-3D backbone and trains a lightweight head to read its shape features into a union of convex polytopes.

    Source: §I · p. 1

  • Results

    I2CD achieves the best volumetric IoU among eight reconstruct-then-decompose pipelines on 227 held-out objects at 6–37× lower end-to-end latency, and reduces xArm7 planner-ready scene time from 328 s to 11 s with comparable success.

    Source: §I · p. 2

  • Takeaways & Limitations

    I2CD geometry is used as delivered in four tested physics engines, and the approach cuts planner-ready cluttered-scene time on a physical xArm7 at comparable execution success.

    Source: §V · p. 7

    I2CD inherits frozen-backbone errors such as missed thin structures and hallucinated or over-smoothed parts; its surface fidelity trails the best pipeline by 3.6 F-score points, a limitation most likely to bind in contact-rich tasks such as nonprehensile pushing.

    Source: §V · p. 7

Abstract

from arXiv · show

Physics simulators and motion planners require convex collision geometry, yet image-to-3D generative models output dense, frequently non-manifold visual meshes. Bridging the two today takes a slow, brittle reconstruct-then-decompose pipeline of repair, decimation, and approximate convex decomposition. We present I2CD, which predicts a convex decomposition directly from a single RGB image. Rather than train a new image-to-3D model, I2CD freezes the pretrained Hunyuan3D-2 image-conditioned diffusion transformer and shape decoder and trains only a lightweight cross-attention head (38M parameters, under ten GPU-hours) whose learned "convex-slot" tokens emit the halfplane parameters of $K$ convex polytopes. The output is compact, convex by construction, and loads into physics engines without any post-processing, in ${\sim}0.5$s per image. On $227$ held-out OmniObject3D and Google Scanned Objects instances, I2CD attains the highest volumetric IoU among eight reconstruct-then-decompose pipelines while running $6$-$37\times$ faster end-to-end. In a cross-simulator study in MuJoCo, PyBullet, Genesis, and Isaac Sim, every engine uses I2CD geometry as delivered, whereas raw generated meshes "load" everywhere but are silently replaced by a different collision shape in most cases or need seconds to minutes of per-object preprocessing. On a physical xArm7, I2CD produces planner-ready geometry for a $20$-object cluttered scene in $11$s versus $328$s for the strongest baseline, at comparable pick-and-place execution success ($85$ vs. $90$ of $100$ trials).

I. INTRODUCTION

I2CD replaces the fragile reconstruct-then-decompose route from visual meshes with direct convex geometry prediction from one RGB image. It adapts a frozen image-to-3D foundation model and reports stronger volumetric IoU, lower latency, and comparable robot execution success against the supplied baselines.

I. INTRODUCTION

  • Image-to-3D generators optimize visual fidelity and emit dense meshes with topological defects, making collision geometry dependent on a slow, error-sensitive repair and decomposition chain.Skipping that chain can leave a silently substituted collision shape.Source: §I · p. 1
  • I2CD freezes Hunyuan3D-2’s image-conditioned transformer and ShapeVAE decoder, training a lightweight cross-attention head whose convex-slot tokens emit halfplanes for convex polytopes.The 38 M-parameter head is trained in under ten GPU-hours; I2CD reuses the pretrained shape prior rather than introducing a new generator or convex representation.Source: §I · pp. 1–2
  • I2CD achieves the best volumetric IoU among eight reconstruct-then-decompose pipelines on 227 held-out objects at 6–37× lower end-to-end latency.The benchmark pairs Hunyuan3D-2 and SAM3D with CoACD, V-HACD, CvxNet, and BSP-Net.Source: §I · p. 2
  • MuJoCo, PyBullet, Genesis, and Isaac Sim use I2CD geometry as delivered, whereas raw generated meshes are silently replaced or require costly preprocessing.The study tests simulation readiness beyond whether a mesh merely loads.Source: §I · p. 2
  • On a 20-object xArm7 scene, I2CD cuts planner-ready geometry time from 328 s to 11 s with comparable success: 85 versus 90 of 100 trials.The comparison reports physical pick-and-place execution trials.Source: §I · p. 2

II. RELATED WORK

Prior image-to-3D robotics pipelines use generated meshes followed by post-processing, while learned convex methods have limited open-world applicability. I2CD reads pretrained shape features into convex geometry and is positioned as complementary to scene-level optimization.

II. RELATED WORK

  • Image-to-3D models used for robotics have routed their mesh outputs through post-processing, despite pretrained shape decoders encoding a general-purpose geometric prior.I2CD uses a small head to read that prior into a different output modality.Source: §II · p. 2
  • Approximate convex decomposition works well on watertight, well-conditioned meshes but degrades on noisy non-manifold outputs and takes seconds to tens of seconds per clean object.Common tools include V-HACD and CoACD.Source: §II · p. 2
  • CvxNet and BSP-Net offer single-view RGB modes but are category-specific ShapeNet models without open-world weights, while other learned approaches use different representations or inputs.The passage also describes RGB-D fitting and per-point feature-field supervision approaches.Source: §II · p. 2
  • I2CD’s frozen DiT and decoder features feed a cross-attention head whose learned convex-slot tokens emit halfplane parameters for convex parts.The predicted occupancy is their union, and per-convex meshes are extracted for simulator use.Source: Figure 2 · p. 2
  • I2CD is an object-level, feed-forward source of convex geometry, complementary to SPARCS’s scene-level optimization of object shapes and poses.I2CD outputs could initialize SPARCS, whose physics-aware refinement could correct pose-registration errors that dominate the reported robot failures.Source: §II · p. 2

III. METHOD

I2CD leaves image-to-shape reasoning to a pretrained generator and learns a readout that predicts a union of convex polytopes. Its representation uses halfplane intersections and a differentiable occupancy relaxation, with exported parts bounded by construction.

III. METHOD

  • A frozen diffusion transformer samples a shape latent from the image, a frozen decoder expands it into shape features, and a small trained head turns them into convexes.The head’s learned readout maps the pretrained features to the output geometry.Source: §III · p. 2

A. Convex Shape Representation

  • Each convex part is an intersection of H halfplanes, and the object is represented by the union of K such polytopes.A point lies in a part when all its signed halfplane distances are nonpositive.Source: A. Convex Shape Representation · pp. 2–3
  • LogSumExp soft intersection, a sigmoid indicator, and hard-max union relax the nondifferentiable membership test for learning.With α = 75 and H = 12, points 0.1 outside receive occupancy at most about 5×10^-4; points 0.1 inside receive at least about 0.98 at βk = 50.Source: A. Convex Shape Representation · p. 3
  • Each part is convex and non-empty, but boundedness is not guaranteed by the parameterization, so export clips every convex to the cube [−0.6, 0.6]^3.An exact recession-cone test found zero unbounded parts among 2,724 benchmark parts, and the clipping was never active empirically.Source: A. Convex Shape Representation · p. 3

B. Pretrained Image-to-Shape Backbone

I2CD uses Hunyuan3D-2-mini as a fixed, frozen backbone and replaces its SDF readout with a convex-geometry head.

B. Pretrained Image-to-Shape Backbone

  • The fixed Hunyuan3D-2-mini backbone comprises a frozen image-conditioned flow-matching DiT and a frozen ShapeVAE decoder.The DiT samples a 512-token shape latent; the decoder maps it to per-token shape features.Source: B. Pretrained Image-to-Shape Backbone · p. 3
  • The method discards Hunyuan’s SDF readout and attaches its head to the decoder’s shape features.The decoder is a linear projection followed by a 16-layer self-attention transformer of width 1024.Source: B. Pretrained Image-to-Shape Backbone · p. 3
  • Because both backbone components are frozen, training does not update the billion-parameter model or run the DiT; training latents are encoded once and cached.The frozen encoder uses 5,120 random and 5,120 sharp-edge surface samples per training mesh.Source: B. Pretrained Image-to-Shape Backbone · p. 3

C. Convex Head

The convex head uses learned slots to predict all convex-part parameters from frozen-model features in one pass. Training combines occupancy supervision with a containment-based contrastive loss to shape the part assignments.

C. Convex Head

  • Each learned convex-slot token queries the features, and a cross-attention head predicts all K convex parameter tuples in one pass.Residual cross-attention blocks let slots query the features; self-attention coordinates part assignments, and the readout predicts translations, normals, offsets, and sharpness.Source: C. Convex Head · p. 3
  • At K=H=12, the head has approximately 38 million parameters, a small fraction of the frozen backbone.The reported readout maps each slot to a translation, normals, offsets, and sharpness, with parameter ranges enforced by construction.Source: C. Convex Head · p. 3

D. Training

  • Training uses cached latents from centered, unit-scaled artist-created meshes, which fit within [-0.5, 0.5]^3 with a 0.1 export-cube margin.The training meshes are from MolmoSpaces.Source: D. Training · p. 3
  • Class-balanced BCE addresses the roughly 6% occupied training queries; without balancing, the model collapses to predicting empty space.Each mesh supplies 4,096 precomputed query points, half near the surface and half uniformly sampled in the export cube; 2,048 are drawn per step.Source: D. Training · p. 3
  • A convex-pair contrastive loss counters slab-stacking by assigning shared convex responsibility only to point pairs whose segment stays inside the shape.The method uses surface anchors, inside-segment positives, and outside-segment negatives, and applies the loss directly to per-convex responsibilities without an auxiliary feature field.Source: D. Training · p. 3
  • The objective combines occupancy BCE with 0.1-weighted contrastive loss; the reported checkpoint is the validation-best model at epoch 210 after about 9 GPU-hours.Training uses AdamW; the loss weights α=75 and τ=0.1 follow prior work and are not tuned.Source: D. Training · p. 3

E. Inference and Export

At test time, I2CD processes a single image and exports convex parts for simulation. The paper evaluates geometric fidelity, simulator compatibility, physical pick-and-place, and ablations.

E. Inference and Export

  • A single image is processed end-to-end in approximately 0.5 seconds on one RTX 5000 Ada GPU.The pipeline includes background removal, 10-step DiT sampling, the frozen ShapeVAE decoder, and the convex head.Source: E. Inference and Export · pp. 3–4
  • Each predicted convex is meshed independently, and parts with maximum occupancy below 0.5 are dropped before export.Only 13 of 2,724 predicted parts were dropped on the benchmark, averaging 11.94 parts per object.Source: E. Inference and Export · p. 4
  • Because exported parts are convex, MuJoCo, PyBullet, Genesis, and Isaac Lab accept the geometry as-is without repair, decimation, or decomposition.The paper evaluates geometric fidelity, simulation-readiness across four engines, physical xArm7 pick-and-place, and ablations.Source: E. Inference and Export · p. 4, §IV · p. 4

A. Geometric Fidelity

I2CD is evaluated against reconstruct-then-decompose pipelines on 227 held-out OmniObject3D and GSO objects. It leads in pooled volumetric IoU and runtime, with dataset-specific results qualified by the different reference geometries.

A. Geometric Fidelity

  • The evaluation uses 127 OmniObject3D scans and 100 watertight GSO objects, with the rendered 0th view as input and original meshes as ground truth.Both test sets are disjoint from the 121,056 artist-created MolmoSpaces training meshes.Source: A. Geometric Fidelity · p. 4
  • 98% of 127 HY3D meshes contain non-manifold edges, only 2% pass watertightness, and 59% contain multiple components.These meshes have a median of 1.6 million faces; the extra components are fragments with less than 0.1% of vertices.Source: A. Geometric Fidelity · p. 4
  • 65.7% pooled IoU, versus 62.8% for HY3D+CoACD, is the highest result; I2CD runs in 0.51 s, 6–37× faster than complete pipelines.Table I covers 227 objects and reports mean ± standard deviation, with IoU split by dataset; Fig. 3 gives a qualitative comparison and rounded mean end-to-end times.Source: A. Geometric Fidelity · p. 4, Table I · p. 4, Figure 3 · p. 5
  • On watertight GSO objects, I2CD scores 68.9% IoU versus at most 63.8% for pipelines; on OmniObject3D, it ties the best pipeline at 63.1% versus 62.9%.GSO IoU uses the exact mesh reference, while open OmniObject3D scans use a convex-hull reference that rewards inflation and hides concavity errors.Source: A. Geometric Fidelity · p. 4
  • On OmniObject3D, I2CD has fewer severe failures: IoU below 40% occurs on 5% of objects, versus 14–17% for CoACD/V-HACD pipelines.Those pipelines beat I2CD on 55–65% of objects by a median of 3–4 points, so the reported advantage is robustness rather than per-object dominance.Source: A. Geometric Fidelity · p. 4
  • Across four DiT sampling seeds, OmniObject3D IoU is 62.1 ± 0.7, with a median per-object spread of 1.7 points.Surface metrics favor SAM3D+CoACD, which has the best F-score and Chamfer distance but 20.2% IoU variance; I2CD trails its F-score by 3.6 points.Source: A. Geometric Fidelity · p. 4

B. Simulation-Readiness Across Physics Engines

Across four physics engines, I2CD geometry is used as delivered, while raw meshes may be silently changed or require costly preprocessing. The study measures whether collision geometry is preserved, rather than whether it matches the object.

Raw meshes and default handling

  • About 60% of raw-mesh objects receive the wrong collision shape under default handling, as MuJoCo, PyBullet, and PhysX replace dynamic meshes with a single convex hull.That proxy is also the auto-hull geometry that fails 95 of 100 trials in the robot study.Source: B. Simulation-Readiness Across Physics… · p. 5, Table II · p. 5

Built-in decomposition

I2CD geometry

Non-convex colliders

  • PyBullet’s exact concave mode registers zero dynamic object–object contacts, while PhysX SDF cooking takes 3.75 s per object, crashes on six, and leaves 14% slowly rolling.Genesis’s SDF collider is omitted because a 20-object pilot exceeded 180 GB RAM or 15 minutes on 13 objects.Source: Table II · p. 5, B. Simulation-Readiness Across Physics… · p. 6

C. Pick-and-Place on a Physical xArm7 in Clutter

The xArm7 study tests whether geometry pipelines support pick-and-place through a 20-object cluttered scene. I2CD sharply reduces scene preparation time while achieving execution success comparable to SAM3D+CoACD.

Setup and protocol

  • Each of 20 objects serves as a target against 19 obstacles; each target is tested with five planner seeds and a four-phase plan under a 5 s timeout per phase.A trial requires all phases and collision-free physical execution to succeed; geometry is the only changed baseline component.Source: C. Pick-and-Place on a Physical xArm7… · p. 6, Table III · p. 6

Results

D. Ablations

The ablations examine objective design, model capacity, head depth, meshing resolution, and the train–test latent gap. They identify objective and capacity trade-offs while showing that sampling noise and backbone latents qualify smaller performance differences.

Objective

  • +1.6 IoU, +1.1 F, and −0.5 mm CD: the convex-pair contrastive term improves all three metrics and replaces thin slabs with spatially coherent parts.Removing class balancing drops IoU by 7.5 points, while MSE fails as its gradient vanishes in saturated sigmoid tails.Source: D. Ablations · p. 6

Capacity

  • 58.6–65.5 IoU: capacity sweeps show fewer, richer parts help, while K=24 can split thin objects into slivers.K=6, H=12 is best, but K=H=12 remains the default to match the baselines’ ≤12-hull budget.Source: D. Ablations · p. 6, Table V · p. 7

Head depth

  • A minimal head with one cross-attention layer and no self-attention is within 0.6 IoU of the default; differences below about one point match measured seed noise.The two K=H=12 runs score 63.1 and 62.0, and head depth is not the performance driver within that resolution.Source: D. Ablations · p. 7

Marching-cubes resolution

  • Changing marching-cubes resolution from R=32 to R=128 shifts IoU by less than 0.15 points, while meshing time rises from 4.4 to 222 ms per object.The residual surface-metric gap to SAM3D+CoACD is attributed to predicted geometry, not export.Source: D. Ablations · p. 7

Train–test latent gap

  • 68.9% IoU with ground-truth-mesh latents versus 65.7% with DiT samples yields a 3.2-point train–test gap attributed to image-conditioned sampling.The passage states this gap will shrink with stronger backbones.Source: D. Ablations · p. 7

V. CONCLUSION AND LIMITATIONS

I2CD adapts a frozen image-to-3D model to emit convex collision geometry directly, matching or exceeding eight reconstruct-then-decompose pipelines at lower latency. The conclusion also identifies limits in surface fidelity, inherited backbone errors, and the scope of the real-robot study.

Conclusion

  • The qualitative objective ablation shows I2CD producing compact, coherent parts, while class-balanced BCE alone tiles interiors with thin slabs and unbalanced BCE under-fills shapes.Source: Figure 6 · p. 7
  • I2CD matches or exceeds eight reconstruct-then-decompose pipelines in volumetric IoU at 6–37× lower latency, with geometry used as delivered across four physics engines.The 38 M-parameter head trains in under ten GPU-hours; the geometry settles stably in MuJoCo, PyBullet, Genesis, and Isaac Sim without preprocessing.Source: §V · p. 7

Limitations

  • I2CD inherits frozen-backbone errors, and its surface fidelity trails the best pipeline by 3.6 F-score points; this is most likely to matter in contact-rich tasks.The limitation passage identifies missed thin structures and hallucinated or over-smoothed parts; for quasi-static planning, the surface-fidelity gap matters little.Source: §V · p. 7
  • Convex slots group geometry by proximity rather than semantics, and the real-robot study covers one tabletop scene and a quasi-static task.Complex support relations and contact-rich tasks are identified as settings for more stringent future testing.Source: §V · p. 7

Related papers

Loading 2610.03453v1…