PaperlayerView exact revision on arXiv

Source-linked AI summary

Native and Compact Structured Latents for 3D Generation

Jianfeng Xiang, Xiaoxue Chen, Sicheng Xu, Ruicheng Wang, Zelong Lv, Yu Deng, Hongyuan Zhu, Yue Dong, Hao Zhao, Nicholas Jing Yuan, Jiaolong Yang

arXiv:2512.14692v1cs.CVcs.AI

TL;DR

Existing 3D representations inadequately capture arbitrary topology and detailed appearance while remaining suitable for compact neural latents. The paper introduces O-Voxel and a Sparse Compression VAE, then trains large flow-matching models that report efficient inference and superior geometry and material quality. Its scope remains bounded by voxel resolution and occasional small holes in decoded or generated meshes.

  • Problem

    Existing representations struggle to faithfully capture arbitrary 3D topology, detailed appearance, and material information while being processed into neural latents.

  • Method

    The paper uses O-Voxel for native geometry-and-appearance representation, a Sparse Compression VAE for compact latents, and large flow-matching models for 3D generation.

  • Results

    The models report substantially superior geometry and material quality while generating fully textured assets in approximately 3s at 512^3, 17s at 1024^3, and 60s at 1536^3 on an NVIDIA H100.

  • Takeaways & Limitations

    The approach provides a comprehensive yet compact structured 3D latent representation with high generation efficiency and quality.

  • Takeaways & Limitations

    O-Voxel is bounded by voxel resolution, which can cause aliasing and blurred appearance for features smaller than the voxel size.

Abstract

from arXiv · show

Recent advancements in 3D generative modeling have significantly improved the generation realism, yet the field is still hampered by existing representations, which struggle to capture assets with complex topologies and detailed appearance. This paper present an approach for learning a structured latent representation from native 3D data to address this challenge. At its core is a new sparse voxel structure called O-Voxel, an omni-voxel representation that encodes both geometry and appearance. O-Voxel can robustly model arbitrary topology, including open, non-manifold, and fully-enclosed surfaces, while capturing comprehensive surface attributes beyond texture color, such as physically-based rendering parameters. Based on O-Voxel, we design a Sparse Compression VAE which provides a high spatial compression rate and a compact latent space. We train large-scale flow-matching models comprising 4B parameters for 3D generation using diverse public 3D asset datasets. Despite their scale, inference remains highly efficient. Meanwhile, the geometry and material quality of our generated assets far exceed those of existing models. We believe our approach offers a significant advancement in 3D generative modeling.

1. Introduction

The paper introduces a native, structured 3D representation and latent-generation pipeline designed to capture arbitrary geometry, appearance, and materials efficiently. Its models combine O-Voxel, Sparse Compression VAE, and large flow-matching generation with high reconstruction and generation quality.

  • Existing representations struggle with arbitrary topology, full asset information, and appearance or material modeling.Field-based methods have limitations for open, non-manifold, and enclosed geometry, while many works neglect appearance.
  • O-Voxel encodes geometry and appearance while supporting open, non-manifold, and fully-enclosed surfaces.Its flexible dual grid preserves sharp edges and normal discontinuities, and it represents PBR parameters and opacity for re-lighting and translucent surfaces.
  • O-Voxel converts meshes in a few seconds on one CPU and reconstructs surfaces and materials within tens of milliseconds.Both directions are optimization-free and rendering-free.
  • The Sparse Compression VAE achieves 16× spatial downsampling and encodes a fully textured 1024^3 asset into approximately 9.6K latent tokens.The method reports negligible perceptual degradation and higher reconstruction quality than prior methods using substantially fewer tokens.
  • Large flow-matching models contain about 4 billion parameters and generate fully textured assets in approximately 3s at 512^3, 17s at 1024^3, and 60s at 1536^3 on an NVIDIA H100.The paper reports substantially higher geometry and material quality than existing models.

2. Related Work

Prior 3D generation representations trade topology and appearance coverage against structural regularity, compression, or processing efficiency. The paper positions its native structured latent pipeline as addressing these limitations without multiview asset-processing stages.

  • 3D Representations for Generation: Implicit fields and iso-surface-based structured representations provide high-resolution geometry but cannot reliably represent open or non-manifold surfaces.NeRF also incurs low geometry quality and heavy sampling costs.
  • 3D Representations for Generation: Unstructured meshes, point clouds, and Gaussians offer explicit geometry but lack structural regularity for neural processing and latent compression.
  • Latent 3D Representations: Unstructured latents compress strongly but are typically limited in reconstruction fidelity, whereas structured sparse latents improve geometric accuracy at the cost of more tokens.
  • Large 3D Asset Generation Models and Systems: Many large 3D systems separate shape generation from multiview texture synthesis, requiring rendering, baking, and alignment that hinder scalability and can cause appearance inconsistencies.The paper contrasts this with native, end-to-end 3D asset generation.

3. Method

The method introduces O-Voxel, a native sparse voxel representation that jointly models arbitrary-topology geometry and material attributes, then compresses it with SC-VAE for structured latent generation. A three-stage pipeline separately generates sparse structure, geometry latents, and spatially aligned material latents.

  • O-Voxel representation: O-Voxel represents geometry and appearance on active voxels, including arbitrary topology such as open, non-manifold, and fully-enclosed surfaces.Its flexible dual grid adjusts dual-vertex positions and face existence to represent complex surfaces.
  • O-Voxel representation: The flexible dual grid derives surface connectivity from intersected voxel edges and uses Hermite data with a QEF to position dual vertices.Additional boundary-edge and regularization terms improve open-surface alignment and stabilize vertex optimization.
  • O-Voxel representation: O-Voxel supports instant, optimization-free, rendering-free conversion between meshes and sparse voxels, with reconstruction producing render-ready surfaces and materials.Mesh conversion takes seconds on one CPU, while surface and material reconstruction takes tens of milliseconds.
  • O-Voxel representation: Its material features encode PBR attributes aligned with geometry, including base color, metallic ratio, roughness, and opacity.Texture attributes are sampled into active voxels and reconstructed through trilinear interpolation.
  • Sparse Compression VAE: SC-VAE is a fully sparse-convolutional U-shaped VAE that compresses O-Voxel data through residual autoencoding layers and reconstructs it symmetrically.The residual shortcuts rearrange information between spatial and channel dimensions during sparse-grid downsampling and upsampling.
  • Generative modeling: The generation pipeline predicts sparse structure, geometry latents, and material latents, with a sparse DiT generating PBR materials conditioned on the image and geometry.All stages operate in the native 3D latent domain, maintaining spatial alignment under arbitrary topology.

4. Experiments

The experiments evaluate reconstruction, image-to-3D generation, texture synthesis, ablations, and test-time scaling. Across these settings, the method combines strong geometric and material fidelity with compact representations and efficient generation.

  • Shape Reconstruction: The method consistently outperforms four shape-reconstruction baselines across every reported metric while using fewer tokens and less runtime.The evaluation measures mesh fidelity, visible-surface fidelity, and efficiency.
  • Material Reconstruction: 38.89 dB / 0.033 on PBR attributes and 38.69 dB / 0.026 on shaded images demonstrate faithful material reproduction and geometry–appearance alignment.The metrics are PSNR / LPIPS, respectively.
  • Image-to-3D Generation: Generated assets preserve fine structures and topologically complex shapes while producing vivid PBR textures with physically consistent shading, including translucent and reflective materials.Examples include detailed gears, enclosed cockpits, open leaves and flowers, glass, and metal.
  • Image-to-3D Generation: The method achieves superior qualitative generation quality, with accurate geometry, physically plausible materials, and faithful prompt alignment compared with five state-of-the-art systems.The comparison includes TRELLIS, Hi3DGen, Direct3D-s2, Step1X-3D, and Hunyuan3D 2.1.
  • Quantitative and User Evaluation: The method achieves the highest alignment score across CLIP, ULIP-2, and Uni3D evaluations, and participants favor it in a study using 100 image prompts.About 40 participants evaluated perceptual quality under identical, uncurated conditions.
  • Shape-Conditioned Texture Generation: Native 3D appearance reasoning produces sharper textures, consistent shape–material alignment, and textures for internal surfaces without multi-view or UV seam artifacts.The texture generator is evaluated against multi-view fusion and UV-based methods.
  • Ablation and Design Analysis: At 16× compression, the sparse residual design preserves high fidelity, whereas average-pooling and nearest-neighbor upsampling degrade sharply; at 32×, degradation reaches 526% MD and 1.6 dB PSNR.The baseline degradation is 69% MD and 0.5 dB PSNR at 16×, worsening at 32×.
  • Ablation and Design Analysis: The optimized residual block improves reconstruction quality over standard residual blocks, reducing the reported quality drop without changing runtime.The standard-block baseline increases MD by 16% and decreases PSNR by 0.6 dB.

5. Conclusion

The paper presents O-Voxel and a Sparse Compression VAE as a comprehensive, compact latent representation for 3D generation. Its flow-matching models achieve substantially higher generation quality while maintaining efficient operation.

  • Conclusion: The framework learns a comprehensive yet compact structured 3D latent representation for generation.Its core components are O-Voxel and a Sparse Compression VAE.
  • Conclusion: O-Voxel encodes complex geometry and materials and supports bidirectional conversion between standard assets and the representation.The conversion pipeline covers meshes, PBR textures, shape reconstruction, and texture reconstruction.
  • Conclusion: The large flow-matching models deliver substantially superior generation quality compared with existing methods while maintaining high efficiency.The conclusion attributes the learned latent space to the framework’s representation and generation approach.

A.2. Network Architectures

The section describes the sparse-convolutional SC-VAE and Transformer-based generative models, along with O-Voxel conversion and conditioning components. The architecture uses sparse processing and explicit geometric conditioning to support scalable 3D generation.

  • Sparse Compression VAE: The SC-VAE is a fully sparse-convolutional U-shaped VAE with ConvNeXt-style residual blocks and Residual AutoEncoding layers.Its decoder mirrors the encoder architecture.
  • Generative Models: The generation framework uses three Transformer-based encoder-only models without token packing or skip connections.The design prioritizes a clean and scalable architecture.
  • Conditioning: Shape information is concatenated channel-wise with the material-stage input tensor to provide explicit geometric conditioning.This conditioning is intended to improve material-shape alignment.
  • O-Voxel-to-Mesh Conversion: O-Voxel-to-mesh conversion creates vertices from dual-grid data and connects them across active edges into triangle faces.Quadrilaterals are split adaptively using splitting weights.
  • Attention Stabilization: RoPE supports generalization across input resolutions, while QK-Norm applies RMS normalization to queries and keys before attention.The QK-Norm scheme is used to improve training stability.

A.3. Training Details

The section details two-stage SC-VAE training, material feature conversion, rendering-based perceptual supervision, and rectified-flow generative modeling. It also describes sparse-kernel acceleration and resolution-agnostic application of the trained VAE.

  • SC-VAE Training: SC-VAE training begins with direct O-Voxel feature regression at 256^3 resolution and then adds rendering-based perceptual loss at 512^3.The second stage targets geometric sharpness and high-frequency material details.
  • Material Conversion: Material features are extracted for active voxels by collecting weighted samples from intersecting triangles and averaging the sampled PBR attributes.The resulting features are stored as O-Voxel material features.
  • Perceptual Loss: The perceptual distance combines L1, SSIM, and LPIPS terms as dp(a, b) = ∥a − b∥1 + 0.2 · dSSIM + 0.2 · dLPIPS.The prose identifies silhouette, depth, normal, base-color, and metallic-roughness-related quantities among the supervised renderings.
  • Material Conversion: O-Voxel material features can be applied to reconstructed meshes as vertex materials or texture maps.Both outputs produce meshes with PBR materials applied.
  • Generative Training: The rectified-flow model linearly interpolates between data and noise, then learns a time-dependent vector field with a Conditional Flow Matching objective.The forward path is x(t) = (1 − t)x0 + tϵ.
  • Sparse Convolution Backend: The Triton backend accelerates sparse convolutions across NVIDIA and AMD hardware using fused masked implicit GEMM kernels.Gray-code voxel ordering improves SIMD efficiency by reducing warp divergence and wasted computation.

C. Data Preparation Details

The data pipeline curates assets with usable PBR materials, augments material diversity, filters by estimated aesthetics, and renders varied views for image-conditioned training. Evaluation protocols and dataset composition are documented alongside preparation.

  • Asset Curation: The curated training collection excludes 3D-FUTURE because it lacks Physically-Based Rendering materials.The remaining assets provide the basis for shape-model training.
  • Material Filtering: Filtering to assets using a standard metallic-roughness PBR workflow yields approximately 350,000 assets for material-VAE training.A custom Blender script parses materials from the raw assets.
  • Quality Control: TexVerse is added to increase high-quality PBR-material diversity, while assets with estimated aesthetic scores below 4.5 are removed.Thumbnail images from Sketchfab provide the aesthetic-score estimate.
  • Image-Conditioned Training: Blender renders diverse asset views and applies augmentations such as randomly sampling camera field of view from 10° to 70°.These augmentations target ambiguities in real-world image inputs.
  • Evaluation Preparation: The paper reports quantitative comparisons and ablation studies using defined numerical evaluation metrics.Detailed calculation protocols are provided for the experiments.

D.1.1. Reconstruction Experiments

The reconstruction experiments evaluate geometry, visible shape, fine surface detail, and appearance using complementary mesh, point-cloud, and rendering metrics. The protocols distinguish enclosed geometry from visible surfaces and normalize meshes before measurement.

  • Geometry Accuracy: Mesh Distance evaluates discrepancy across complete triangle meshes, including fully enclosed surfaces, using dense surface sampling.It is presented as more stable than Chamfer Distance with respect to point-cloud density.
  • Visible-Surface Accuracy: Chamfer Distance evaluates visible outer-shell geometry from point clouds generated through depth rendering from 100 uniformly sampled camera views.The F-score uses a threshold of τ = 1 × 10^-6.
  • Surface Detail: Fine surface detail is measured with PSNR and LPIPS on normal maps rendered from four fixed camera positions.The views use yaw angles of 30°, 120°, 210°, and 300°.
  • Evaluation Protocol: All ground-truth and predicted meshes are normalized to fit within a unit cube before metric calculation.This normalization precedes the geometric evaluation definitions.
  • Appearance Fidelity: Appearance fidelity compares reconstructed and ground-truth PBR attribute maps and shaded images rendered with the same fixed-camera setup.PSNR and LPIPS are computed on the rendered outputs.

D.1.2. Generation Experiments

The generation experiments evaluate image-to-3D quality with both automated metrics and a human preference study, using diverse prompts and interactive comparisons. The study also reports compact-latent reconstruction quality across complex shapes and materials.

  • 100 diverse image prompts test image-to-3D generation across intricate geometries, dramatic lighting, and materials including metal, leather, rust, and glass.
  • CLIP, ULIP-2, and Uni3D measure semantic, visual, geometric, and appearance alignment between generated assets and input image prompts.
  • Human evaluation compares overall rendered quality and isolated shape quality through randomized, interactive turntable inspections.
  • About 40 participants produce global preference rates by recording each selected model as a win over competing options.
  • The compact latent space reconstructs hard-surface objects, thin structures, open surfaces, text, complex geometries, and detailed PBR materials with high fidelity.
  • Extended comparisons show sharper high-frequency geometry, more accurate enclosed structures, and visibly lower reconstruction errors than baselines.

E.2. Image to 3D Asset Generation

Image-to-3D results demonstrate broad generation coverage across organic, mechanical, and character assets while jointly modeling detailed geometry and physically based materials. Comparisons emphasize sharper geometry and stronger PBR material quality than several recent methods.

  • The method generates organic structures, hard-surface machinery, and detailed characters with rendered outputs, normal maps, PBR maps, and relighting results.
  • Generated assets jointly exhibit high-fidelity geometry and plausible PBR materials that respond correctly under novel lighting conditions.
  • Several baselines, including Step1X-3D, TRELLIS, Direct3D-S2, and Hi3DGen, do not provide PBR materials in the comparison.
  • Compared with recent methods, the results show sharper, more coherent geometric details and more accurate mechanical joints and character faces.

F. Limitation Discussion and Future Work

The discussion identifies limitations in spatial resolution, decoding stability, and semantic representation. Future work targets more stable reconstruction and richer structural or semantic encoding.

  • Voxel resolution limits representation power, causing aliasing when features smaller than a voxel share a voxel.
  • Close parallel surfaces may yield misplaced dual vertices and averaged material attributes, producing geometric errors and blurred appearance.
  • Reconstructed and generated meshes sometimes contain small holes because sparse decoding may not produce perfectly closed, manifold surfaces.
  • Standard hole-filling post-processing can mostly rectify these defects, while decoder stability remains an improvement target.
  • O-Voxel focuses on geometry and material but does not explicitly encode higher-level structural or semantic information.
  • Future extensions may incorporate part-level segmentation and graph-based topological structure.
Loading 2512.14692v1…