PaperlayerView exact revision on arXiv

Source-linked AI summary

FILIGREE3D: Scaling Sparse Latent Flow Matching for Ultra-High-Resolution Image-to-3D Generation

Hongjie Li, Xinran Yang, Xiuchao Wu, Jiangjing Lyu, Chengfei Lv

arXiv:2609.34900v1cs.CV

TL;DR

Single-image 3D generation still struggles to recover fine geometry, while scaling resolution raises computational costs. Filigree3D uses sparse latent flow matching with structure-aware scaling, multi-level image conditioning, and resolution-gain data curation. It reports improved geometric detail over existing methods and competitive performance with commercial systems.

  • Problem

    Existing image-to-3D systems may oversmooth or omit thin structures, sharp creases, shallow reliefs, and subtle surface variations, while higher resolution increases computational costs.

    Source: §1 · p. 2

  • Method

    Filigree3D uses sparse latent flow matching with structure-aware scaling, block-specific multi-level image conditioning, and resolution-gain data curation for single-image 3D generation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Filigree3D improves geometric detail over existing image-to-3D methods while remaining competitive with commercial systems, and outperforms compared baselines on most metrics across two benchmarks.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    At effective 2048^3 resolution, Filigree3D recovers finer geometry while preserving global consistency under a fixed token budget, with modest computational overhead.

    Source: §5 · p. 9

Abstract

from arXiv · show

Scaling image-to-3D generation to ultra-high resolutions requires controlling rapidly growing computational costs without sacrificing fine geometric detail. We present \textbf{Filigree3D}, a sparse latent flow-matching framework that generates 3D geometry from a single image at voxel resolutions up to $2048^3$, with straightforward extensibility to $4096^3$. To make training tractable, we introduce Structure-Aware Sparse Scaling, which combines spatial bounding with alternating local-global attention to constrain token growth while preserving both fine-scale details and long-range structural context. To enhance detail reconstruction, we curate training samples based on their high-resolution geometric gains and inject multi-scale image features into a sparse 3D DiT, effectively coupling structural semantics with fine-grained visual cues. Furthermore, a visibility-aware voxel regularization strategy improves robustness against sparse perturbations and facilitates the completion of unobserved geometry. Under our default configuration, Filigree3D maintains peak GPU memory consumption within practical limits for contemporary hardware, enabling the generation of highly intricate 3D geometry in approximately one minute. Extensive experiments demonstrate that our method yields substantial improvements in overall geometric fidelity and fine-detail preservation compared to existing baselines, validating practical, detail-preserving 3D generation at unprecedented resolutions.

1 INTRODUCTION

High-resolution single-image 3D generation remains limited by rapidly growing compute, the need to balance local detail with global coherence, and the need for informative conditioning and detail-bearing supervision. Filigree3D addresses these challenges with sparse scaling, multi-level conditioning, and resolution-gain data curation.

1 INTRODUCTION

  • Sparse representations avoid allocating features in empty space, but active elements still grow sharply as voxel size decreases, making global attention difficult under practical memory budgets.High-resolution generation must bound fine-scale computation without discarding local neighborhoods that define geometric detail.Source: §1 · p. 2
  • Spatial cropping and local attention constrain compute but limit direct long-range interaction, while independent crops may create artificial boundaries and partial-training/full-object-inference mismatch.The method therefore aims to preserve fine local interactions and communication across the full shape.Source: §1 · p. 2
  • Single-view images constrain visible surfaces but leave self-occluded regions to learned shape priors, while high-resolution data collections vary in quality and detail-bearing value.The paper motivates intermediate- and final-layer visual features and prioritizing assets that reveal additional geometry at the target resolution.Source: §1 · p. 2
  • Filigree3D outperforms existing image-to-3D methods in geometric detail while remaining competitive with commercial systems.The paper attributes this to designs for ultra-high-resolution generation and fine-detail preservation.Source: §1 · p. 2
  • Filigree3D generates single-image 3D geometry at effective 2048^3 resolution, scales to 4096^3, and combines sparse scaling, multi-level conditioning, and resolution-gain data curation.The framework also supports joint full-support inference.Source: §1 · p. 2

2 RELATED WORK

Prior image-to-3D work spans feed-forward reconstruction, generative priors, and increasingly sparse or structured 3D representations. Filigree3D targets scaling the sparse generative backbone beyond prior effective resolutions.

2 RELATED WORK

  • Earlier image-to-3D approaches include per-instance score-distillation optimization, feed-forward reconstruction, and generative priors over native 3D representations.The passage describes flow-matching systems, shape- VAE-based DiTs, image-conditioned latent diffusion, and pixel-aligned feature conditioning among these directions.Source: §2 · p. 3
  • Sparse and structured representations include set-based tokens, voxel hierarchies, octrees, primitives, constructive geometry, and part-level attention, with several systems scaling to 1024^3.The passage distinguishes latent density, decoder resolution, and nominal voxel resolution as separate aspects of scalability.Source: §2 · p. 3
  • Filigree3D inherits sparse flow matching for image-conditioned generation but targets scaling the sparse generative backbone beyond 1024^3.The related-work discussion identifies this as distinct from prior representation-level modifications.Source: §2 · p. 3

3 METHOD

Filigree3D extends sparse latent flow matching for high-resolution single-image 3D generation with bounded sparse processing, block-specific image guidance, visibility-aware regularization, and resolution-sensitive training data.

3.1 PRELIMINARY

  • O-Voxel represents geometry with surface-intersecting voxels, which an SC-VAE encodes into sparse latents for an image-conditioned DiT to denoise and decode into a mesh.The latent encoder reduces each spatial dimension by 16; inference integrates the learned flow from noise to the data endpoint before decoding.Source: §3.1 · pp. 3–4

3.2 STRUCTURE-AWARE SPARSE SCALING

  • Structure-Aware Sparse Scaling caps training tokens with occupancy-balanced core–halo crops, explicit crop-window conditioning, and local attention supplemented by periodic coarse-global communication.The denoiser processes the full context, but the flow-matching loss applies only to the core, avoiding direct supervision near artificial crop boundaries; samples within budget use complete support.Source: §3.2 · pp. 4–5

3.3 BLOCK-SPECIFIC MULTI-LEVEL CONDITION INJECTION

  • Four levels of frozen DINOv2 features are fused by independent DiT-block modules, allowing blocks to combine visual abstraction levels while keeping conditioning-token count fixed.The feature levels are concatenated along channels without fixed weights before block-specific fusion.Source: §3.3 · pp. 5–6

3.4 VISIBILITY-AWARE VOXEL REGULARIZATION

  • Visibility-aware regularization perturbs and masks unseen voxels more strongly, weights their regression more heavily, and regularizes adjacent unseen-voxel features during training.The regularization is training-only and adds no inference overhead.Source: §3.4 · p. 6

3.5 RESOLUTION-GAIN-AWARE DATA CURATION.

  • Resolution-gain curation screens assets for geometry missing in low-resolution reconstructions but recovered at high resolution, then retains only clear, spatially coherent recoveries.The screening compares low- and high-resolution reconstructions against shared ground-truth surface samples, including detail-aware sampling of high-curvature or normal-varying regions.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments evaluate Filigree3D on two detail-rich benchmarks, finding improved geometric fidelity and detail preservation across open-source and commercial comparisons. Ablations and resolution comparisons further examine the model’s components and scaling behavior.

4.1 EXPERIMENT SETUP

  • Evaluation uses 537 detail-rich Toys4K objects and 120 FineGeo3D cases, comparing four named open-source baselines with identical normalization and sampling protocols.Metrics include CD, F-scores at 0.01 and 0.002, ULIP-2, Uni3D, and inference time.Source: §4.1 · pp. 6–7

4.2 EXPERIMENT RESULTS

  • On Toys4K and FineGeo3D, Filigree3D preserves thin bars, hollow ornaments, sharp creases, and shallow reliefs better than existing methods.It also produces more plausible geometry in occluded or partially visible regions while retaining detail on visible surfaces.Source: Figure 3 · p. 7, Figure 4 · p. 8, §4.2 · p. 8
  • Across Toys4K and FineGeo3D, Filigree3D outperforms all compared baselines on most metrics, with particularly large gains on fine-structure fidelity.The evaluation covers geometric fidelity, semantic alignment, and inference time.Source: Table 1 · p. 7, §4.2 · p. 8

4.1 EXPERIMENT SETUP

  • Training uses TRELLIS-500K with additional geometric-complexity filtering and 200K collected high-quality, high-precision examples; the model trains for 10 days on 16 NVIDIA H20 GPUs.The setup also fine-tunes a separate SC-VAE for each target resolution and uses a 50,000-token budget.Source: §4.1 · p. 7

4.3 ABLATION STUDY

  • Removing multi-scale features, global information flow, voxel regularization, or data curation degrades FineGeo3D performance; the full model achieves the best results.The components have complementary contributions to geometric fidelity and semantic consistency.Source: Table 2 · p. 9, §4.3 · p. 9
  • From 1536^3 to 4096^3, increasing resolution progressively improves geometric fidelity; against commercial systems, Filigree3D better preserves intricate patterns and high-frequency details.The commercial comparison uses systems at their highest-quality settings.Source: Figure 5 · p. 9, §4.3 · p. 9

5 CONCLUSION

Filigree3D targets single-image 3D generation at effective 2048^3 resolution and scales to 4096^3. The conclusion attributes finer geometry and global consistency under a fixed token budget to jointly designed sparse computation, multi-level conditioning, and resolution-aware supervision.

5 CONCLUSION

  • Filigree3D generates single-image 3D geometry at effective 2048^3 resolution, scalable to 4096^3.The conclusion reports finer geometry with modest overhead and performance competitive with leading commercial systems.Source: §5 · p. 9

A.1 MODEL DETAILS

The model details specify inherited TRELLIS.2 configurations and Filigree3D-specific denoiser, sparse-scaling, and conditioning settings.

A.1 MODEL DETAILS

  • The denoiser uses 32-channel structured latents, 30 Transformer blocks, an MLP expansion ratio of 5.3334, and rotary positional embeddings.It is optimized with AdamW at a learning rate of 1 × 10^-4.Source: §A.1 · p. 13
  • Filigree3D uses a 2,048-wide, 16-head denoiser with 2.2B parameters and a 50,000-token budget for Structure-Aware Sparse Scaling.DINOv2 features from layers 5, 7, 11, and 23 condition every DiT block via cross-attention; local windows have width 32, with coarse-global communication every six blocks.Source: §A.1 · p. 13

A.2 IMPLEMENTATION DETAILS

The implementation adapts the geometry autoencoder and sparse latent processing to high-resolution grids. The denoiser is trained in stages and uses 12 inference steps.

A.2 IMPLEMENTATION DETAILS

  • The SC-VAE is fine-tuned separately for each target resolution to address resolution-dependent limitations.The passage names the resolution-specific autoencoder adaptation but does not specify the two limitations here.Source: §A.2 · p. 13
  • 64-bit voxel keys are used when R^3 exceeds 2^32 to avoid collisions that could connect distant voxels.The 32-bit key space is insufficient for a 2048^3 grid.Source: §A.2 · p. 13
  • The 4096^3 grid is divided into 4 × 4 × 4 blocks of size 1024^3, encoded locally, then assembled into a 256^3 latent grid.Each block yields a 64^3 latent block with 16× spatial downsampling; the approach applies when block size is at most 1024 and divisible by 16.Source: §A.2 · p. 13
  • The denoiser is trained first on TRELLIS-500K, then the 2048^3 model is fine-tuned on a high-detail subset and an additional 200K objects.The sparse-structure DiT is also fine-tuned to generate structures on a 128^3 voxel grid; inference uses 12 sampling steps.Source: §A.2 · p. 14

A.2.1 GEOMETRY METRICS

The evaluation combines geometric distance and thresholded F-scores with image–shape semantic similarities. It measures visible outer surfaces using sampled point clouds.

A.2.1 GEOMETRY METRICS

  • The evaluation reports Chamfer Distance, F-scores at 0.01 and 0.002, ULIP-2 similarity, and Uni3D similarity.The two F-score thresholds provide one less strict and one stricter threshold.Source: §A.2.1 · pp. 14–15
  • Evaluation uses only the externally observable outer surface, marking faces visible when at least one of 20,000 sphere-cast rays intersects them.The union of marked faces defines the visible submesh.Source: §A.2.1 · p. 14
  • The predicted and reference point clouds are each sampled with 100,000 area-weighted points from their visible submeshes.Source: §A.2.1 · p. 14
  • Chamfer distances are bidirectional and untrimmed, with nearest-neighbor distances computed using a KD-tree.The passages introduce the nearest-neighbor and directional mean-distance calculations.Source: §A.2.1 · p. 14
  • On FineGeo3D, the qualitative comparisons report finer geometric structures and more faithful detail preservation than existing methods.Source: Figure 6 · p. 15
  • F-scores use precision and recall at a distance threshold; after similarity alignment, absolute thresholds are comparable across methods.The reported thresholds are 0.01 and 0.002; the score is set to zero when both precision and recall are zero.Source: §A.2.1 · p. 15
  • ULIP-2 and Uni3D similarities compare generated point-cloud embeddings with the reference-image embedding using cosine similarity.The same CLIP vision encoder embeds the reference image used to condition all methods.Source: §A.2.1 · p. 15

A.3 ADDITIONAL COMPARISONS AND 20483 GENERATION RESULTS

Additional FineGeo3D comparisons report finer local geometry and thin structures alongside coherent global shapes. Further 2048^3 examples show fine-detail recovery while preserving overall shape consistency.

A.3 ADDITIONAL COMPARISONS AND 2048^3 GENERATION RESULTS

  • FineGeo3D comparisons report better recovery of fine local geometry and thin structures while maintaining coherent global shapes across diverse categories.The comparisons include Direct3D-S2 (Wu et al., 2024), Hunyuan3D 2.1 (Team Hunyuan3D et al., 2025), TRELLIS.2 (Xiang et al., 2025a), and Pixal3D (Li et al., 2026).Source: §A.3 · p. 15
  • At 2048^3, additional generations recover finer geometric structures without compromising overall shape consistency.The examples include objects with intricate surfaces and slender components.Source: §A.3 · p. 16, Figure 7 · p. 16

Related papers

Loading 2609.34900v1…