PaperlayerView exact revision on arXiv

Source-linked AI summary

Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion

Bowen Cui, Weijie Wang, Zeyu Zhang, Yefei He, Mingda Lin, Haoyu Zhao, Yuanyu He, Donny Y. Chen, Feng Chen, Bohan Zhuang

arXiv:2608.19567v3cs.CV

TL;DR

High-fidelity text-to-3D generation remains costly because autoregressive decoding is sequential and global diffusion or flow refinement repeatedly processes complete representations. Block3D generates contiguous blocks autoregressively, jointly denoises active-block tokens, and corrects low-confidence tokens before finalization; it reports faster generation with strong geometric fidelity.

  • Problem

    High-fidelity text-to-3D generation remains expensive because autoregressive methods decode sequentially, while diffusion and flow methods repeatedly refine complete representations.

  • Method

    Block3D partitions discrete shape-token sequences into contiguous blocks, generates blocks left to right, jointly denoises active-block tokens, and applies confidence-guided intra-block correction.

  • Results

    5.15× speedup reduces mean end-to-end generation time from 25.71 seconds to 4.99 seconds versus the fine-tuned Cube baseline, while Block3D achieves strong geometric fidelity.

  • Takeaways & Limitations

    Block3D reduces sequential generation latency while retaining competitive text-shape alignment and improving paired geometric fidelity relative to Cube.

  • Takeaways & Limitations

    Completed blocks remain fixed, so Block3D does not remove generated-prefix exposure bias and future work must study cross-block refinement and broader representations and datasets.

Abstract

from arXiv · show

While text-to-3D generation has advanced rapidly, achieving high geometric fidelity at low inference cost remains challenging. Existing text-to-3D methods either decode discrete shape tokens autoregressively or iteratively refine global 3D representations with diffusion or flow models. However, autoregressive decoding is sequential and cannot revise errors, whereas diffusion and flow-matching models repeatedly process the full representation, making high-quality generation increasingly expensive. In this paper, we propose Block3D, a block-wise diffusion framework that partitions the discrete shape-token sequence into contiguous blocks, generates the blocks autoregressively, and jointly denoises all tokens within the current block. To alleviate error accumulation, we introduce confidence-guided intra-block correction, which revises low-confidence tokens before each block is finalized. On a held-out set from TRELLIS-500K, Block3D reduces mean end-to-end generation time from 25.71 seconds to 4.99 seconds, achieving a $5.15\times$ speedup over the fine-tuned autoregressive baseline without sacrificing geometric fidelity.

1 Introduction

Block3D addresses the cost–fidelity trade-off in text-to-3D generation by generating contiguous latent blocks autoregressively while jointly denoising and correcting tokens within each active block. Experiments report faster generation and strong geometric fidelity relative to the fine-tuned Cube baseline.

  • Motivation: Existing text-to-3D systems either decode long shape-token sequences sequentially or repeatedly refine complete representations, increasing inference cost as detail grows.These approaches create a central efficiency challenge for high-fidelity geometry.
  • Method: Block3D shifts autoregressive causal dependencies from individual shape tokens to contiguous latent blocks and jointly denoises all tokens within the current block.Blocks are generated from left to right, while tokens inside the active block use parallel bidirectional denoising.
  • Method: Confidence-guided intra-block correction revises low-confidence tokens before a block is finalized, combining mask-to-token recovery with token-to-token editing.This mechanism is designed to alleviate error accumulation within each block.
  • Results: 5.15× speedup reduces mean end-to-end generation time from 25.71 seconds to 4.99 seconds on 100 held-out objects versus the separately fine-tuned Cube baseline.The reported evaluation also finds the best geometric metrics among the evaluated methods.
  • Results: Block3D achieves the best geometric metrics among evaluated methods while substantially reducing sequential generation latency.The contribution summary reports both geometric superiority and lower generation time.

2 Related Work

Related text-to-3D work spans optimization and diffusion or flow-based generation, sequence prediction, and block-wise denoising. Block3D combines block-causal generation with parallel active-block editing while retaining a fixed discrete representation and frozen completed prefixes.

  • Diffusion and flow methods: Diffusion and flow-based methods generate or refine 3D representations through optimization, learned diffusion, flow matching, staged pipelines, or related iterative strategies.These approaches include global, implicit, multi-view, and structured-latent representations.
  • Sequence prediction: Sequence-based methods represent 3D shapes as quantized or vector-quantized codes, meshes, serialized geometry, or language-model tokens.The related systems vary in tokenization, multimodal modeling, and the length or locality of autoregressive dependence.
  • Limitations of autoregression: Token-wise autoregression makes emitted codes irreversible and leaves Block3D with generated-prefix exposure bias because completed blocks remain fixed.Block3D adds only a bounded revision opportunity within the active block before commitment.
  • Discrete denoising: PartDiffuser uses semantic parts and TSSR uses global remasking, whereas Block3D operates on part-free fixed-length Cube codes and freezes completed prefix blocks.This distinguishes Block3D’s causal grouping and refinement scope from those related discrete denoising approaches.
  • Block-wise diffusion: Block Diffusion supplies block-level autoregressive factorization, active-block bidirectional denoising, and prefix caching, while LLaDA2.1 supplies M2T/T2T editing mechanisms.Block3D adapts these mechanisms with shape-code corruption, a residual objective, and confidence-gated revision.

3 Method

Block3D generates fixed-length 3D shape codes in contiguous causal blocks, jointly denoising each active block while preserving editable states and cached prefixes. Its training and decoding combine block-causal attention, masked and substituted corruption, model-based rollout, and confidence-guided updates.

  • 3.1 Overview: Block3D uses Cube's fixed-length shape representation, dividing N = 1024 discrete codes into K contiguous blocks while retaining the frozen tokenizer and decoder.The generator is conditioned on text features and produces blocks from left to right.
  • 3.4 Bounded Confidence-Guided Decoding: A deterministic reveal quota ensures masks are removed within at most T iterations, while completed prefix blocks are committed and frozen.High-confidence masked-to-token updates may reveal more positions than the quota's lower bound.
  • 3.2 Block-Causal Shape-Code Denoising: Each active block receives bidirectional attention across its positions and attends to the complete generated prefix, while future blocks remain absent.Prefix queries retain token-causal visibility during inference.
  • 3.3 Edit-Aware Corruption and Training: Training constructs clean and corrupted block contexts jointly, exposing the model to masked and substituted shape codes before supervising residual errors after one model-based rollout.The corruption stream is selected per sample, and the rollout can produce masks, correct codes, or incorrect model-generated codes.
  • 3.2 Block-Causal Shape-Code Denoising: Prefix caching reduces repeated computation, but inference still differs from teacher-forced training and retains generated-prefix exposure bias across completed blocks.Block revisions are therefore limited to the currently active block.
  • 3.4 Bounded Confidence-Guided Decoding: Confidence-guided decoding fills masked positions and replaces filled codes within the active block, using conditional and unconditional branches with separate confidence thresholds.The deterministic sampler applies these updates within each block before committing its codes.

4 Experiments

Block3D is evaluated on held-out TRELLIS-500K objects using geometric fidelity, text-shape alignment, latency, and ablation studies. It achieves strong geometry and substantially lower generation time, while experiments identify quality–latency trade-offs for block size and denoising steps.

  • Experimental setup: Block3D is evaluated on 100 held-out TRELLIS-500K objects with paired text prompts and independently trained ablation configurations.The evaluation uses frozen Cube tokenization and compares against ShapeLLM-Omni, TRELLIS-text, AR3D-R1, and Cube.
  • Experimental setup: The evaluation reports Chamfer-L1, normal consistency, F@1% geometry, CLIPScore text-shape alignment, and end-to-end latency on one NVIDIA A100.Latency includes condition encoding, shape-code generation, and mesh decoding, excluding model loading and disk I/O.
  • Main results: Block3D attains the strongest CD-L1, NC, and F@1% scores among evaluated methods while retaining competitive CLIPScore.The controlled Cube comparison indicates improved paired geometric fidelity alongside faster generation.
  • Main results: 4.99 seconds is Block3D’s mean end-to-end generation time, corresponding to a 5.15× speedup over the controlled Cube baseline.Median and P90 latency also improve across the 100 prompts.
  • Ablation study: Moving from B = 64 to 96 lowers latency from 4.99 to 3.62 seconds but reduces F@1% from 0.309 to 0.279, while B = 256 lowers F@1% to 0.103.The paper therefore uses B = 64 as the quality–latency balance.
  • Ablation study: At T = 20, latency rises to 13.66 seconds while CD-L1 remains 0.078 and F@1% is 0.303, compared with 4.99 seconds and 0.309 at T = 4.The paper therefore uses four denoising steps; adding T2T editing further reduces CD-L1 by 4.7% and raises F@1% from 0.287 to 0.309.

5 Conclusion

Block3D replaces Cube’s token-wise prior with block-causal denoising and confidence-guided correction, improving geometry and reducing generation time while retaining competitive text-shape alignment.

  • Conclusion: Block3D generates shape-code blocks from left to right while jointly denoising and correcting active-block tokens.Completed blocks remain fixed during generation.
  • Conclusion: Block3D improves paired geometry and reduces Cube’s mean generation time from 25.71 to 4.99 seconds while retaining competitive text-shape alignment.The conclusion identifies cross-block refinement and broader shape representations and datasets as future work.

A Complete Method Details

Block3D preserves Cube’s frozen discrete shape representation and decoder while conditioning generation on frozen CLIP text features through a fine-tuned shape generator.

  • Shape Representation: Block3D retains Cube’s frozen vector-quantized shape representation for text-to-shape generation.The representation uses a fixed-length discrete shape-code sequence.
  • Shape Representation: The frozen shape decoder converts completed valid shape-code sequences into output meshes.The auxiliary mask symbol is excluded from output normalization and never passed to the decoder.
  • Condition Representation: Frozen CLIP ViT-L/14 produces 77 token-level condition features from each text prompt.Reported experiments use text alone without an appended bounding-box token.
  • Trainable Generator: The trainable generator is initialized from Cube’s 23-layer DualStream RoFormer generator.The VQ encoder, codebook, CLIP text encoder, and shape decoder remain frozen during fine-tuning.

A.2 Training Attention and Position IDs

Training partitions fixed-length shape codes into contiguous blocks and uses masked attention to expose each corrupted block only to its condition, clean prefix, and own corrupted state.

  • Block Notation: For block size B, the N shape positions are divided into K contiguous blocks.Block r contains positions from (r−1)B+1 through min(rB,N).
  • Training Construction: Training concatenates clean and corrupted sequences while sharing one condition sequence.The clean and corrupted copies are indexed blockwise for vectorized attention computation.
  • Attention Visibility: Condition queries attend only to condition tokens, while every shape query can attend to all condition tokens.The attention mask isolates condition queries from the shape sequence.
  • Attention Visibility: A clean block attends bidirectionally to itself and preceding clean blocks, never to corrupted copies.This provides clean teacher-forced context without exposing later blocks.
  • Attention Visibility: A corrupted block attends to the condition, clean preceding blocks, and its own corrupted tokens, excluding clean targets and future blocks.Corrupted blocks cannot read another corrupted block.
  • Position IDs: Clean and corrupted copies share logical shape positions, preserving global position IDs despite physical duplication.At inference, prefix queries are token-causal and active-block queries are bidirectional within the active block.
  • Training–Inference Distinction: Training uses teacher-forced clean prefixes, whereas inference uses generated token-causal prefixes; caching does not remove this exposure difference.Completed blocks remain immutable in both cache and generated sequence.
  • Corruption and Rollout: The corruption stream mixes mask-to-token and token-to-token corruption with independently sampled block corruption rates.The reported corruption range is [0.45, 0.95], and the T2T stream probability is 0.5.

B.1 Data Split and Prompt Provenance

The evaluation uses a fixed, seeded TRELLIS-500K manifest with paired prompts, exact asset-level separation from training, and recomputable provenance and preprocessing properties.

  • B.1 Data Split and Prompt Provenance: Each evaluation asset retains its original TRELLIS-500K paired text prompt for every generation method.The same stored prompt is supplied without method-specific editing.
  • B.1 Data Split and Prompt Provenance: Seed 42 selects 100 distinct evaluation records spanning 53 Sketchfab, 46 GitHub, and one ABO source.Every method receives the same ordered prompt list and generates one output per prompt.
  • B.1 Data Split and Prompt Provenance: The split enforces exact asset-level disjointness between evaluation and fine-tuning pools.It does not claim category-level or semantic near-duplicate removal.
  • B.1 Data Split and Prompt Provenance: The manifest is characterized using properties recomputable directly from its ordered records rather than a hand-assigned category taxonomy.
  • B.1 Data Split and Prompt Provenance: Figure C reports source counts, verbatim prompt-length bins, and Equation B.1 anisotropy for the seeded sample without manual balancing.The source counts sum to 100.
  • B.1 Data Split and Prompt Provenance: Target anisotropy is computed from scale-invariant bounding-box vectors.
  • B.1 Data Split and Prompt Provenance: Prompt lengths span 10–38 words with median 19.0, while 36 of 100 targets have anisotropy above 3.Bounding-box vectors are evaluator-only inputs.
  • B.2 Data Preprocessing: Meshes undergo Cube preprocessing that removes invalid geometry, preserves orientation, retains nonempty components, and normalizes scale.No category-specific rotation, canonical-pose fitting, or ICP alignment is applied.

B.3 Training Configuration

The study fixes a reproducible training and evaluation protocol, selecting block and denoising settings by measured latency–geometry trade-offs and reporting standardized geometry, alignment, and timing metrics.

  • Training Configuration: Block3D trains only the text-to-shape generator on 300K objects from the released Cube initialization.The tokenizer, codebook, CLIP encoder, and shape decoder remain frozen.
  • Training Configuration: Training uses 35K optimizer updates in bfloat16 on four NVIDIA A100 80GB GPUs with global batch size 40.
  • Training Configuration: The training setup uses one optimizer update per loaded global batch and tracks sample presentations rather than unique objects.
  • Training Configuration: The corruption process analytically expects roughly 19.2 clean anchors per full block and 20 T2T-stream samples per global batch.These are expectations, not additional empirical measurements.
  • Training Configuration: The final 35K checkpoint is evaluated without periodic validation, sample generation, or geometry-based checkpoint selection.
  • Parameter Selection: The reported operating point uses B=64, T=4, thresholds (0.95,0.9), guidance g=3.0, corruption range [0.45,0.95], and one rollout.Block size controls left-to-right stages, while T controls active-block refinement opportunities.
  • Parameter Selection: Increasing block size reduces latency but eventually weakens paired geometric fidelity, while additional denoising steps increase latency without monotonic geometry gains.These studies motivate B=64 and T=4 as the reported quality–latency balance.
  • Evaluation Protocol: Geometry evaluation normalizes meshes consistently, samples 8,192 surface points, and computes symmetric nearest-neighbor metrics.Reported metrics include CD-L1, NC, and F@1%.

C.5 Statistical Scope and Evaluation Boundaries

The evaluation uses paired prompt–target records with equal prompt weighting and reports geometry, CLIP alignment, and latency over the same 100 prompts. All main quality-table records are valid, so reported means are unaffected by missing-value handling or failure penalties.

  • 100 paired prompts are evaluated, with each method generating one shape per prompt under the same single-generation budget.
  • Geometry and CLIP metrics are macro-averaged over 100 ordered outputs, giving each prompt equal weight.
  • Latency is summarized over the same 100 prompts using mean, median, P90, and standard deviation.
  • All 100 records succeed for generation, geometry evaluation, and CLIP evaluation in every main quality-table row.No missing-value imputation or failure penalty affects the reported means.

D Qualitative Results and Prompts

The qualitative appendix separates Block3D’s independently generated mesh components from Blender’s scene assembly and documents the prompts, viewpoints, comparison panels, and reproducibility materials. These examples illustrate asset composability and additional text-to-shape coverage rather than complete scene generation or quantitative evaluation.

  • D Qualitative Results and Prompts: Figures D and E separate teaser presentation from generated components, while Figure G adds multiple viewpoints of individual Block3D meshes with complete prompts.Figure G broadens coverage to human figures, statues, accessories, manufactured objects, animals, and characters.
  • D.1 Teaser Prompts and Isolated Assets: The teaser arena is a Blender scene assembled from independently generated Block3D mesh outputs, not a direct multi-object generation result.Blender arranges the imported objects and applies shared scene, camera, and lighting settings.
  • D.1 Teaser Prompts and Isolated Assets: Each unique teaser component is generated once, while Blender duplicates, transforms, colors, and arranges meshes without modifying their geometry.
  • D.1 Teaser Prompts and Isolated Assets: The appendix lists component prompts and identifies repeated ranges as scene instances reusing the same generated mesh.Multiplicity and placement are determined during scene assembly.
  • D.1 Teaser Prompts and Isolated Assets: The main qualitative comparison uses complete prompts and renders Block3D alongside Cube, TRELLIS-text, AR3D-R1, and ShapeLLM-Omni from front and back views.Red boxes mark missing or distorted local geometry as presentation annotations rather than additional measurements.
  • D.1 Teaser Prompts and Isolated Assets: The five qualitative prompts cover an articulated character, animal, bust, architecture, and manufactured object, but are excluded from the random100 quantitative evaluation.The prompt list and camera normalization are shared across methods.
  • E.1 Archive Contents and Implementation Map: The archive provides training, inference, configuration, and evaluation materials but does not redistribute checkpoints, tokenizer weights, baseline repositories, generated meshes, or TRELLIS-500K assets.The README specifies the required Cube release and distinguishes reported initialization from random initialization.
  • E.2 Training and Inference Entry Points: The reported text-to-mesh protocol uses one native decoded mesh per prompt with B = 64, T = 4, g = 3.0, and optional postprocessing disabled.The 100-prompt manifest is evaluated by invoking the same single-prompt entry point once per record.

E.3 Data, Evaluation, and Environment Records

The reproducibility materials define the data split, evaluation outputs, and environment records for the reported experiments. They fix the evaluation manifest, document deterministic evaluation settings, and record software, hardware, and run metadata.

  • The split script fixes a 100-record evaluation manifest, excludes those assets from the TRELLIS-500K source pool, and selects 300K training records with seed 42.It stops on missing records or overlap and writes manifests, exclusions, and a split summary.
  • The evaluation script writes per-sample JSONL/CSV records and an aggregate summary using 8,192 surface samples, fixed seeds, F@1% target normalization, and eight CLIP renders.Invalid or missing outputs are recorded rather than regenerated.
  • The environment requires Python 3.10 and PyTorch 2.2.2 or later with CUDA, while PyTorch3D is required for the paper-aligned CLIP renderer.Each training launch stores configuration, command, seed, revision, platform, resource, software, and GPU metadata.
Loading 2608.19567v3…