Octrees as an Explicit 3D Language

• Randomly emptying some second-to-last-level octree nodes and omitting their descendants yields a shorter S-Octree while retaining coordinate- and depth-anchored occupancy tokens.The representation preserves locality and hierarchy for generating and interpreting shapes.Source: §1 · p. 2
• Full-rank 3D branches handle mesh and position-query tokens while text and image tokens retain the frozen vision-language pathway, with shared self-attention connecting the streams.Separate mesh embeddings and an output head keep occupancy prediction distinct from the language vocabulary.Source: §1 · p. 2
• OctLLM uses position-aware masks for generation and direct S-Octree input for 3D understanding.Source: §1 · p. 2
• OctLLM achieves state-of-the-art 3D generation and understanding among unified multimodal LLMs while preserving the backbone’s language ability.Source: §1 · p. 2

Unified Multimodal Generation and Understanding

• Unified multimodal systems extend next-token prediction across modalities in 2D, while reviewed 3D systems use tokens or encoders that do not preserve explicit hierarchical structure.The cited 3D approaches include VQVAE indices (Yin et al., 2023), continuous point features, and coordinate text (Wang et al., 2024c).Source: §2 · pp. 2–3

Autoregressive 3D Generation and Octree Representations

• Autoregressive 3D work serializes meshes or discretizes space with octrees, while continuous-latent generators do not provide a language an LLM can read and write.OctLLM extends octree representations to a general-purpose LLM while keeping token coordinates and depths explicit.Source: §2 · p. 3

3D Understanding and Language-Preserving Adaptation

• 3D understanding research connects encoders or point tokens to LLMs, while modality-decoupled adaptation routes modalities through separate weights and shared self-attention.OctLLM brings this adaptation principle to 3D with full-rank branches that own their embeddings and output head.Source: §2 · p. 3

3.1 OCTREE-BASED 3D TOKENIZATION

• Z-order octree occupancy is packed eight bits per byte-level mesh token, shortening the raw binary sequence eightfold while preserving discrete voxel structure.Source: §3.1 · p. 3
• S-Octree randomly empties second-to-last-level nodes and drops finest descendants, trading omitted occupancy for shorter sequences that completion later restores.Source: §3.1 · pp. 3–4

3.3 SPARSE VOXEL COMPLETION NETWORK

• A 3D U-Net completes partial S-Octree occupancy, then occupied voxels condition a flow-based decoder that produces the mesh.Source: Figure 2 · p. 4

3.2 3D-AWARE DUAL-STREAM LANGUAGE MODEL

• Mesh bytes and position-query masks use trainable 3D branches, while text and image tokens retain frozen transformations and all streams share self-attention.Source: §3.2 · pp. 4–5
• 3D coordinates and depth embeddings locate octree tokens at their proper scale; predictive masks query target locations while prior masks are suppressed as keys.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

3.4 TRAINING OBJECTIVES

• Training mixes generation and understanding: position-aware masks query mesh bytes for generation, while S-Octree inputs supervise natural-language responses using separate word and mesh vocabularies.Source: §3.4 · p. 6

4.1 IMPLEMENTATION DETAILS

• Training uses 195K assets and 585K instructions, with a separately trained completion network; 3D branches add about 2.80B trainable parameters.The S-Octree has six levels and drops second-to-last-level nodes with probability 0.5; the backbone and vision encoder are frozen.Source: §4.1 · pp. 6–7

4.2 QUANTITATIVE COMPARISON

• OctLLM outperforms multimodal LLMs on image-conditioned generation and leads them on every PointLLM-200 understanding metric, while text-conditioned generation results vary by metric.TRELLIS attains the best text-conditioned generation scores overall; OctLLM ranks second on most metrics.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
• OctLLM reproduces the backbone exactly on both multiple-choice benchmarks and differs by less than one point on GSM8K; masking mesh tokens at inference eliminates the residual IFEval drop.The text-only pathway remains frozen, and text tokens never enter a 3D branch.Source: §4.2 · pp. 8–9, Table 2 · p. 8

4.3 QUALITATIVE COMPARISON

• Across qualitative examples, OctLLM follows image-conditioned part layout and text-specified composition more closely than the compared multimodal LLMs; it also describes 3D inputs in more detail and with fewer errors.For image-conditioned generation it is visually closest to TRELLIS; on text prompts, LLaMA-Mesh returns no valid mesh for one example.Source: §4.3 · p. 9

5 CONCLUSION

• OctLLM's explicit, coordinate- and depth-anchored occupancy sequence supports leading generation and understanding among unified multimodal LLMs while reproducing the backbone on general language benchmarks.The conclusion contrasts this language preservation with 3D LLMs that update pretrained weights and lose a third to a half of their accuracy.Source: §5 · p. 9
• The authors report using generative AI for drafting, literature assistance, code implementation, caption generation, and automated evaluation, while stating it did not formulate the core idea or interpret results.They state that authors reviewed the AI-assisted text, references, code, and generated data.Source: AI USE STATEMENT · p. 10
• The authors state that the paper and appendix provide reproduction details and that source code, model weights, scripts, and workflow instructions are publicly released.The described materials cover representation, training, evaluation, preprocessing, rendering, and metric computation.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX – OCTREES AS AN EXPLICIT 3D LANGUAGE

• The supplement is organized around procedures, experimental setup, additional qualitative results, design ablations, and limitations and future directions.Sections A through E cover those respective topics.Source: APPENDIX – OCTREES AS AN EXPLICIT 3D… · p. 16

A TRAINING AND INFERENCE PROCEDURES

• The language model and completion network are optimized independently: the former learns the shared 3D language for generation and understanding, while the latter restores occupancy removed by sparsification.Here, c denotes a text or image condition, M a mesh, S its serialized S-Octree, and G and GS the complete and sparse occupancy grids.Source: A TRAINING AND INFERENCE PROCEDURES · p. 16

A.1 TRAINING PROCEDURE

• For training, each mesh's S-Octree is sparsified and serialized once; generation uses position-aware mask queries to supervise mesh bytes, while understanding supplies the S-Octree as input and supervises the text answer.The completion network separately learns from paired sparse and complete occupancy grids.Source: §A.1 · p. 16

A.2 GENERATION PROCEDURE

• At inference, the partial octree determines each next node in coarse-to-fine Z-order; OctLLM queries its position and depth, samples one of 256 occupancy bytes, and stops at the target depth.The completed sparse grid is passed through the completion network, thresholded at 0.5, and used as TRELLIS's structural condition alongside the original text or image as semantic condition.Source: §A.2 · p. 17, Figure S1 · p. 17

A.3 SERIALIZED OUTPUT

• The serialized output begins with a natural-language preamble and contains 1,023 occupancy-byte tokens between mesh boundary tokens in the illustrated example.The vocabulary form uses <mesh0> through <mesh255>; figures abbreviate the same tokens as O0 through O255.Source: §A.3 · p. 18

B.1 ADDITIONAL DATASET DETAILS

• Training meshes are centered, isotropically scaled, and converted to depth-6 O-CNN octrees; preprocessing excludes assets with fewer than 100 or more than 4,000 mesh-byte tokens.The listed ShapeNet categories are airplane, car, chair, rifle, and table.Source: §B.1 · p. 18

B.2 OPTIMIZATION DETAILS

• Table S1 lists optimization settings not specified in Section 4.Source: Table S1 · p. 19

B.3 RENDERING PROTOCOL

• Meshes are rendered at 1024 × 1024 from 24 camera directions with identical lighting and camera sampling across methods; untextured baselines receive materials from Hunyuan3D-2.1 before rendering.The camera directions follow a spherical Hammersley sequence.Source: §B.3 · p. 19

B.4 GENERATION METRICS

• FID compares empirical feature means and covariances, while KID is the unbiased squared MMD.Both metrics use Inception-V3 and DINOv2 ViT-L/14 register-token features; KID values are scaled by 100 in Table 1.Source: §B.4 · p. 19
• Text-to-3D CLIP averages caption similarity over views 014–017, while image-to-3D uses each asset’s best-matching view among 24 renders.The maximum is used because the condition image shows one unknown viewpoint; averaging all views would penalize unshown viewpoints.Source: §B.4 · p. 20

B.5 UNDERSTANDING EVALUATION

• Metrics are macro-averaged over matched assets and scaled by 100; judge scores average valid outputs, with minor exceptions for LLaMA-Mesh and PointLLM-7B.BLEU uses sentence-level method-1 smoothing, ROUGE reports F1, and Sentence-BERT and SimCSE use specified pretrained models; each method receives its native representation.Source: §B.5 · p. 20
• Table S2 adds PointLLM-13B and lexical metrics, but conclusions rely on semantic similarity and judge scores because lexical overlap can favor short generic captions.Figure S2 qualitatively contrasts terse or appearance-focused baselines with OctLLM descriptions that better cover visible parts and spatial arrangement.Source: §B.5 · p. 20, Figure S2 · p. 20

C ADDITIONAL RESULTS AND EVALUATION DETAILS

• Figures S3 and S4 provide additional image- and text-conditioned mesh results from multiple viewpoints.The complete-output subsection gives the fixed text-generation instruction, full condition strings, and unedited OctLLM descriptions.Source: C ADDITIONAL RESULTS AND EVALUATION… · p. 21, §C.1 · p. 21

C.2 JUDGE PROMPTS

• The reference-based judge scores identity, geometry, appearance, fine-grained attributes, and caption quality; a separate render-grounded judge rates visible-object faithfulness from four views.The prompts specify score formats and penalties for hallucinated, empty, or incorrect captions.Source: §C.2 · pp. 21–22

C.3 AUTOREGRESSIVE INFERENCE EFFICIENCY

• Timing uses sequential batch-size-one runs on one RTX 5090, averages all 200 outputs, and includes OctLLM’s constrained octree loop and structural checks.Input processing, reconstruction, and rendering are excluded.Source: §C.3 · p. 22
• Supplementary figures show additional image- and text-conditioned generated meshes from multiple viewpoints.The accompanying prompt examples include objects such as a guitar, car, dolphin, and ladder.Source: Figure S3 · p. 23, Figure S4 · p. 24, §C.3 · p. 24
• OctLLM averages 1,277.06 output tokens and 37.59 seconds per generation, with latency competitive among compared methods and faster than OctGPT and LLaMA-Mesh.Token counts use method-native output units and representation-specific structural tokens, so they do not share a vocabulary.Source: §C.3 · p. 25, Table S3 · p. 25

Explicit 3D Language versus Latent Codes

• S-Octree reduces image-to-3D FID by 20.5% and improves both understanding metrics at the same KID versus 1,024 ShapeLLM-Omni VQVAE tokens.Only the representation changes in this comparison, supporting the benefit of exposing spatial coordinates and hierarchy.Source: D ABLATION STUDY · p. 25, Table S4 · p. 25

Preserving Language Ability

• OctLLM matches the backbone on MMLU, HellaSwag, and IFEval and stays within 0.5 GSM8K points, while rank-64 LoRA loses 5.1 MMLU and 18.1 HellaSwag points.The LoRA control also uses UltraChat; OctLLM confines 3D-gradient updates to routed branches.Source: D ABLATION STUDY · p. 25, Table S5 · p. 26

Components of the S-Octree Pathway

• Removing predictive mask tokens causes the largest S-Octree component-ablation degradation; position embeddings and mask suppression add further gains.The components respectively specify predicted occupancy locations, ground queries in 3D, and prevent earlier placeholders from contaminating later context.Source: D ABLATION STUDY · p. 26, Table S6 · p. 26

Placement of Routed 3D Branches

• Uniform eight-branch placement reduces image- and text-conditioned FID by 22.7% and 17.8% versus the next-best placement, while remaining within 4.2 points of the best targeted placement on both semantic metrics.First- and last-layer placements favor understanding; uniform placement gives the best generation scores under both conditions.Source: D ABLATION STUDY · p. 26, Table S7 · p. 26

Scaling Routed 3D Capacity

• FID falls from 37.63 with four branches to 26.31 with eight, while KID improves to 0.80; the four-to-six-branch change has little FID effect.The variants use 0.93B, 1.40B, and 1.86B trainable parameters for four, six, and eight branches, respectively.Source: D ABLATION STUDY · p. 26, Figure S5 · p. 26

Full-Rank 3D Branches versus Token-Routed LoRA

• Full-rank branches outperform rank-256 token-routed LoRA on three of four generation metrics, including image-conditioned FID 26.31 versus 32.44 and KID 0.80 versus 1.06.On full training data, full-rank branches train stably, whereas high-rank LoRA suffers gradient explosions.Source: D ABLATION STUDY · p. 27, Table S8 · p. 27

S-Octree Efficiency

• S-Octree reduces peak memory from 58 GB to 44 GB and epoch time from 2.17 to 1.37 hours (1.6×) across five ShapeNet categories.The reduction brings training within a 48 GB accelerator budget; completion restores pruned fine-level occupancy.Source: D ABLATION STUDY · p. 27, Figure S7 · p. 28
• Random emptying preserves generation quality closely: image-conditioned FID/KID are 26.31/0.80 for S-Octree and 27.54/0.83 for full octrees.In this ablation, random emptying followed by completion reduces training time, memory use, and autoregressive decoding workload; the default bottlenecked U-Net with BCE achieves the best completion FID and KID.Source: D ABLATION STUDY · pp. 27–28, Figure S6 · p. 27, Table S9 · p. 28

E LIMITATIONS

• 1,793 tokens on average versus ShapeLLM-Omni’s fixed 1,024-token representation increases training time and GPU memory usage.The variable-length S-Octree allocates fewer tokens to simple shapes and more to complex ones; spatially adaptive refinement is proposed as a way to shorten sequences.Source: E LIMITATIONS · p. 28
• Errors in S-Octree generation can propagate or be amplified by 3D U-Net completion, producing holes and other artifacts that lower generation quality.Geometry passes through both learned predictions.Source: E LIMITATIONS · p. 28
• OctLLM does not support 3D understanding of appearance or 3D editing; future work plans to add native color and material modeling and editing.The authors also aim to reduce dependence on an external 3D generator.Source: E LIMITATIONS · p. 28