PaperlayerView exact revision on arXiv

Source-linked AI summary

AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer

Lingting Zhu, Shengju Qian, Haidi Fan, Jiayu Dong, Zhenchao Jin, Siwei Zhou, Gen Dong, Xin Wang, Lequan Yu

arXiv:2602.12100v1cs.CV

TL;DR

Modular 3D asset generation needs high-quality, diverse outputs for professional and UGC use, but existing workflows and data availability are limited. AssetFormer models modular assets as primitive sequences with autoregressive Transformers, adapting tokenization and decoding; the paper reports promising generation quality, including a combined-data FID of 55.186, while noting scope limitations around text-only input and fixed vocabularies.

  • Problem

    High-quality, diverse modular 3D asset generation is needed for professional development and UGC, but existing methods face application limitations and modular-asset training data is scarce.

  • Method

    AssetFormer represents modular assets as primitive sequences and adapts autoregressive Transformer tokenization, ordering, and decoding strategies for text-conditioned generation.

  • Results

    FID 55.186 is achieved when synthesized and user-created data are combined, while token re-ordering and SlowFast decoding improve generation quality and efficiency without sacrificing performance.

  • Takeaways & Limitations

    AssetFormer provides a flexible framework for high-fidelity modular 3D asset generation and offers practical guidance for professional and UGC applications.

  • Takeaways & Limitations

    AssetFormer accepts only text input, while image-based conditioning remains unexplored and fixed discrete vocabularies require additional design for varying design spaces.

Abstract

from arXiv · show

The digital industry demands high-quality, diverse modular 3D assets, especially for user-generated content~(UGC). In this work, we introduce AssetFormer, an autoregressive Transformer-based model designed to generate modular 3D assets from textual descriptions. Our pilot study leverages real-world modular assets collected from online platforms. AssetFormer tackles the challenge of creating assets composed of primitives that adhere to constrained design parameters for various applications. By innovatively adapting module sequencing and decoding techniques inspired by language models, our approach enhances asset generation quality through autoregressive modeling. Initial results indicate the effectiveness of AssetFormer in streamlining asset creation for professional development and UGC scenarios. This work presents a flexible framework extendable to various types of modular 3D assets, contributing to the broader field of 3D content generation. The code is available at https://github.com/Advocate99/AssetFormer.

1 INTRODUCTION

The introduction motivates modular 3D asset generation by highlighting the limitations of existing representations, scarce public modular-asset data, and the need for text-customized generation. AssetFormer addresses this need with autoregressive modeling tailored to primitive sequencing, tokenization, and decoding.

  • Motivation: Traditional 3D representations face limitations in real-world applications despite progress in geometry and texture, particularly against professional game-development quality standards.The cited passage also notes that artists may face time-intensive workflows.
  • Modular representation: Modular assets combine primitives with attributes such as orientation and position, enabling complex forms to be assembled and rendered through configurations.The modular approach is associated with rapid prototyping, consistency, and game-engine integration.
  • Data gap: Public training data for modular 3D assets is scarce because many asset libraries are proprietary, so the study uses modules and data collected from an online UGC platform.The contribution includes a large-scale dataset collected and cleaned from an online game platform.
  • Research goal: The study aims to generate diverse modular 3D assets customized from textual descriptions.This goal targets both complex asset creation from basic components and broader participation in creation.
  • Approach: AssetFormer treats each asset as an ordered series of modules with placement and orientation decisions, adapting autoregressive modeling to 3D structure generation.The framework designs tokenization and decoding strategies using primitive connectivity to capture hierarchical and spatial relationships.
  • Contributions: The contributions include an autoregressive framework, a real-world modular-asset dataset, analysis of tokenization order and decoding, and high-quality contextually appropriate asset generation.The authors describe these contributions as relevant to professional development and UGC scenarios.

2 RELATED WORK

Related work covers generative visual modeling, 3D generation, and autoregressive modeling. It positions AssetFormer within autoregressive Transformer approaches while emphasizing the distinct tokenization and sequence-ordering challenges of 3D assets.

  • Generative Visual Modeling: Generative visual modeling includes GANs, autoregressive models, flows, and VAEs applied to image, video, and 3D creation.The related work cites multimodal autoregressive Transformers as examples of visual generation systems.
  • Autoregressive Modeling: AssetFormer applies autoregressive Transformer design rationales specifically to modular 3D generation.Its focus is the downstream visual-creation setting of modular 3D assets.
  • 3D Generation: Recent 3D-generation methods create complex representations from text or sparse images using voxels, point clouds, neural fields, and meshes.The passage situates autoregressive Transformer-based mesh generation among these representation choices.
  • Autoregressive Modeling: Autoregressive Transformers offer scalability and complex-dependency modeling, but adapting them to visual and 3D domains raises tokenization and sequence-ordering challenges.These challenges are central to applying next-token prediction beyond language.

3 METHOD

AssetFormer represents user-created modular assets as ordered primitive tokens and generates them autoregressively from text. The method combines discrete attribute modeling, spatially informed token ordering, constrained decoding, and faster sampling for controllable asset creation.

  • 3.1 PROBLEM FORMULATION: Each asset is modeled as a sequence of primitives with discrete class, rotation, and 3D position attributes.The generative model maps a textual description to a set of primitive tuples.
  • 3.1 PROBLEM FORMULATION: The dataset combines complex, varied real user-created assets with an additional procedurally generated source for studying data-source effects.Real assets come from an online UGC platform, while PCG randomizes attributes such as storeys and key-module positions.
  • 3.2 AUTOREGRESSIVE TRANSFORMER MODELING: A decoder-only Transformer predicts the next token using cross-entropy loss, with text features projected into pre-filled token sequences.AssetFormer uses a Llama backbone, custom vocabularies, and one-dimensional rotary positional embeddings.
  • 3.2 AUTOREGRESSIVE TRANSFORMER MODELING: Attribute-specific vocabularies represent primitive classes, rotations, and coordinates, while inference filters logits to preserve valid token-type transitions.The joint vocabulary includes an end-of-sequence token, and decoding re-normalizes the remaining valid distribution.
  • 3.2 AUTOREGRESSIVE TRANSFORMER MODELING: DFS and BFS traversal impose spatially connected module orders; empirically, DFS performs slightly better and supports training without constraining rendering deployment.The traversal starts from a lower asset corner and produces a permutation of primitive indices.
  • 3.3 AUTOREGRESSIVE TRANSFORMER DECODING: Top-k sampling balances generated-asset quality and diversity, while SlowFast decoding uses smaller and larger models for easy and difficult tokens.The approach adapts speculative decoding and is intended to accelerate autoregressive decoding without compromising quality, with minimal additional training.

4 EXPERIMENTS

Experiments evaluate AssetFormer’s dataset construction, baselines, token ordering, data sources, and SlowFast decoding for modular 3D asset generation.

  • Implementation Details: The dataset combines procedurally synthesized and real user-created assets, maps them to 25 primitives, and filters samples using GPT-4o queries and manual review.
  • Implementation Details: AssetFormer uses GPT-4o-generated phrase bundles for text control and a joint vocabulary of 214 tokens covering primitives, rotations, and positions.
  • Comparison with the Baselines: Top-k sampling outperforms greedy and beam search, while the PCG baseline covers less of the data distribution and produces less detailed structures.
  • Ablation Study on Token Orders: DFS and BFS primitive token re-ordering outperform learning sequences in raw order, with raw ordering producing isolated parts and visible artifacts.
  • Ablation Study on Data Sources: Combining procedural and collected data yields an FID of 55.186, compared with 63.381 for collected data alone and 113.560 for synthesized data alone.
  • Analysis on SlowFast Decoding: SlowFast decoding accelerates generation without sacrificing performance by assigning simple primitives to a draft model and complex elements to a larger model.

5 CONCLUSION

AssetFormer is an autoregressive Transformer framework for modular 3D asset generation, with text-only conditioning and fixed discrete vocabularies defining current scope boundaries.

  • 5 CONCLUSION: AssetFormer models modular 3D assets from primitives using an autoregressive Transformer framework.The framework adapts token sequencing and decoding techniques inspired by language models for asset generation.
  • 5 CONCLUSION: The current system accepts only text input, leaving image-based conditioning unexplored.The authors explicitly identify image-based conditioning as uncertain and unexplored.
  • 5 CONCLUSION: AssetFormer relies on fixed discrete vocabularies, requiring additional design considerations for varying design spaces.

A.1 USER STUDY

A six-person user study compared ground-truth, PCG-generated, and AssetFormer building samples across four criteria using 1–5 ratings.

  • A.1 USER STUDY: The user study involved 6 participants aged between 22 and 28 years.
  • A.1 USER STUDY: Participants graded building samples for compactness, diversity, aesthetic, and complexity.
  • A.1 USER STUDY: Each grading batch contained six building samples and included ground truth, PCG-generated, and AssetFormer synthetic data.
  • A.1 USER STUDY: Ratings were reported on a scale of 1–5.

A.2 COMPARISON WITH MESHGPT

The MeshGPT comparison evaluates modular and mesh representations while showing that AssetFormer’s compact generated structures preserve detailed internal geometry and support textured runtime integration.

  • A.2 COMPARISON WITH MESHGPT: AssetFormer outputs can be converted to triangle meshes for comparison with MeshGPT.The comparison converts building data to triangle meshes and extracts vertex and face information for MeshGPT training.
  • A.2 COMPARISON WITH MESHGPT: Transparent renderings and inside viewpoints show compact arrangements of primitives in AssetFormer-generated assets.
  • A.2 COMPARISON WITH MESHGPT: Modular representation is described as lossless, ready for game engines, and directly usable in applications.
  • A.2 COMPARISON WITH MESHGPT: X-Ray visualizations reveal intricate internal building structures in generated samples.
  • A.2 COMPARISON WITH MESHGPT: Generated primitives can be mapped to diverse textured modules with different levels of detail and integrated into engine runtime.The modular representation also supports procedural and generative texture rendering techniques.
  • A.2 COMPARISON WITH MESHGPT: Mesh representation can require extremely long token sequences and post-processing to merge close points, whereas modules are compactly connected.

A.3 ALGORITHM DESCRIPTION

The algorithm section contrasts procedural building synthesis with AssetFormer’s accelerated autoregressive decoding and illustrates generated structures and textured modular outputs.

  • A.3 ALGORITHM DESCRIPTION: Procedural Content Generation randomly sets width, length, and floor height before assembling walls, floors, roofs, and components from primitives.The method is effective for simple buildings but struggles with varied complex buildings and intricate building distributions.
  • A.3 ALGORITHM DESCRIPTION: Transparent visualizations highlight complex and compact internal structures in generated buildings.
  • A.3 ALGORITHM DESCRIPTION: Generated assets support textured module mapping across different levels of detail and viewpoints.The assets are described as seamlessly integrable into engine runtime.
  • A.3 ALGORITHM DESCRIPTION: The decoding algorithm takes a text prompt, lookahead K, and target sequence length T as inputs.
  • A.3 ALGORITHM DESCRIPTION: SlowFast Decoding uses a draft model to propose tokens and a target model to compute logits for token acceptance or rejection.The approach is adapted from speculative decoding with models of different sizes.

A.4 EMERGENT EDITING CAPABILITIES

AssetFormer demonstrates emergent editing of modular buildings without task-specific training, completing missing primitives and extending structures with roof components.

  • The editing task extends existing modular structures and incorporates roof components.The application is demonstrated using modular building representations and DFS-based token reordering before prediction.
  • The model’s pre-training on text-to-modular building data captures structural constraints and semantic relationships in modular architectures.
  • Without further training, AssetFormer edits modular buildings by completing missing target primitives in a sequence inpainting setup.The model was not explicitly trained for editing or exposed to the inpainting task’s distinct token-order patterns.

A.5 DIVERSITY FOR THE SAME PROMPT

AssetFormer generates diverse modular assets from identical text prompts through Transformer sampling, with Figure 10 showing cases for two different prompts.

  • Figure 10 presents two rows corresponding to two different prompts while showcasing generated results with the same prompts.
  • The same prompts produce diverse generated asset samples through Transformer sampling.

A.6 MORE ASSETS: GALLERY IN UNREAL ENGINE

The paper presents additional generated assets, their modular components, prompt construction, and integration into Unreal Engine. The materials include visual galleries and statistics comparing PCG and collected real data.

  • Figure 11 showcases an asset gallery in Unreal Engine for editing and assembling cohesive production-ready collections.
  • Generated modular assets support texture mapping, optional geometry mapping, and adjustable lighting for integration into Unreal Engine.The authors state that this integration directly supports real-world 3D content production workflows.
  • GPT-4o generates building descriptions from rendered images using curated phrases for building type, height, and precise visual features.The prompt format includes optional feature phrases such as roof type and window quantity.
  • The dataset includes roof, wall, and other component primitives, displayed in three columns.
  • Table 7 documents the prompt used to query GPT-4o for building descriptions.
  • Primitive and text-phrase distributions are compared between PCG data and collected real data.The text statistics cover first, second, and remaining phrases used in the dataset.
Loading 2602.12100v1…