PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation

Yiwen Tang, Zoey Guo, Kaixin Zhu, Ray Zhang, Qizhi Chen, Dongzhi Jiang, Junli Liu, Bohan Zeng, Haoming Song, Delin Qu, Tianyi Bai, Dan Xu, Wentao Zhang, Bin Zhao

arXiv:2512.10949v1cs.CVcs.AIcs.CL

TL;DR

RL remains underexplored for text-to-3D generation, where coupled geometry and texture make reward and algorithm choices sensitive. The paper systematically studies RL, introduces MME-3DR and Hi-GRPO, and develops AR3D-R1. The model demonstrates superior performance on MME-3DR and existing benchmarks, with improvements in geometry consistency and texture quality.

  • Problem

    RL strategies for text-to-3D remain underexplored, despite 3D assets’ coupled geometry and texture making training sensitive to reward design and algorithm choices.

    Source: §1 · p. 1

  • Method

    The paper systematically studies RL for text-to-3D, introduces the MME-3DR benchmark and hierarchical Hi-GRPO, and develops AR3D-R1.

    Source: §1 · p. 3

  • Results

    AR3D-R1 demonstrates superior performance on MME-3DR and existing benchmarks, with significant improvements in geometry consistency and texture quality.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The study provides insights into RL-driven 3D generation through its analysis of rewards, algorithms, benchmarks, and hierarchical generation.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reinforcement learning (RL), earlier proven to be effective in large language and multi-modal models, has been successfully extended to enhance 2D image generation recently. However, applying RL to 3D generation remains largely unexplored due to the higher spatial complexity of 3D objects, which require globally consistent geometry and fine-grained local textures. This makes 3D generation significantly sensitive to reward designs and RL algorithms. To address these challenges, we conduct the first systematic study of RL for text-to-3D autoregressive generation across several dimensions. (1) Reward designs: We evaluate reward dimensions and model choices, showing that alignment with human preference is crucial, and that general multi-modal models provide robust signal for 3D attributes. (2) RL algorithms: We study GRPO variants, highlighting the effectiveness of token-level optimization, and further investigate the scaling of training data and iterations. (3) Text-to-3D Benchmarks: Since existing benchmarks fail to measure implicit reasoning abilities in 3D generation models, we introduce MME-3DR. (4) Advanced RL paradigms: Motivated by the natural hierarchy of 3D generation, we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles. Based on these insights, we develop AR3D-R1, the first RL-enhanced text-to-3D model, expert from coarse shape to texture refinement. We hope this study provides insights into RL-driven reasoning for 3D generation. Code is released at https://github.com/Ivan-Tang-3D/3DGen-R1.

1. Introduction

RL for text-to-3D remains underexplored, and 3D assets’ coupled geometry and texture make reward design and algorithm choices especially sensitive. The paper studies these choices, introduces the MME-3DR reasoning benchmark and Hi-GRPO, and develops AR3D-R1.

1. Introduction

  • The study finds that human-preference alignment is crucial, while prompt consistency and 3D aesthetic priors further improve generation quality.General LMMs show robust signals for 3D-relevant attributes, although exclusive reliance on them for task-specific evaluation introduces systematic bias.Source: §1 · p. 2
  • Token-level loss averaging benefits 3D autoregressive generation more than sequence-level operations, and simple DAPO dynamic sampling stabilizes training.Data scaling improves performance, whereas iteration scaling requires careful calibration.Source: §1 · p. 2
  • MME-3DR addresses benchmarks that overestimate models by testing implicit reasoning across 249 annotated objects in five challenging categories.Experiments on several text-to-3D models validate the benchmark’s effectiveness.Source: §1 · p. 2
  • MME-3DR exposes failures across five reasoning-intensive categories and reports substantial post-RL improvements across all five.The categories include spatial and structural geometry, mechanical affordances, biological and organic shapes, rare objects requiring world knowledge, and stylized representation.Source: §1 · pp. 2–3
  • Hi-GRPO jointly optimizes global geometry and local texture through a coarse-shape stage followed by texture refinement, using dedicated reward ensembles.The first stage plans global structure and produces a coarse shape; the second uses its reasoning and the original prompt to generate a refined object.Source: §1 · p. 3
  • AR3D-R1 shows coarse-to-fine generation, strong reasoning on MME-3DR, and benchmark performance that outperforms Trellis.Its inference progresses from rough shapes to detailed textures, aligning with the Hi-GRPO training procedure.Source: §1 · p. 3
  • The paper presents the first systematic study of RL for text-to-3D autoregressive generation, examining reward design, algorithms, benchmarks, and RL paradigms.Its framework builds on GRPO and ShapeLLM-Omni and uses textual reasoning to guide token-level 3D generation.Source: §1 · pp. 1, 3
  • The study identifies 3D generation’s hierarchy from coherent geometry to texture-refined objects and proposes Hi-GRPO to optimize both steps in one iteration.It develops AR3D-R1 as an RL-enhanced text-to-3D autoregressive model.Source: §1 · p. 3

2. Related Work

Prior work extends RL from language and multimodal reasoning to 2D generation, while RL for autoregressive 3D remains comparatively unexplored. The paper positions its study around reasoning-guided generation with ShapeLLM-Omni and GRPO.

2.1. RL for LLM

  • RL has supported reasoning in language and multimodal models, including mathematical problem-solving and code generation.The cited work describes Chain-of-Thought reasoning with reward-guided correctness and format compliance.Source: §2.1 · p. 3

2.2. RL for 2D Generation

  • Text-to-image and diffusion or flow-generation studies apply RL to improve alignment, aesthetics, consistency, controllability, or stability.The cited autoregressive image approaches include semantic planning and token-level generation, while diffusion and flow approaches use stepwise or objective-coupled strategies.Source: §2.2 · p. 3

2.3. Text-to-3D Generation

  • Autoregressive 3D models tokenize objects, but RL remains underexplored amid greater spatial complexity and stricter geometry and detail constraints.ShapeLLM-Omni combines a multimodal LLM with a 3D VQVAE for discrete-token generation and understanding.Source: §2.3 · pp. 3–4

3. Preliminary

  • The preliminary setup uses ShapeLLM-Omni with GRPO, generating textual descriptions before conditioned 3D objects to investigate reasoning-guided training.The training prompts comprise 8,400 captions, and the Toys4K test set contains 800 randomly selected samples.Source: §3 · pp. 4–5

4. Impact of Different Reward Models

Reward studies identify human preference as a core signal, while algorithm studies favor token-level optimization and show that scaling requires calibration.

4. Impact of Different Reward Models

  • HPS V2.1 provides the strongest single-reward gains, and pairing it with UnifiedReward or Qwen2.5-VL adds up to 0.6 performance improvement.Specialized models are more robust for a given reward dimension, while LMMs generalize better for multi-view 3D consistency; Qwen2.5-VL consistency scoring improves CLIP score by 0.6.Source: §4 · pp. 5–6

5. Impact of Different RL Algorithms

  • Token-level averaging yields larger gains than sequence-level importance sampling and clipping under identical reward settings.The comparison evaluates GRPO, DAPO, and GSPO for autoregressive 3D generation.Source: §5 · p. 6
  • Dynamic Sampling improves vanilla GRPO by 0.6, whereas removing the KL penalty causes a 0.4 drop.Decoupled Clip still yields gains by encouraging exploration of low-probability tokens.Source: §5 · p. 6
  • Data expansions of 1.5×, 2×, and 3× yield gains of 0.4, 0.2, and 0.4, respectively; doubling iterations gains 0.9, but tripling them reduces performance.The authors associate the decline with generalization deterioration, likely from overfitting on preference features.Source: Figure 3 · p. 5, §5 · p. 6

6. Exploration of Text-to-3D Benchmarks

MME-3DR targets implicit reasoning in text-to-3D generation, and the paper reports improvements from RL across its reasoning categories. Hi-GRPO organizes generation from global structure to local refinement, with AR3D-R1 achieving the best reported benchmark results.

6. Exploration of Text-to-3D Benchmarks

  • MME-3DR is a 249-object benchmark spanning five reasoning categories, with objects excluded from prior text-to-3D model training and current RL training.Its categories cover spatial structures, mechanical components, biological or dynamic forms, rare concepts, and stylized objects.Source: §6 · p. 7
  • RL improves ShapeLLM-Omni by 5–6 points overall across MME-3DR categories, with notable gains in stylized representation.The cited analysis says models struggle with three categories while performing reasonably on mechanical and non-rigid biological objects.Source: §6 · p. 7

7. Exploration of RL Paradigms

  • Hi-GRPO separates each iteration into global semantic planning and coarse-shape generation, followed by visual reasoning and local appearance refinement.Its reward design includes ensembles tailored to the steps and dimensions of 3D quality assessment.Source: §7 · pp. 8–9
  • AR3D-R1 achieves optimal results on MME-3DR and sampled Toys4K, with improved cross-dataset robustness over other text-to-3D models.The authors also report high-quality meshes across animal and mechanical categories.Source: Table 4 · p. 9, Figure 8 · p. 9, §7 · p. 10
  • AR3D-R1 progresses from rough shapes to refined objects, adding local colors, textures, and part structures as generation proceeds.The authors describe this coarse-to-fine progression for truck, helmet, and computer examples.Source: Figure 9 · p. 9, §7 · p. 10

8. Conclusion

The paper presents a systematic study of RL for autoregressive text-to-3D generation and introduces Hi-GRPO and AR3D-R1. It reports superior performance on MME-3DR and existing benchmarks, with improvements in geometry consistency and texture quality.

8. Conclusion

  • The study examines reward design, RL algorithms, benchmarks, and RL paradigms for text-to-3D autoregressive generation.Hi-GRPO uses dedicated reward ensembles to optimize generation from coarse shapes to fine textures.Source: §8 · p. 10

Contributions of Co-first Authors

The listed co-first-author contributions center on experiments and manuscript work, while Ray Zhang led the project and its experimental design. The appendix overview identifies experimental settings, Hi-GRPO details, ablations, and additional visualizations.

Contributions of Co-first Authors

  • Ray Zhang led the project, proposing the idea and overseeing experimental design and manuscript planning.Yiwen Tang conducted experiments and primary manuscript writing; Zoey Guo and Kaixin Zhu contributed to experiments and manuscript refinement.Source: Contributions of Co-first Authors · p. 10

Overview

  • The appendix is organized around experimental settings, Hi-GRPO details, ablations, and additional visualizations.Source: Overview · p. 11

A. Experimental Settings

  • Training prompts came from Objaverse-XL, HSSD, and ABO, and evaluation used Toys4K.The listed datasets span large 3D-object collections, indoor scenes, and household products; Toys4K contains about 4,000 instances across around 105 categories.Source: A. Experimental Settings · p. 11

B.1. Two-Step Generation Process

Generation proceeds from semantic planning and coarse geometry to visual reasoning and refined geometry. The two stages use viewpoint-based human-preference and unified-reward evaluations.

B.1. Two-Step Generation Process

  • Generation first produces semantic reasoning for global geometry and coarse 3D tokens, then visual reasoning for local details and refined 3D tokens.The coarse and refined outputs are decoded into meshes.Source: §B.1 · p. 11

B.2.1. Human Preference Model

  • HPS V2.1 scores text-image similarity across six uniformly distributed viewpoints, taking the maximum as the human-preference reward.The reward range is [0, 1].Source: §B.2.1 · p. 11

B.2.2. Unified Reward Model

  • UnifiedReward scores coarse-shape prompt alignment, while UnifiedReward-2.0 evaluates textured objects for alignment, logical coherence, and style appeal.The coarse-shape score takes the maximum across six viewpoints and ranges from 1 to 5; the textured-object score sums three dimensions across viewpoints, with range [3, 15].Source: §B.2.2 · p. 11

B.2.3. 2D Large Multi-modal Model

The 2D multimodal reward evaluates category match and cross-view appearance, while the 3D reward assesses components directly from point clouds. The authors motivate the latter by limits in detecting 3D components from multi-view observations.

B.2.3. 2D Large Multi-modal Model

  • Qwen2.5-VL-7B evaluates category match across six viewpoints and scores color smoothness, material realism and coherence, and texture rationality.The category reward ranges over {0, 1}; the three appearance dimensions together range from [0, 3].Source: §B.2.3 · p. 11

B.2.4. 3D Large Multi-modal Model

  • Because 2D LMMs struggle to detect 3D components from multi-view observations, the method assesses completeness directly from 3D point clouds.Source: §B.2.4 · p. 11
  • The refined mesh is sampled into a dense point cloud, then ShapeLLM evaluates each prompted component for existence and geometric completeness, shape correctness, and quantity matching.Scores are averaged across components, with a reward range of [0, 2].Source: §B.2.4 · pp. 11–12

B.2.5. Dimension-Normalized Reward Ensemble

The reward ensemble normalizes reward contributions by evaluation dimensions and routes final-stage reward to the high-level step through λ. Loss computation is stage-specific and includes asymmetric clipping and token-level normalization.

B.2.5. Dimension-Normalized Reward Ensemble

  • Reward contributions are normalized by evaluation dimensions to limit domination by simple summation and remain stable when rewards are added or removed.Source: §B.2.5 · p. 12
  • With λ = 1.0, final-output quality directly supervises the high-level generation step.Step 2 rewards are backpropagated to step 1 through λ.Source: §B.2.5 · p. 12

B.3. Loss Computation

  • Each stage computes token-level log probabilities by concatenating reasoning-token and mesh-token probabilities, and the loss is computed separately for stages 1 and 2.The loss uses asymmetric clipping; a higher threshold gives low-probability tokens more room to increase probability, promoting exploration and preventing entropy collapse.Source: §B.3 · pp. 12–13
  • Token-level averaging normalizes the loss by valid-token count, and the objective includes token-level KL regularization.Source: §B.3 · p. 13

C.1. Reward Analysis.

Hi-GRPO combines global-to-local generation with step-specific reward ensembles and improves geometry, fine-grained textures, and prompt alignment. Reward analysis also finds gains from assigning rewards by generation step.

C.1. Reward Analysis.

  • Step-2 rewards from refined objects struggle to optimize coarse geometry and fine texture together, while combining RHPM_2 yields only marginal improvements.Reward selection and combination are evaluated for AR3D-R1.Source: §C.1 · p. 13
  • The passage reports CLIP scores improving by 2.1 point and says component-level rewards prove critical for correct part positioning, quantity, and structural plausibility.Source: §C.1 · p. 14

C.2. Effectiveness of Hi-GRPO.

  • Hi-GRPO’s hierarchical RL paradigm achieves substantial improvements across geometry, fine-grained textures, and prompt alignment.It combines global-to-local generation with step-specific reward ensembles.Source: §C.2 · p. 14
  • 0.9-point CLIP score improvement comes from textual reasoning-guided GRPO over direct 3D token optimization; textual reasoning also enables effective global planning for large and small objects.Both approaches use the HPSV2.1+UnifiedReward+Qwen2.5-VL reward system.Source: §C.2 · p. 14

D. Additional Visualizations

Loading 2512.10949v1…