PaperlayerView exact revision on arXiv

Source-linked AI summary

OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generation

Bingyang Cui, Yujie Zhang, Yiling Xu, Yunfeng Guan

arXiv:2610.03423v1cs.CV

TL;DR

Multi-dimensional T23D RL must improve qualities such as alignment and texture, while existing joint reward optimization does not explicitly model dependencies among dimensions. OuroReward schedules rewards using a dependency-aware cycle and adaptive transitions, and AdaSelect selects capability-matched prompts. Experiments report consistent quality improvements across dimensions and better training stability.

  • Problem

    Existing T23D RL methods optimize multiple quality dimensions jointly without explicitly modeling their dependencies, despite the need to improve qualities such as semantic alignment and texture clarity.

    Source: §I · p. 1

  • Method

    OuroReward builds a dependency-aware cyclic reward order with headroom-based transitions, while AdaSelect selects informative prompts aligned with the model’s current capability.

    Source: §I · p. 2

  • Results

    Experiments across T23D models and RL algorithms consistently improve generation quality across dimensions and substantially enhance training stability.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework addresses both generation quality across dimensions and training stability in multi-dimensional T23D RL.

    Source: §V · p. 12

Abstract

from arXiv · show

Reinforcement learning (RL) for Text-to-3D (T23D) generation requires optimization across multiple quality dimensions such as semantic alignment and texture clarity. Existing methods typically optimize these dimensions simultaneously through multiple reward aggregation, without explicitly modeling inter-dimension dependencies. This can cause imbalanced optimization and persistent interference among conflicting dimensions. To address this limitation, we propose OuroReward, an interference-aware sequential reward scheduling strategy for T23D RL. OuroReward first estimates pairwise dependencies among dimensions and constructs a cyclic optimization path that minimizes cumulative interference. By incorporating the tail-to-head dependency, the cycle captures global compatibility across the entire schedule. Then, OuroReward converts the cycle into a one-pass sequence, and starts optimization from the dimension with the lowest aggregate interference. Rather than assigning a fixed optimization budget to each dimension-wise reward, training adaptively determines when to advance to the next reward according to the remaining optimization headroom of the current one. We further introduce AdaSelect, an adaptive prompt selection strategy that identifies reliable and informative prompts aligned with the model's current capability. By focusing policy updates on these prompts, AdaSelect effectively improves training stability. Extensive experiments across different T23D models and RL algorithms demonstrate that our framework consistently improves generation quality across multiple dimensions.

I. INTRODUCTION

The introduction frames T23D RL as a multi-dimensional optimization problem and proposes OuroReward for dependency-aware reward scheduling alongside AdaSelect for capability-matched prompt selection. The paper reports consistent generation-quality gains across dimensions and improved training stability, while the generative-model overview contrasts autoregressive and diffusion approaches.

I. INTRODUCTION

  • Joint reward aggregation can let readily optimized dimensions dominate, while reweighting and gradient reconciliation still optimize rewards within the same policy update.The introduction motivates separating competing rewards into sequential optimization stages.Source: §I · p. 1
  • OuroReward estimates pairwise dependencies, constructs a globally compatible cycle, and uses headroom-aware transitions in a one-pass reward sequence.The cycle includes the tail-to-head dependency, and optimization starts from the most compatible dimension.Source: §I · p. 2
  • AdaSelect selects informative prompts aligned with the model’s current capability to improve T23D RL training stability.It considers capability compatibility, rollout characteristics, and selection history.Source: §I · p. 2
  • Experiments across T23D models and RL algorithms consistently improve generation quality across dimensions and substantially enhance training stability.Source: §I · p. 2

A. T23D Generative Models

  • Autoregressive generators decode discretized 3D token sequences, whereas diffusion generators recover representations through iterative denoising; flow-matching feedforward models balance efficiency and fidelity.The overview notes that autoregressive tokenization can compromise fine geometry and increase decoding cost, while optimization-based diffusion can be computationally expensive.Source: A. T23D Generative Models · pp. 2–3

B. RL Training Strategy

Prior RL strategies address reward imbalance and gradient conflict through reweighting, normalization, or gradient-level coordination, but still optimize dimensions jointly. The methodology combines dependency-aware reward scheduling with adaptive prompt selection, which periodically ranks prompts for the current training stage.

B. RL Training Strategy

  • Diffusion RL methods include policy-gradient and preference-based approaches, while trajectory-level methods can incur substantial computation and memory overhead.DiffusionNFT and AWM reduce such overhead by changing how optimization uses the forward process or native score/flow-matching objective.Source: B. RL Training Strategy · p. 3
  • Prior multi-dimensional RL methods reweight rewards, normalize per-reward advantages, or coordinate gradients, yet continue optimizing dimensions simultaneously.The stated gap motivates a scheduling strategy that models inter-dimension dependencies and controls transitions between reward objectives.Source: B. RL Training Strategy · p. 3

III. METHODOLOGY

  • The framework periodically refreshes capability-aligned prompts and schedules dimension-wise rewards by estimated dependencies, switching rewards according to current optimization headroom.Its two components are AdaSelect for prompt selection and OuroReward for reward scheduling.Source: §III · pp. 3–4

A. AdaSelect: Adaptive Sample Selection

2) Efficient Rollout Exploration:

AdaSelect uses reduced-cost rollouts to score candidate prompts for their fit with current model capability, informative reward variation, optimization direction, and selection history. It prioritizes capability and discriminability, then periodically refreshes the active prompt subset as training evolves.

2) Efficient Rollout Exploration:

  • Rollout sampling steps, group size, rendered resolution, and viewpoints are each set to 50% of training settings, reducing cost while preserving key geometric and appearance characteristics.The resulting samples are intended to provide sufficient information for assessing candidate-prompt suitability.Source: 2) Efficient Rollout Exploration · p. 4

3) Prompt Score Computation:

  • Model capability scores how closely a prompt's aggregate reward representation matches the centroid of candidate-prompt rewards, indicating fit with the current curriculum stage.The prompt-level representation aggregates rewards over its rollouts, and distance to the centroid is transformed into a normalized score using a Gaussian kernel.Source: 3) Prompt Score Computation · pp. 4–5
  • Rollout discriminability scores prompts by average reward variation across dimensions, favoring prompts whose group rollouts provide comparative signals for policy optimization.Nearly identical rollout rewards provide weak comparative signals.Source: 3) Prompt Score Computation · p. 5
  • Direction consistency evaluates whether a prompt's optimization direction aligns with the desired multidimensional direction, distinguishing prompts with comparable aggregate rewards but different reward distributions.The desired direction is referenced to the current capability centroid.Source: 3) Prompt Score Computation · p. 5
  • Selection frequency favors prompts selected fewer times, encouraging broader training-prompt coverage rather than repeatedly using the same prompts.The score uses prior selection count, a scaling factor, and a decay rate.Source: 3) Prompt Score Computation · p. 5
  • Capability and discriminability are the primary score criteria, while direction consistency and selection frequency are auxiliary; the top-scoring prompts form an active subset that is periodically refreshed.AdaSelect rescoring refreshes the subset to track the model's evolving capability.Source: 3) Prompt Score Computation · pp. 5–6

B. OuroReward: Interference-Aware Sequential Reward Scheduling

OuroReward replaces simultaneous reward optimization with interference-aware sequential scheduling. It constructs a globally compatible cycle, starts from the most compatible dimension, and advances rewards according to optimization headroom.

B. OuroReward: Interference-Aware Sequential Reward Scheduling

  • OuroReward has two components: Multi-Dimensional Cycle Construction estimates directed dependencies and builds a compatibility-maximizing path, while Dependency-Guided Sequential Optimization schedules dimension-wise rewards.Dependencies are estimated using gradient cosine similarity and reward-change correlation.Source: B. OuroReward: Interference-Aware… · p. 6
  • The cycle includes the tail-to-head dependency, capturing global compatibility across the path rather than only compatibility between locally adjacent dimensions.Source: B. OuroReward: Interference-Aware… · p. 6
  • Optimization starts from the most compatible dimension, unrolls the cycle into a one-pass sequence, and adaptively switches reward objectives according to remaining optimization headroom.The sequential schedule is intended to mitigate cumulative inter-dimension interference and stabilize T23D RL training.Source: B. OuroReward: Interference-Aware… · p. 6

1) Multi-Dimensional Cycle Construction:

OuroReward estimates directed dependencies between reward dimensions and uses them to construct a cyclic optimization order. The cycle accounts for the tail-to-head link, favoring globally compatible paths over locally chosen transitions.

1) Multi-Dimensional Cycle Construction:

  • Directed dependency scores combine accumulated gradient cosine similarity with reward-change correlation measured across consecutive validation steps.Negative gradient similarity indicates potential gradient conflict.Source: 1) Multi-Dimensional Cycle Construction · pp. 6–7
  • Each candidate cycle accumulates dependency scores between adjacent dimensions, including the tail-to-head edge, and the method selects the highest-scoring path.A larger cycle score indicates stronger overall compatibility and lower cumulative interference.Source: 1) Multi-Dimensional Cycle Construction · p. 7
  • Evaluating the complete cyclic structure avoids greedy transition choices that may be locally optimal but globally suboptimal.Source: 1) Multi-Dimensional Cycle Construction · p. 7

2) Dependency-Guided Sequential Optimization:

OuroReward converts the dependency cycle into a one-pass reward sequence, then transitions between objectives according to the current reward’s optimization headroom. Training advances when the current objective stabilizes.

2) Dependency-Guided Sequential Optimization:

  • The cycle’s starting reward is the dimension with the highest aggregate dependency score to the others, resolving the cycle’s rotational ambiguity.Source: 2) Dependency-Guided Sequential… · p. 7
  • OuroReward forms the training reward by adaptively weighting the current dimension’s reward and the next reward along the sequence.The headroom estimate measures remaining optimization potential; rewards are normalized before it is computed.Source: 2) Dependency-Guided Sequential… · p. 7
  • When the adaptive coefficient changes by less than ϵ across consecutive steps, training advances to the next reward and eventually returns from the terminal reward to the start.The one-pass schedule is described as reducing persistent interference from simultaneous optimization.Source: 2) Dependency-Guided Sequential… · p. 7

C. Training Policy

The framework uses multidimensional reward models to guide prompt selection and reward scheduling, while leaving policy optimization to the chosen RL algorithm. It supports multiple evaluator types and RL algorithms.

C. Training Policy

  • Projection-based, model-based, and LLM-based evaluators each produce scalar rewards for specific quality dimensions.AdaSelect uses the rewards to assess prompt informativeness, while OuroReward uses them to estimate inter-dimension dependencies.Source: C. Training Policy · p. 7
  • Figure 5 summarizes prompt-dataset composition, corpus vocabulary, and prompt-length distribution.Its composition chart uses inner and outer circles for interaction types and attribute categories, respectively.Source: Figure 5 · p. 8
  • OuroReward controls reward scheduling independently of the policy optimizer and can be integrated with DiffusionNFT, GRPO, and AWM.Source: C. Training Policy · p. 8

A. Datasets and Evaluation Metrics

The experiments evaluate T23D models on a constructed prompt set using multiple evaluators and human preference. Across these evaluations, AdaSelect and OuroReward improve generation quality and training stability.

A. Datasets and Evaluation Metrics

  • The test set contains 1,000 prompts, and evaluation averages eight generated rollouts per prompt across projection-, model-, and LLM-based evaluators.A user study samples 300 prompts, with 10 participants ranking each prompt’s generated assets.Source: A. Datasets and Evaluation Metrics · p. 8

B. Implementation Details

  • Experiments use TRELLIS-base with DiffusionNFT as the primary setup, with Hunyuan3D-2.1 and GRPO used to assess generalizability.Training considers Alignment, Geometry, Texture, and Overall Quality; the selected cyclic path is O→A→G→T.Source: B. Implementation Details · p. 8

1) Quantitative Results:

  • OuroReward with AdaSelect improves results on proxy rewards and unseen evaluators, while conventional RL can improve proxy scores but degrade unseen-evaluator performance.Table I also includes human preference; Rank2Score with DiffusionNFT achieves the best overall performance.Source: 1) Quantitative Results · p. 9, Table I · p. 9
  • The learning curves and dimension-wise proxy rewards show more stable, consistent improvements and better-balanced gains across quality dimensions.The figure compares TRELLIS-base and Hunyuan3D-2.1; the text reports that weaker dimensions receive more substantial improvements.Source: 1) Quantitative Results · pp. 9–10, Figure 6 · p. 9

2) Qualitative Results:

Qualitative examples show that direct RL can improve one quality dimension while degrading another, whereas OuroReward produces more balanced results. The interference analysis and reward-hacking evaluation likewise favor sequential scheduling over joint optimization.

2) Qualitative Results:

3) Inter-Dimension Interference Analysis:

  • OuroReward achieves higher gradient similarity and reward-change correlation across training steps than Average aggregation and FocalReward.Table II compares these measures under the stated Rank2Score and DiffusionNFT setting; FocalReward helps some weakly correlated pairs but weakens some well-aligned pairs.Source: 3) Inter-Dimension Interference Analysis · p. 10, Table II · p. 11

4) Reward Hacking Analysis:

  • OuroReward improves more consistently across training rewards, unseen evaluators, and human preference, while conventional RL gains may fail to transfer beyond proxy rewards.The reported divergence suggests conventional policies may exploit evaluator-specific biases; OuroReward’s more consistent gains indicate better generalization.Source: 4) Reward Hacking Analysis · p. 11

D. Ablation Study

The ablations find benefits from both AdaSelect and OuroReward, with their combination performing best overall. They also support the chosen dependency criteria, sequence order, and proxy reward selection.

D. Ablation Study

  • The ablations use TRELLIS, Rank2Score, and DiffusionNFT by default, assessing variants with the proxy model and unseen evaluators.This setup is used to examine optimization performance and generalization.Source: D. Ablation Study · p. 11

1) Ablation on Proposed Module:

  • Combining AdaSelect and OuroReward achieves the best overall performance, while each module independently improves on the baseline.The baseline uses random prompt sampling and uniform reward aggregation; OuroReward yields a larger gain than AdaSelect alone.Source: 1) Ablation on Proposed Module · p. 11

2) Ablation on Dependency Score Criteria:

  • Combining gradient cosine similarity with reward-change PLCC yields the path O→A→G→T and a more effective optimization path than using either criterion alone.Gradient similarity measures compatibility between update directions; reward-change PLCC captures consistency of reward dynamics.Source: 2) Ablation on Dependency Score Criteria · p. 12, Table V · p. 12

3) Ablation on Sequential Optimization Order:

  • The proposed O→A→G→T order performs best across evaluation metrics compared with random, reversed, and shifted alternatives.The comparisons support both cumulative-conflict path ordering and starting with the dimension having least conflict with the others.Source: 3) Ablation on Sequential Optimization… · p. 12, Table VI · p. 12

4) Ablation on Proxy Reward Model:

  • Rank2Score performs relatively strongly across multiple unseen evaluators, although optimizing a specific proxy reward can yield less consistent gains on unseen evaluators.The proxy-reward ablation compares Uni3D, Qwen3-VL-8B-Instruct, HyperScore, and Rank2Score.Source: 4) Ablation on Proxy Reward Model · p. 12, Table VII · p. 12

V. CONCLUSION

  • The conclusion reports improvements in generation quality across dimensions and training stability, while identifying broader configurations and architectures as future evaluation scope.The stated future work covers broader reward configurations, RL algorithms, and T23D architectures.Source: §V · p. 12

APPENDIX A REWARD MODELS FOR T23D GENERATION

The appendix compares reward-model families, describes prompt-dataset construction, and gives further results with Hunyuan3D-2.1. It reports an accuracy-efficiency trade-off favoring projection-based models and balanced qualitative gains from OuroReward.

APPENDIX A REWARD MODELS FOR T23D GENERATION

  • Projection-based reward models offer the best overall accuracy-efficiency trade-off, with HyperScore and Rank2Score performing strongly across multiple quality dimensions.Their reported inference cost is approximately 10^-1 seconds per sample; LLM-based methods cost one to three orders of magnitude more and are more instruction-sensitive.Source: Appendix A · p. 13, Table S8 · p. 13

APPENDIX B GPT INSTRUCTION FOR DATASET CONSTRUCTION

  • GPT-5.6 refines and expands prompts from existing benchmarks using a compositional schema, filtering duplicates and semantically implausible combinations.The schema covers subjects, attributes, materials, actions, relationships, scenes, and styles.Source: Appendix B · pp. 13–14

APPENDIX C MORE EXPERIMENTAL RESULTS

  • With Hunyuan3D-2.1, OuroReward produces more balanced improvements than the base model, GRPO, or DiffusionNFT across the illustrated quality dimensions.For the stool prompt, GRPO improves alignment but leaves fractured legs, while DiffusionNFT improves texture but leaves alignment and geometry limited.Source: Figure S9 · p. 13, Appendix C · p. 14
Loading 2610.03423v1…