PaperlayerView exact revision on arXiv

Source-linked AI summary

RewardDance: Reward Scaling in Visual Generation

Jie Wu, Yu Gao, Zilyu Ye, Ming Li, Liang Li, Hanzhong Guo, Jie Liu, Zeyue Xue, Xiaoxia Hou, Wei Liu, Yan Zeng, Weilin Huang

arXiv:2509.08826v1cs.CV

TL;DR

Visual reward-model scaling remains limited by architectural and paradigm mismatches, motivating RewardDance. RewardDance uses a generative yes-token reward and scales model size and context; experiments show consistent quality gains across visual-generation tasks under RL fine-tuning and test-time scaling.

  • Problem

    Visual reward models face architectural limitations or paradigm mismatches that constrain effective scaling, while reward-model scaling remains underexplored.

  • Method

    RewardDance converts reward scores into a VLM’s predicted probability of a "yes" token and scales model size from 1B to 26B parameters alongside task-aware context.

  • Results

    Across text-to-image, text-to-video, and image-to-video tasks, scaling reward-model size and context consistently improves reward-signal and final generation quality under RL fine-tuning and test-time scaling.

  • Takeaways & Limitations

    The work establishes scalability as a foundational principle for visual reward models and reports more robust reward modeling for future visual generation.

  • Takeaways & Limitations

    The paper identifies cross-modal reward-signal scaling for future many-to-vision tasks as an important area requiring further exploration.

Abstract

from arXiv · show

Reward Models (RMs) are critical for improving generation models via Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation remains largely unexplored. It primarily due to fundamental limitations in existing approaches: CLIP-based RMs suffer from architectural and input modality constraints, while prevalent Bradley-Terry losses are fundamentally misaligned with the next-token prediction mechanism of Vision-Language Models (VLMs), hindering effective scaling. More critically, the RLHF optimization process is plagued by Reward Hacking issue, where models exploit flaws in the reward signal without improving true quality. To address these challenges, we introduce RewardDance, a scalable reward modeling framework that overcomes these barriers through a novel generative reward paradigm. By reformulating the reward score as the model's probability of predicting a "yes" token, indicating that the generated image outperforms a reference image according to specific criteria, RewardDance intrinsically aligns reward objectives with VLM architectures. This alignment unlocks scaling across two dimensions: (1) Model Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context Scaling: Integration of task-specific instructions, reference examples, and chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that RewardDance significantly surpasses state-of-the-art methods in text-to-image, text-to-video, and image-to-video generation. Crucially, we resolve the persistent challenge of "reward hacking": Our large-scale RMs exhibit and maintain high reward variance during RL fine-tuning, proving their resistance to hacking and ability to produce diverse, high-quality outputs. It greatly relieves the mode collapse problem that plagues smaller models.

1 Introduction

RewardDance presents scalability as a central principle for visual reward models, combining a generative reward paradigm with scaling in model size and context. Experiments report improved reward-model behavior and visual generation quality across these dimensions.

  • Model Scaling: Model scaling expands RewardDance reward models from 1B to 26B parameters and links parameter count to reward-model performance and generation quality.The framework systematically scales VLM capacity across this range.
  • Context Scaling: Context scaling incorporates task-aware instructions, reference examples, and chain-of-thought reasoning to support more robust and accurate reward judgments.These inputs extend beyond traditional image-text pairs.
  • Experimental Findings: Across text-to-image, text-to-video, and image-to-video generation, stronger reward models show higher exploration, less reward hacking, and consistently improved generation quality under RL and test-time scaling.The experiments cover open-source and proprietary models and report stable scaling effects.
  • Scalability as the Principle for RMs: RewardDance identifies scalability as a fundamental design principle for visual reward models.The framework addresses an underexplored dimension in existing work.
  • Generative Reward Modeling: Generative reward modeling reframes reward prediction as next-token prediction, aligning the objective with VLM autoregressive mechanisms.The approach replaces the prior paradigm mismatch with a generative formulation.

2 Related Work

Related work progresses from CLIP-based reward models to regression-based VLM reward models and newer generative approaches. These lines of work differ in their architectural choices, training paradigms, and treatment of scaling.

  • CLIP-based Reward Models: Early reward models such as ImageReward, PickScore, and HPSv2 fine-tuned CLIP models to produce human-preference scores.These approaches used CLIP as the reward-model backbone.
  • CLIP-based Reward Models: CLIP-based methods were limited by architectural constraints and single-modality design, while CLIP scaling and task generalization remained difficult.The paper contrasts these limitations with later VLM-based approaches.
  • Regression-based VLM Reward Models: VLM-based reward models commonly appended regression heads and trained with Bradley–Terry loss to output reward signals.These methods are categorized as regression-based reward models.
  • Scaling and Generative Paradigms: WorldPM explored reward-model scaling through enhanced data and model architectures, while other work pursued generative paradigms without effective scaling.The paper characterizes progress across these approaches as fragmented.
  • RLHF for Diffusion Models: RLHF pipelines train reward models on human preference data and use them to guide diffusion generation, with DDPO and ReFL providing distinct optimization strategies.DDPO adapts PPO to diffusion, whereas ReFL directly optimizes outputs using gradients from a frozen reward model.

3 Method

RewardDance replaces conventional regressive reward modeling with a generative VLM-aligned paradigm and scales reward modeling through model size, context, and reasoning. Its reward signal is the probability of a “yes” token in comparative judgments, with task-aware instructions, references, and CoT reasoning enriching evaluation.

  • Core generative paradigm: Conventional reward models use pointwise scalar prediction, whereas RewardDance treats reward modeling as generative next-token prediction.Prior approaches include CLIP similarity and VLM regression heads optimized with Bradley–Terry loss.
  • Model scaling: The framework scales reward models from 1B to 26B parameters using VLM variants.The method links increasing model capacity to reward-model performance and generation quality.
  • Context scaling: RewardDance scales context with task-aware instructions, reference images, and Chain-of-Thought reasoning.The model can generate both a decision and a rationale, making reward judgments more interpretable.
  • Core generative paradigm: RewardDance scores whether one image is superior by predicting “yes” or “no” from two images, a prompt, and a task-aware instruction.The reward is the predicted probability of the “yes” token.
  • Generative variants: RewardDance also supports pointwise generative scoring, where the “yes” probability evaluates whether a generated image meets instructed quality standards.This variant uses prompts and generated images without a reference image and adds a weighted cross-entropy loss during training.
  • Training and inference: During optimization, RewardDance supplies comparative preference signals, including references selected through Best-of-N sampling and verifier-based path pruning.The framework is used with ReFL for RL fine-tuning and Search over Paths for inference-time scaling.

4 Experiments

The experiments evaluate RewardDance across text-to-image, text-to-video, and image-to-video generation, analyzing reward-model scaling, inference-time scaling, and architectural choices through experiments and ablations.

  • Experimental scope: Experiments cover text-to-image, text-to-video, and image-to-video generation.The evaluation also studies reward-model scaling, inference-time scaling, and architectural choices.

4.1 Evaluation Baselines, Benchmarks and Metrics

The evaluation compares RewardDance with academic and industry baselines using image and video benchmarks designed around alignment and human preference judgments.

  • Baselines: Image experiments compare against representative academic and industry text-to-image systems.The listed baselines include models such as FLUX.1-dev, Midjourney V6.1, Seedream 2.0, and Qwen-Image.
  • Baselines: Video experiments compare against leading industry and open models including Kling, Veo, Wan 2.1, Sora, and Runway Gen 4.These baselines cover both text-to-video and image-to-video comparisons.
  • Benchmarks: Bench-240 evaluates text-to-image generation with 240 prompts calibrated using representative benchmarks and user preference surveys.Its maximum score is 100 points.
  • Benchmarks: SeedVideoBench-1.0 contains 300 prompts for text-to-video and image-to-video evaluation across diverse application domains.Image-to-video prompts include annotations for the initial frame.
  • Metrics: Image evaluation uses a human-verified image-text alignment score based on whether generated images depict prompt elements.The metric is applied to text-to-image experiments.
  • Metrics: Video evaluation uses the human preference-based GSB score, with G, S, and B denoting Good, Same, and Bad judgments.The score is defined from these three judgment counts.
  • Metrics: Video-text alignment additionally assigns scores of 0, 1, or 2 for complete mismatch, partial match, or perfect match.This rubric enables comparison with industry products.

4.2 Comparison with Reward Model Scaling

Scaling RewardDance improves generation quality across image and video tasks, RL and test-time scaling, and both text-to-video and image-to-video settings. The results also indicate that out-of-domain reward-model accuracy is more predictive of RL outcomes than in-domain accuracy.

  • Image generation: 73.6 versus 67.0: scaling the RM from 1B to 26B raises FLUX.1-dev text-to-image performance under RL fine-tuning.For Seedream-3.0, the corresponding scores increase from 74.1 to 84.8.
  • Image generation: 84.8 versus 74.1: Seedream-3.0 reaches a higher text-to-image score with the 26B RM than its baseline under RL fine-tuning.Test-time scaling also shows a steady improvement trend with Seedream-3.0.
  • Reward-model evaluation: Reward-model accuracy on the in-domain dataset has no strict positive correlation with parameter scale, especially among 1B, 2B, and 4B models.The in-domain dataset contains 2,500 held-out sample pairs.
  • Reward-model evaluation: Out-of-domain reward-model accuracy is described as a more significant predictor of final RL outcomes than in-domain accuracy.The out-of-domain dataset contains over 4,000 sample pairs from public benchmark datasets.
  • Video generation: +49%: the 26B RM improves Seedance-1.0 text-to-video RL performance relative to the SFT baseline.The improvement rises from +28% with the 1B RM.
  • Video generation: +47%: the 26B RM improves Seedance-1.0 image-to-video RL performance relative to the SFT baseline.The improvement increases from +29% with the 1B RM.
  • Overall scaling effects: Scaling reward models consistently improves final generation quality across text-to-image, text-to-video, and image-to-video experiments.The reported gains hold under RL fine-tuning and test-time scaling, with GSB used for video evaluation.

4.3 Comparison with State-of-the-art Generation Models

RewardDance-optimized models achieve state-of-the-art results across text-to-image, text-to-video, and image-to-video benchmarks, outperforming strong academic, industrial, and proprietary baselines.

  • GenEval: 0.79 is the top GenEval overall score for Seedream-3.0 with RewardDance, exceeding SD3 at 0.74.
  • GenEval: +0.10 and +0.09 are the overall gains for RewardDance-enhanced Seedream-3.0 and FLUX.1-dev, respectively.
  • Bench-240: 0.848 is Seedream-3.0 with RewardDance’s highest Overall Score on Bench-240, surpassing Imagen 3 at 0.79, Luma at 0.77, and Midjourney V6.1 at 0.63.
  • SeedVideoBench-1.0: 1.66 is Seedance 1.0’s highest Text-to-Video average score on SeedVideoBench-1.0, above Veo-3.0 at 1.63 and Kling 2.1 at 1.57.
  • SeedVideoBench-1.0: 1.65 is Seedance 1.0’s Image-to-Video score, tying the best-performing model, Kling 2.1.

4.4 Ablation Study

The ablations show that larger reward models, generative scoring, richer references, and reasoning context improve performance while preserving exploration during RL.

  • Reward Dynamics: The 26B reward model maintains substantially greater reward variance throughout training, whereas 2B and 1B models converge to narrow ranges earlier.
  • Reward Dynamics: As reward-model scale increases, final performance and late-training exploration variance improve monotonically.
  • DiT Parameter Analysis: Seedream-3.0 gains +3.92% from 8B to 26B reward models, compared with +2.75% for Seedream-3.0 Lite.
  • Generative Reward Paradigm: Generative reward modeling improves FLUX.1-dev from 70.8 to 71.6 and Seedream from 80.7 to 81.0.
  • Generative Reward Paradigm: Adding reference images further improves FLUX.1-dev from 71.6 to 73.0 and Seedream from 81.0 to 81.6.
  • BoN Reference Examples: Higher-quality Best-of-N reference images yield incremental Seedream-3.0 improvements, with Best-of-16 Top-2 performing best.
  • Reasoning: CoT reasoning raises the Seedream-3.0 score from 81.6 to 83.6.

4.5 Visualizations

Visualizations show that increasing reward-model size improves generation performance for both image and video tasks.

  • Larger reward models show an improving performance trend across both image and video generation tasks.On the image task, the 26B model can completely and correctly generate descriptions involving multi-instance quantity relationships, unlike smaller models.

5 Discussion and Further work

The discussion identifies future work in scaling model parameters, capability dimensions, task scope, multimodal inputs, and contextual information.

  • Parameter Scaling: The current maximum reward-model size is 26B parameters; scaling to 70B or 100B remains future work.
  • Capability Dimension Scaling: The work focuses mainly on foundational vision-language capabilities such as alignment, leaving motion modeling and aesthetic generation for future research.
  • Task Scope Scaling: Applying Reward Scaling to unified understanding-and-generation models remains an open direction across visual understanding, generation, and editing.
  • Multimodal Scaling: Cross-modal Reward Signal Scaling is identified as important for future many-to-vision tasks such as audio/video-to-video generation.
  • Context Scaling: Richer references, more complex instructions, reflection, and in-context learning may further boost model performance.

6 Conclusion

RewardDance addresses a critical gap in visual reward modeling by reframing reward prediction as VLM token generation, aligning it with autoregressive mechanisms. Scaling model size and context richness consistently improves reward signals and final generation quality across multiple visual-generation tasks.

  • 6 Conclusion: RewardDance converts reward scores into a VLM’s predicted probability for a “yes” token, aligning reward prediction with autoregressive next-token generation.This generative paradigm avoids the stated architectural and paradigm mismatches constraining existing visual reward models.
  • 6 Conclusion: The framework scales along two dimensions: model size from 1B to 26B parameters and context richness through task-aware instructions, reference examples, and Chain-of-Thought reasoning.These dimensions define the framework’s central scalability strategy.
  • 6 Conclusion: Experiments across text-to-image, text-to-video, and image-to-video tasks show that scaling consistently improves reward-signal quality.The reported improvements span multiple visual-generation settings rather than a single task.
  • 6 Conclusion: Improved reward signals drive stable and significant gains in final generation quality under both RL fine-tuning and Test-time Scaling.The conclusion identifies both optimization regimes as benefiting from the scaled reward modeling approach.
Loading 2509.08826v1…