PaperlayerView exact revision on arXiv

Source-linked AI summary

Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

Xin Jin, Huanqia Cai, Zhen Li, Zechao Zhan, Dengyang Jiang, Aiming Hao, Yuming Jiang, Xiangpeng Yang, Chunle Guo, Peng Gao, Ming-Ming Cheng, Steven C. H. Hoi

arXiv:2606.09076v3cs.CV

TL;DR

Visual preferences are subjective, making deterministic scalar rewards inadequate for capturing rubric-score uncertainty and fine-grained differences. Z-Reward distills reasoning-conditioned score distributions from a large VLM teacher into an efficient student, with the teacher reaching 89.6% human preference accuracy and the student 88.6%.

  • Problem

    Visual preferences are subjective and better represented as distributions over rubric scores than deterministic scalars, while efficient optimization requires direct reward signals.

  • Method

    Z-Reward trains a reasoning-based VLM teacher with GDSO and distills its score distributions into a compact student through RISD for efficient direct scoring.

  • Results

    89.6% human preference accuracy is achieved by the 27B GDSO teacher, while the 9B RISD student reaches 88.6% and outperforms the OPD student.

  • Takeaways & Limitations

    Z-Reward provides efficient, direct, differentiable reward scoring and serves as an effective signal for text-to-image optimization.

  • Takeaways & Limitations

    The experiments focus on text-to-image generation, while broader modalities and downstream reward-modeling settings remain future directions.

Abstract

from arXiv · show

Reward models are central to text-to-image post-training, but visual preference is subjective and better represented as a distribution over rubric scores than as a deterministic scalar. Existing scalar, score-token, and pairwise reward models over-compress uncertainty and fine-grained score differences, while reasoning-based generative rewards provide stronger judgments but are costly to deploy and difficult to use as direct optimization signals. We propose Z-Reward, a teacher-student reward modeling framework that decouples reasoning-heavy judgment from efficient reward deployment. The teacher is a large VLM that uses reasoning to infer rubric-aligned score distributions, and is trained with Group-wise Direct Score Optimization (GDSO), which combines policy-gradient rewards from distribution expectations with direct pointwise and pairwise supervision on score distributions and score gaps. The student is trained with Reasoning-Internalized Score Distillation (RISD), which transfers the teacher's reasoning-conditioned score distribution into a compact VLM without requiring explicit reasoning chains at inference time. On our internally annotated evaluation set, the 27B GDSO teacher reaches 89.6% human preference accuracy, outperforming SFT, RewardDance, and GRPO, while the 9B RISD student reaches 88.6%, outperforming the OPD baseline and closely matching the larger teacher. We further show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization, yielding a 41.3% net human-preference improvement over the SFT baseline.

1 Introduction

Z-Reward addresses the tension between subjective, uncertainty-rich visual preferences and the need for efficient, differentiable reward signals through a reasoning-based teacher and distilled student. Its 27B teacher achieves 89.6% human preference accuracy, while the 9B student reaches 88.6% and remains efficient for optimization.

  • Motivation: Scalar, score-token, and pairwise reward models efficiently compress visual preference into single values or comparisons, discarding annotator uncertainty and fine-grained score differences.For example, distinct positions around a score boundary can collapse to the same discrete score.
  • Z-Reward framework: Z-Reward trains a large VLM teacher to infer calibrated, reasoning-conditioned score distributions from scalable rubric-based supervision.Reasoning decomposes visual evidence, applies rubric criteria, and allocates probability mass across neighboring score bins.
  • Student distillation: Reasoning-Internalized Score Distillation transfers the teacher’s reasoning-enhanced distribution into a compact student that directly predicts scores without inference-time reasoning chains.This enables efficient deployment and gradient backpropagation.
  • Teacher optimization: Group-wise Direct Score Optimization trains the teacher by optimizing rewards from predicted score distributions with direct distribution-level supervision.The distribution is learned as a latent, reasoning-conditioned distribution rather than requiring repeated human annotations of it.
  • Empirical results: 89.6% human preference accuracy is achieved by the 27B GDSO teacher, outperforming SFT, RewardDance-style supervision, and GRPO.The 9B RISD student reaches 88.6%, outperforming the OPD student while closely matching the larger teacher and supporting text-to-image optimization.

2 Annotation and Datasets

The annotation framework scores four production-critical dimensions with detailed five-level rubrics, while combining diverse prompt sources, multi-annotator evaluation, score refinement, and quality control. It also exposes context mismatches between lengthy annotation materials and deployable pointwise reward models.

  • Annotation document: Annotations cover Text–Image Alignment, Realism, Aesthetics, and Physical Plausibility using five-level rubrics that specify how error patterns affect scores.The rubric avoids relying only on abstract labels such as “minor” or “major.”
  • Data for annotation and evaluation: Prompts come from rewritten internal captions, real-world user or community prompts, and diverse concepts sampled from topology, composition, and LLM expansion.The sources cover compositional phenomena emphasized by text-to-image evaluation benchmarks.
  • Annotation workflow: Annotators assign rubric-based pointwise scores, compare same-prompt candidates within coarse score bins for ±0.5 adjustments, and submit results for quality-control review.Only data from annotators whose audited accuracy exceeds a preset threshold enters the training set.
  • The risk of context mismatch: 307,200 image tokens are required before textual instructions for four dimensions, five score bins, and 15 images per bin, making the full annotation document too long for reward-model context.Annotators can compare same-prompt candidates during score adjustment, whereas a deployable pointwise reward model only observes a narrower context.

3 Method

Z-Reward separates reasoning-intensive teacher training from efficient student deployment: GDSO calibrates reasoning-conditioned score distributions using policy and direct supervision, while RISD transfers those distributions without explicit inference-time reasoning.

  • Teacher modeling: The teacher generates a reasoning trace and predicts a distribution over rubric-aligned score bins for each prompt, image, and reward dimension.The distribution is learned from large-scale score supervision, same-prompt score-gap constraints, and reasoning-conditioned score-token probabilities.
  • Teacher optimization: GDSO augments group-normalized policy optimization with direct supervised gradients on score distributions and same-prompt score gaps.Its rewards use distribution expectations, while direct losses supervise score-bin probabilities and induced score gaps.
  • Teacher optimization: Group-wise supervision applies pointwise losses across all sampled distributions and pairwise losses across every cross-side sampled output pair.This provides multiple reasoning-conditioned distributional views for policy optimization and direct score calibration.
  • Teacher optimization: Pointwise supervision calibrates absolute scores, while pairwise score-gap supervision preserves within-prompt preference direction, quality-difference magnitude, and annotation-scale margins.The gap objective avoids arbitrary margin expansion by matching annotated score gaps rather than only requiring the winner to exceed the loser.
  • Student distillation: RISD trains a compact student to predict the teacher’s reasoning-conditioned score distribution directly, without generating reasoning chains at inference time.The resulting reward model supports efficient pointwise scoring and differentiable reward-guided optimization.

4 Experiment

Experiments show that GDSO provides the strongest 27B teacher performance, while RISD transfers its reasoning-conditioned score distribution to a 9B student with near-teacher preference accuracy. Distribution-expectation rewards improve optimization signals, and RISD substantially reduces decoding cost by producing a score in one token.

  • Reward-model evaluation: 0.8864 HPA and 0.9801 margin HPA make RISD the best 9B model, closely matching the 27B GDSO teacher.RISD is consistently best across the 9B metrics, suggesting effective internalization of the teacher’s reasoning-conditioned score distribution.
  • Reward-model evaluation: GDSO achieves the best results on all metrics for the 27B teacher and improves over GRPO in calibration and preference ranking.The evaluation reports PLCC and SRCC for score calibration, plus HPA and margin HPA for preference ranking; margin HPA uses human score gaps above 0.5.
  • Reward-model evaluation: 0.7817 HPA improves over SFT’s 0.7459 for RewardDance on 9B, but its PLCC and SRCC fall to 0.5182 and 0.4338 from 0.5296 and 0.4942.The contrast indicates that post-hoc pseudo reasoning improves coarse pairwise preference recognition without guaranteeing calibrated rubric scores.
  • Score-distribution reward decoding: Distribution-expectation rewards consistently improve both HPA and margin HPA for GRPO and GDSO compared with parsing textual scores.Text parsing quantizes nearby predictions such as 3.8 and 4.2 to the same score token, whereas distribution expectations preserve finer-grained uncertainty.
  • Reasoning distillation: RISD returns the score in one output token, while OPD and GDSO require about 750 output tokens for long autoregressive reasoning traces.OPD improves over SFT but does not approach the 27B teacher or 9B RISD student; RISD matches SFT’s decoding cost.

5 Validating Z-Reward as an Optimizable Reward Signal

This section validates Z-Reward as an optimizable reward signal by applying its predicted score distributions to reinforcement learning for text-to-image generation. Unlike discrete scalar rewards, Z-Reward provides dense, informative gradient signals for optimization.

  • Application to Text-to-Image Optimization: Z-Reward is applied during the reinforcement-learning stage of text-to-image generation to demonstrate its practical utility.The setting includes prior approaches based on policy-gradient fine-tuning, direct preference optimization, and differentiable reward backpropagation.
  • Dense Gradient Signals: Unlike traditional discrete scalar rewards, Z-Reward’s predicted score distributions provide dense and informative gradient signals.These distributions are used to overcome the sparse guidance associated with discrete scalar rewards.
  • Optimization: The resulting gradients are leveraged to directly optimize text-to-image generation.

SFT RL

Z-Reward is used as a differentiable, multi-dimensional reward signal to optimize image generators through the denoising process. This optimization improves validation rewards across four visual dimensions and yields a 41.3% net human-preference improvement over the SFT baseline.

  • Optimization: Z-Reward backpropagates aggregated student reward predictions through the denoising process to update the generator.The reward dimensions are text–image alignment, realism, aesthetics, and physical plausibility.
  • Optimization: Validation rewards steadily improve across text–image alignment, realism, aesthetics, and physical plausibility.Realism and aesthetics improve faster early, whereas text–image alignment and physical plausibility increase more gradually.
  • Human evaluation: 41.3% net GSB improvement is achieved over the strong SFT baseline with Z-Reward-guided optimization.The evaluation uses blind human judgments on 400 held-out prompts, with Good-Same-Bad measuring whether optimized images are better, comparable, or worse than baseline images.
  • Human evaluation: Qualitative comparisons show improvements in text-image alignment, visual realism, aesthetics, and physical plausibility across held-out prompts.Figure 6 compares SFT baseline generations with Z-Reward-optimized generations for each prompt.

6 Discussions and Future Works

The discussion identifies weak coupling between generated reasoning and final scores as a limitation of the teacher objective, while proposing broader sequence-to-score and unified reward-modeling extensions. Future work includes explicit consistency or contrastive supervision and applications across modalities and evaluation settings.

  • Reasoning-score coupling: Direct supervised losses improve score calibration and preference metrics but can occasionally make final scores depend more on score supervision than reasoning traces.Experiments indicate weak reasoning-score coupling is a minority case, while the metric gains are clear.
  • Reasoning-score coupling: Future work could enforce reasoning-score consistency through explicit checks or contrastive supervision.These additions aim to strengthen the connection between generated reasoning and final scores.
  • Potential Generalization to All Sequence-to-Score Tasks: The formulation can generalize beyond image generation to image-, video-, or text-centered sequence-to-score evaluation and reward modeling for LLMs and VLMs.The paper instantiates the approach on text-to-image generation, while broader modalities and downstream settings remain future directions.
  • Possible Extension to Unified Reward Modeling: The annotation pipeline supports unified reward modeling because scoring multiple candidates under one prompt provides calibrated rubric scores and richer comparison signals than binary preferences.This enables comparison training using annotator adjustments that implicitly encode candidate comparisons.

7 Related Work

Prior visual reward models span scalar, regressive VLM, and generative reasoning-based designs, while preference optimization uses reinforcement learning, pairwise objectives, and direct backpropagation. Z-Reward addresses the trade-off between efficient optimization and richer reasoning by distilling reasoning-enhanced judgments into deployable reward signals.

  • Early visual reward models, including ImageReward, PickScore, and HPSv2, use CLIP-style encoders to predict scalar preference scores.
  • VLM-based reward models replace CLIP encoders with stronger multimodal backbones and regressive reward heads, including VisionReward, VideoAlign, HPSv3, and related systems.
  • Generative reward models exploit VLM next-token prediction and reasoning, with representative systems including DeepSeek-GRM, GenRM-CoT, UnifiedReward, RewardDance, and Edit-R1.
  • Visual preference optimization uses diffusion- or flow-adapted policy gradients, pairwise preference objectives, and direct reward backpropagation, but its effectiveness depends heavily on the reward signal.
  • Scalar rewards are efficient but can compress subjective preferences and enable reward hacking, whereas reasoning-based rewards are semantically richer yet costly or incompatible with direct reward backpropagation.
  • OPD transfers reasoning behaviors from stronger teachers to compact students, but its trajectory-centric objective differs from reward modeling's efficient scoring and differentiable optimization goals.

8 Conclusion

Z-Reward models human preference as a reasoning-conditioned score distribution through a decoupled teacher-student framework. Its GDSO teacher combines policy-gradient learning with direct distributional supervision, while RISD enables a compact student and supports text-to-image optimization.

  • Framework: Z-Reward represents human preference as a reasoning-conditioned score distribution rather than a deterministic scalar.This formulation is designed for visual reward modeling.
  • Teacher training: Group-wise Direct Score Optimization combines policy-gradient learning with direct supervision on score distributions and score gaps.The approach improves calibrated scoring and pairwise preference ranking.
  • Empirical results: The 27B GDSO teacher outperforms SFT, RewardDance, and GRPO, while the 9B RISD student closely matches the larger teacher.Experiments were conducted on an internally annotated benchmark.
  • Applications and scope: The RISD student serves as an effective reward signal for text-to-image optimization.The conclusion also frames Z-Reward as a general sequence-to-score modeling paradigm extendable across modalities.
Loading 2606.09076v3…