Source-linked AI summary
EmoReward: Self-Evolving Reward Calibration for Emotional Video Generation RL
Mingzhe Zheng, Yue Wu, Jing Tan, Chang Zou, Dengyang Jiang, Zeyue Tian, Xuanhua He, Ryan Xu, Yunyang Ge, Sihui Ji, Yunlong Lin, Kaihang Pan, Hongru Yan, Sixiang Chen, Dianyi Wang, Julian Zhang, Weijie Kong, Changlin Li, Lanqing Guo, Zhao Zhong, Qifeng Chen, Harry Yang
TL;DR
Video emotional expression has no uniquely correct target, and fixed visual, motion, and text rewards may not capture emotional intent across contexts. EmoReward calibrates evidence using video-level gates and reliability learned from completed-run traces, then applies fixed multipliers in the next run. Experiments report improvements across multiple video RL algorithms, human evaluations, and selected image-transfer settings, with objective-dependent trade-offs.
Problem
Emotional video quality is subjective and lacks a uniquely correct target, while fixed rewards for visual quality, motion, and text alignment can poorly capture emotional intent.
Source: §1 · p. 2Method
EmoReward gates heterogeneous emotional evidence within each video and uses completed-run reward–outcome traces to set bounded provider multipliers for subsequent training without updating reward models.
Source: §1 · p. 3, §3.2 · p. 5Results
On HunyuanVideo 1.5, guided configurations score higher on VideoAlign across three RL algorithms, while EmoReward leads seven reward policies on reliability selectivity and both composite calibration scores.
Source: §1 · p. 3Takeaways & Limitations
Experiments across video optimizers, image-transfer analyses, and human ratings support adaptive reward calibration while identifying objective-dependent trade-offs.
Source: §1 · p. 3, §5 · p. 10RewardMemory reliability measures agreement with the reference outcome rather than independent emotional fidelity; provider-pool comparisons also vary weights and curricula, so they do not isolate each channel’s causal effect.
Source: §3.5 · p. 6, §4.2 · p. 8
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) has advanced generative model post-training in verifiable domains such as code and mathematics, but evaluating video quality, especially emotional expression, often depends on subjective aesthetics and has no uniquely correct target. Existing approaches typically rely on fixed rewards for visual quality, motion quality, and text alignment that poorly capture emotional intent and can invite reward hacking, while RLHF is difficult to scale and subject to individual aesthetic differences that complicate the calibration of facial, body motion, and chromatic cues across narrative contexts. We introduce EmoReward, a new self-evolving training mechanism that distills evidence reliability from reward traces collected during an actual training run to adaptively guide multi-reward weight allocation in subsequent training by determining which emotional evidence to trust and how much to weight it. EmoReward first plans specific grounding cues from the emotional prompt to guide evaluation, then uses heterogeneous signal primitives to score the generated video during the act stage. Reflective Reward Gating calibrates their contributions through instance-level gates for phase, face confidence, conflict, and subtlety, then renormalizes weights over the active set to produce the sample's fused reward. RewardMemory then estimates each primitive's reliability from rollout scores and outcomes within each emotional regime in the finished training run, then fixes bounded multipliers that calibrate fusion weights in the next training run without updating reward models. Experiments on HunyuanVideo 1.5 show improvements across multiple RL algorithms and compare EmoReward with other agentic reward methods under SAGE-GRPO. Ablations and transfer experiments on SD 3.5 and FLUX further support adaptive reward calibration for visual generation.
1 Introduction
Emotional video rewards lack a uniquely correct target, while fixed quality and alignment rewards may miss emotional intent and subjective preferences are difficult to scale. EmoReward calibrates evidence within videos and across training runs, with experiments reporting gains across video algorithms, human ratings, and selected image-transfer settings.
1 Introduction
- Fixed rewards may inadequately capture emotional intent, while costly human annotation and aesthetic differences complicate calibrating emotional cues across contexts.Fixed fusion can mix informative, missing, and saturated signals, motivating both within-video cue-validity assessment and cross-run allocation learning.Source: §1 · p. 2
- EmoReward transfers evidence reliability from a completed training run to guide multi-reward allocation in the next, without updating reward-model parameters.It is presented as a self-evolving mechanism for emotional video-generation RL.Source: §1 · pp. 2–3
- Plan specifies affect targets and observable cues, while Act scores videos with heterogeneous primitives and keeps criterion scores separate from evidence validity.Reflective Reward Gating uses phase, face confidence, semantic conflict, and subtlety before renormalizing over active providers.Source: §1 · p. 3
- On HunyuanVideo 1.5, guided configurations achieve higher VideoAlign scores across three RL algorithms, and EmoReward leads seven reward policies on reliability selectivity and both composite calibration scores.User studies also report the highest human preference and Good ratings, and the lowest Bad ratings, across emotional fidelity, temporal coherence, and visual quality.Source: §1 · p. 3
- RewardMemory distills evidence reliability into bounded fusion weights, enabling iterative calibration of RL reward signals across training runs.The introduction describes it as, to the authors’ knowledge, the first memory designed for multi-reward systems in video generation.Source: §1 · p. 3
- The authors report that ablations, human evaluations, and experiments across three video RL algorithms and two image backbones validate EmoReward’s effectiveness.Transfer experiments on Stable Diffusion 3.5 and FLUX show gains in selected configurations alongside objective-dependent trade-offs.Source: §1 · p. 3
2 Related Work
Related work spans verifiable and preference-based feedback, agentic evaluation, and visual-generation rewards. EmoReward is distinguished as transferring reward–outcome evidence across completed runs to calibrate frozen providers.
2 Related Work
- EmoReward transfers reward–outcome evidence across completed training runs to calibrate frozen providers, combining sample-level gating with persistent fusion weights.The cited approaches also include routing experts, evolving rewards, and organizing experience.Source: §2 · p. 3
- Visual-creation agents organize generation through planning, tool use, and feedback-driven refinement, including video-agent workflows and image-agent experience reuse.EmoReward is described as evolving video-generation RL training rather than building a workflow or retraining reward models.Source: §2 · p. 3
- Visual-generation work covers learned reward models, preference optimization for images and video, and benchmarks for quality and compositional alignment.Related areas include 3D, audiovisual, animation, editing, distillation, and feature reuse.Source: §2 · pp. 3–4
- The paper situates EmoReward among affect-grounded augmentation, reflective generation, dynamic facial-expression recognition, multimodal emotion understanding, and grounded affect representations.It focuses on how evidence reliability should govern reward fusion across scenes and training runs.Source: §2 · p. 4
3 Method
EmoReward treats emotional reward calibration as both a per-video assessment of evidence and a transfer of provider reliability across completed training runs. Its pipeline grounds prompts, gates and renormalizes active evidence, then uses fixed memory-derived multipliers in subsequent training.
3.1 Preliminaries
- Fixed fusion assumes constant provider trust across scenes, but calibration matters when validity, errors, or relative contributions vary across candidates or when emotion reward is mixed with another reward.A common constant term cancels from the centered GRPO numerator, so removing it alone need not change the standardized advantage.Source: §3.1 · pp. 4–5
3.2 Overview
- EmoReward combines memory-derived regime multipliers with current-video gates, normalizes active providers, and mixes the resulting emotion reward with the original video reward.Without a supported memory entry, unit multipliers recover Reflect-only fusion; invalid evidence receives no mass.Source: §3.2 · p. 5, §3.4 · p. 6
3.3 Plan and Act: Emotional Grounding
- Plan converts the prompt into an EmoScript of affect targets and observable cues; Act scores an already-generated video and records both criterion agreement and evidence usability.Seven providers enter fusion calibration, while chromatic and kinetic readouts are auxiliary.Source: §3.3 · p. 5
3.4 Reflect: Instance-Level Calibration
- Reflect combines phase, face, conflict, and subtlety gates with fixed memory multipliers, then renormalizes weights over active providers.Gates are computed rather than learned; a provider favored by memory still contributes nothing when its current evidence is invalid.Source: §3.4 · p. 6
3.5 Evolve: RewardMemory
- RewardMemory groups completed-run traces by emotional context and estimates provider reliability from score–outcome correlation, using a global fallback when contextual support is insufficient.It converts deviations from mean reliability into bounded multipliers, exports the table once, and keeps it fixed during the next generator-training run; reliability measures agreement with the reference outcome, not independent emotional fidelity.Source: §3.5 · pp. 6–7
4 Experiments
Across three RL algorithms, guided configurations improve all four VideoAlign heads, and the RewardMemory policy leads on reliability selectivity and both composite calibration scores. Transfer and human-evaluation results extend the evidence, with scope and measurement caveats.
4 Experiments
- Experiments use HunyuanVideo 1.5 with DanceGRPO, FlowGRPO, and SAGE-GRPO on a subset of 302 emotional prompts that does not overlap with training.VideoAlign is also an optimization objective, so agreement with it is distinguished from independent emotional fidelity.Source: §4 · p. 7
4.1 EmoReward as a Base-RL-Agnostic Reward Add-on
- Overall improves by 0.3218 for DanceGRPO with Reflect, 0.2343 for FlowGRPO with EmoReward, and 0.5645 for SAGE-GRPO with EmoReward.All four VideoAlign heads increase for each algorithm; PickScore and CLIPScore changes are smaller and mixed.Source: §4.1 · p. 7, Table 1 · p. 8
4.2 Ablations and Mechanism Analysis
- Under SAGE-GRPO, adding RewardMemory yields the highest reliability selectivity (0.428) and both composite scores (core 0.922, emo 0.884); without memory, the variant leads the standalone preference pillar.SNR∗ is a dispersion statistic, not a measure of gradient quality.Source: §4.2 · p. 8, Table 2 · p. 8
- The provider-pool comparison motivates the interface but does not isolate each channel’s causal effect because weights and curricula also differ.Seven channels enter memory calibration; chromatic and kinetic readouts remain auxiliary.Source: §4.2 · p. 8, Figure 5 · p. 9
4.1 EmoReward as a Base-RL-Agnostic Reward Add-on
- The four VideoAlign heads improve in every emotion category, while PickScore and CLIPScore show smaller, nonuniform changes.Categories overlap within the evaluated set.Source: §4.1 · pp. 7–8, Figure 4 · p. 9
4.3 Generalization to image backbones
- Image-transfer results are objective-dependent: SD3.5–DanceGRPO improves on HPSv3 with calibration, whereas FLUX’s HPSv3-optimized baseline remains strongest on that metric.Unequal SD3.5 training budgets limit causal attribution.Source: §4.3 · p. 9
4.4 Human evaluation
- EmoReward obtains the highest Good shares (60.0%, 54.3%, 57.2%) versus SAGE-GRPO (53.0%, 51.5%, 45.0%) and the lowest Bad shares on all three criteria.Eighteen participants rated three methods on 30 prompts; the reported margins do not resolve variation across people or prompts.Source: Table 3 · p. 10, §4.4 · p. 10
5 Conclusion
The conclusion presents EmoReward as a self-evolving reward-calibration framework for emotional video-generation RL. Experiments and analyses support adaptive calibration while identifying objective-dependent trade-offs.
5 Conclusion
- EmoReward combines instance-level Reflective Reward Gating with RewardMemory, transferring provider-reliability estimates between training runs without updating reward-model parameters.The conclusion frames this as a self-evolving reward-calibration framework for emotional video-generation reinforcement learning.Source: §5 · p. 10
- Experiments across video optimizers, ablations, image-transfer analyses, and human ratings support the utility of adaptive calibration while revealing objective-dependent trade-offs.Source: §5 · p. 10
- The findings suggest that experience from a completed run can inform how trust is allocated among heterogeneous reward providers.The conclusion presents reward-system evolution across training as a foundation for further research.Source: §5 · p. 10
Reproducibility Statement
The appendix documents the reward interface, calibration rule, derivation, and evaluation protocol. Editable figures and scripts for tables and visualizations are also identified as source artifacts.
Reproducibility Statement
- The appendix specifies the reward interface and calibration rule.Source: Reproducibility Statement · p. 11
- It provides the mathematical derivation, evaluation protocol, complete comparison tables, and qualitative examples.Source: Reproducibility Statement · p. 11
- Source artifacts include editable figures and scripts for constructing tables and visualizations.Source: Reproducibility Statement · p. 11
AI Use Statement
AI assistance was used for manuscript editing, mathematical checks, and figure preparation. The authors state responsibility for verifying the scientific claims and final submission.
AI Use Statement
- AI assistance supported manuscript editing, mathematical checks, and figure preparation.Source: AI Use Statement · p. 11
- The authors state that they are responsible for verifying the scientific claims and final submission.Source: AI Use Statement · p. 11
- Experimental comparison images and video frames came from evaluated models, with no additional media synthesized to illustrate outcomes.Source: AI Use Statement · p. 11
A Limitations and Future Work
The evaluation focuses on emotion, and broader use requires task-appropriate providers and meaningful calibration outcomes. The paper evaluates one reference-to-calibrated transition, while repeated evolution and model-level human feedback remain proposed extensions.
A Limitations and Future Work
- The evaluation focuses on emotion; broader applications require a task-appropriate provider pool and a meaningful calibration outcome.Action fidelity and audio–visual correspondence are suggested as visual extensions, but transfer effectiveness remains empirical.Source: A Limitations and Future Work · p. 16
- Experiments test a reference-to-calibrated transition, not sustained improvement across many training generations.Repeated calibration would need to address distribution shift, stale evidence, and feedback loops that reinforce proxy errors.Source: A Limitations and Future Work · p. 16
- The proposed model-level human-feedback mechanism is not implemented in the reported ordinal study.Mapping subjective feedback to providers and regimes, reconciling raters, and preventing compensating failures remain open design questions.Source: A Limitations and Future Work · p. 16
B Human Evaluation: Ordinal Ratings
Ordinal human ratings show higher Good shares and lower Bad shares for EmoReward than SAGE-GRPO across emotional fidelity, temporal coherence, and visual quality. The broader methods and diagnostics describe how provider scores are gated, calibrated across runs, and evaluated.
B Human Evaluation: Ordinal Ratings
- The ordinal study compares GT, SAGE-GRPO, and SAGE-GRPO with EmoReward using separate Bad, Normal, or Good ratings on three criteria.It includes 18 participants and 30 prompts, with one video per prompt and method; GT is the pretrained generator baseline, not a ground-truth video.Source: B Human Evaluation: Ordinal Ratings · p. 16
- EmoReward’s Good share rises by 7.0, 2.8, and 12.2 percentage points over SAGE-GRPO for emotional fidelity, temporal coherence, and visual quality, while Bad shares fall on all three criteria.Bad shares fall from 31.5% to 15.4% for emotional fidelity, 28.9% to 20.2% for temporal coherence, and 30.9% to 15.9% for visual quality.Source: B Human Evaluation: Ordinal Ratings · pp. 16–17
- Aggregate human-rating margins do not support uncertainty estimates that account for repeated judgments by participant and prompt, or significance claims.Participant- and prompt-level records are needed for those estimates; the aggregates also do not document recruitment, presentation order, blinding, exclusions, or prompt-selection provenance.Source: B Human Evaluation: Ordinal Ratings · p. 17
C Act: Provider Interface and Reference Scoring Rules
- The Act stage scores generated videos with six primitive families, implemented through nine observation channels that return bounded scores and validity signals.The channels include facial, semantic-emotion, body/temporal, chromatic, kinetic, and motion-naturalness evidence; unavailable evidence receives a neutral score but zero effective contribution.Source: C Act: Provider Interface and… · p. 17, Table 4 · p. 18, §C.7 · p. 20
D Reflect: Instance-Dependent Gate Construction
- Reflect gates provider contributions using training phase, face evidence, semantic conflict, subtlety, and observation availability before normalizing weights over active providers.Availability masking and confidence together determine validity, with motion confidence incorporated once rather than multiplied twice.Source: D Reflect: Instance-Dependent Gate… · p. 20
E Evolve: RewardMemory Construction and Use
- RewardMemory estimates provider reliability from score–VideoAlign correlations within arousal and face-activity groups, then applies capped multipliers in the next run without changing reward models.The multipliers remain fixed during that run, while current-video reflection can still suppress unavailable or unsuitable cues.Source: E Evolve: RewardMemory Construction… · p. 21
F Derivation of the Closed-Form KL Update
- The paper distinguishes reward-allocation updates from generator optimization: EmoReward changes the scalar reward, while optimizer-specific transition and regularization choices remain unchanged.A common additive reward baseline cancels in the stated unclipped allocation solution; the bound and cold-start properties do not establish improved generalization or emotional fidelity.Source: F Derivation of the Closed-Form KL… · pp. 21–22
G.1 Reward Diagnostics
- The reported reward diagnostics include within-group ranking agreement, relative dispersion, and reliability selectivity, but these measures do not independently establish human or emotional quality.ROA measures agreement with VideoAlign, and the reported implementation’s treatment of reward ties differs from the stated symmetric half-credit convention.Source: §G.1 · p. 22
G.2 Normalized Pillars and Composite Scores
The section defines three normalized pillars and combines them into specified composite scores. It cautions that the Emotion Fidelity pillar uses a general visual-preference proxy, not an independent measure of emotional fidelity.
G.2 Normalized Pillars and Composite Scores
- Normalization is specific to the compared policy set, so the resulting composite scores are not absolute measures transferable across benchmarks.The specified composites include an equal-weight aggregation and a preference-emphasized aggregation.Source: §G.2 · p. 23
- EF is the normalized archived mean ImageReward score, a general visual-preference proxy rather than an independent measurement of emotion fidelity.The benchmark retains the “Emotion Fidelity” abbreviation, but the authors do not interpret it as measuring emotion fidelity independently.Source: §G.2 · p. 23
- GD is the normalized coefficient of variation, used as the relative-dispersion pillar.The benchmark calls it Gating Discriminativeness, but the authors state that it does not measure gradient quality.Source: §G.2 · p. 23
- GR multiplies correlation-profile concentration by agreement with the reference evaluator, but these diagnostic summaries do not establish independent reliability ground truth.The GR pillar is normalized before inclusion in the composites.Source: §G.2 · p. 23
- Full EmoReward scores 0.922 on core and 0.884 on emo, leading under both specified weightings; Reflect without memory leads EF.The comparison uses (λEF, λGD, λGR) = (0.5, 0.25, 0.25), and the results are limited to the defined composites and policy set.Source: §G.2 · p. 23
H Extended Evaluation and Ablations
The extended evaluation examines provider-pool configurations and RewardMemory, including sensitivity across memory settings and an illustrative replay. The replay distinguishes a reallocated reward calculation from the reward logged during the historical rollout.
H Extended Evaluation and Ablations
- Provider configurations show trade-offs across text alignment, preference, and semantic or motion cues, but differ in fusion weights and curricula and are not a single-factor ablation.In the semantic-cue configuration, CLIP has zero fusion weight and cannot explain a training gain attributed to an active CLIP reward.Source: H Extended Evaluation and Ablations · p. 23, Table 5 · p. 23
- The RewardMemory calibration table estimates provider reliability and unnormalized multipliers from 3,232 reflective reward traces; Macro’s association partly shares VideoAlign information with the outcome.The table caption specifies τ = 0.5 and c = 1.Source: Table 6 · p. 24
- The memory rule exceeds fixed weighting in 23 of 27 settings at the final window, but none has a positive difference averaged over all windows.Settings reuse the same traces rather than independent training seeds, so the authors frame this as sensitivity analysis, not monotonic improvement with accumulating memory.Source: H Extended Evaluation and Ablations · p. 24
I RewardMemory: A Worked Rollout Example
- The worked example uses four candidates from one prompt’s reference rollout group, illustrating local behavior rather than the population estimate used to construct RewardMemory.The agreement analysis separately covers 1,504 groups and 6,016 videos, distinct from the fixed-checkpoint evaluation sets.Source: I RewardMemory: A Worked Rollout Example · p. 24, H Extended Evaluation and Ablations · p. 24
- In the illustrative replay, reweighting changes the weighted mean from 0.578 to 0.563, while the actual logged emotion reward is 0.721.The replay uses an exported global multiplier table and illustrative initial weights, not reconstructed runtime gates; allocation changes alone do not establish improved generated quality.Source: Figure 7 · p. 24, I RewardMemory: A Worked Rollout Example · p. 25
J Generalization to Image Generation
Image-transfer results vary by objective and configuration: SD3.5–DanceGRPO improves on HPSv3, while other settings include declines and an HPSv3-optimized baseline lead. The section also describes the comparison protocols and representative video prompts.
J Generalization to Image Generation
- The image-transfer evaluation covers 256 DrawBench prompts per checkpoint across 18 settings and 84 checkpoints, using internal Emo and HPSv3 as reward measures.Neither internal Emo nor HPSv3 independently measures perceived emotion; checkpoint variation is not independent training-seed variation, and paired significance is not claimed.Source: J Generalization to Image Generation · p. 25
- SD3.5–DanceGRPO HPSv3 rises from 9.552 (baseline) to 10.096 (Reflect) and 10.302 (EmoReward), while transfer is objective-dependent rather than uniformly beneficial.The HPSv3-optimized baseline leads on HPSv3 across optimizers; SD3.5–FlowGRPO declines on both HPSv3 and internal Emo.Source: J Generalization to Image Generation · p. 25
J.1 Qualitative comparison protocol
- Image plates compare baseline, Reflect, and EmoReward on curated prompts, but unequal update budgets and unavailable image seeds limit attribution of visual differences to calibration.The first two FLUX baseline examples use earlier saved outputs; video rows instead share prompts and recorded sampling seeds.Source: §J.1 · p. 25, Figure 8 · p. 26
(c) FLUX · DanceGRPO
- The FLUX–DanceGRPO panels label baseline, Reflect, and EmoReward configurations; their captions do not report a comparative outcome..Source: (c) FLUX · DanceGRPO · p. 27
(e) FLUX · SAGE-GRPO
- The FLUX transfer figure compares HPSv3 RL, Reflective gating, and EmoReward across five backbone–optimizer settings and four shared prompts.The figure caption does not state a winner for FLUX–SAGE-GRPO.Source: Figure 9 · p. 27
K Video Comparisons
- The video-comparison captions identify DanceGRPO, FlowGRPO, and SAGE-GRPO examples spanning hesitation, a quiet exchange, unease, a café pause, stage composure, and recognition.The captions describe prompt scenes rather than reporting comparative outcomes.Source: Figure 10 · p. 28, Figure 11 · p. 28, Figure 12 · p. 29