EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

Zilong Huang; Kong Aik Lee; Zezhong Jin; Man-Wai Mak; Tatsuya Kawahara

Source-linked AI summary

EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

Zilong Huang, Kong Aik Lee, Zezhong Jin, Man-Wai Mak, Tatsuya Kawahara

arXiv:2610.13353v1Oct 9, 2026cs.MMcs.LGeess.AS

TL;DR

Open-ended emotion descriptions allow multiple valid expressions, while preference labels alone do not show where an alternative is wrong or how to correct it. EmoGavel combines construction-derived grounding supervision with preference-gated reinforcement learning. Experiments report improved natural and controlled-error preference performance alongside strong error-grounding performance.

Problem
Pairwise preference labels for open-ended emotion descriptions identify the better description but do not show which content is inconsistent or how it should be corrected.
Source: §1 · p. 1
Method
EmoGavel combines Grounded SFT on construction-derived preference and grounding targets with GA-GRPO using preference-gated grounding rewards and dynamic zero-variance rescue.
Source: §1 · p. 1
Results
EmoGavel improves natural preference prediction and controlled-error robustness while achieving strong error-grounding performance.
Source: §3.2 · p. 3, §3.3 · p. 4
Takeaways & Limitations
Preference prediction alone does not establish that a model can identify or correct erroneous content, so error grounding is a distinct outcome to evaluate.
Source: §3.3 · p. 4
GA-GRPO's evidence recovery on modality omission falls below Grounded SFT, indicating weaker alignment with omission recovery than with replacement-based localization and repair.
Source: §3.3 · p. 4

Abstract

from arXiv · show

Multimodal emotion preference learning uses pairwise comparisons to identify which open-ended emotion descriptions align more closely with human judgments. However, existing methods primarily focus on the final preference decision and provide little explicit supervision for localizing and correcting erroneous content. To address this limitation, we propose EmoGavel, a Grounding-Aware reinforcement learning framework with Verifiable rewards for multimodal Emotion preference Learning. We first introduce grounded supervised fine-tuning (Grounded SFT), which exploits controlled-error construction information to jointly learn preference prediction, error localization, repair, and missing-evidence recovery. We then optimize these verifiable outcomes with grounding-aware group relative policy optimization (GA-GRPO), combining preference-gated grounding rewards with dynamic zero-variance rescue to prioritize preference correctness and recover learning signals from all-wrong groups. Experiments on EmoPrefer and controlled-error evaluation sets show that EmoGavel improves both natural preference prediction and controlled-error discrimination while maintaining strong error-grounding performance. Our code and model checkpoints will be released once the paper is accepted.

1. INTRODUCTION

Pairwise preference prediction helps evaluate open-ended emotion descriptions, but does not reveal which content is emotionally inconsistent or how to correct it. EmoGavel uses construction-derived grounding supervision and reinforcement learning to extend preference learning with explicit error-grounding outcomes.

Pairwise preference prediction addresses the difficulty of evaluating open-ended emotion descriptions, but does not identify emotionally inconsistent content or how to correct it.Source: §1 · p. 1
Prior SFT/DPO judges and EAPO's controlled negatives [17] still optimize primarily for binary preference, not error localization or repair.Source: §1 · p. 1
Because each controlled negative modifies a preferred description, its construction preserves original-to-corrupted correspondence beyond the resulting preference label.Source: §1 · p. 1
EmoGavel combines Grounded SFT with GA-GRPO to learn preference prediction, error localization, repair, and missing-evidence recovery using verifiable outcomes.GA-GRPO uses preference-gated grounding rewards and dynamic zero-variance rescue to recover learning signals from all-wrong groups.Source: §1 · p. 1

2. METHOD

EmoGavel combines grounded supervised fine-tuning with Grounding-Aware GRPO to jointly optimize emotion preference prediction and error grounding. Controlled-error construction supplies targets for localizing, repairing, or recovering missing content, while preference-gated rewards and zero-variance rescue shape reinforcement learning.

2.1. Overall Architecture

EmoGavel uses Grounded SFT to learn preference prediction and error grounding, then GA-GRPO to optimize their verifiable outcomes.Grounded SFT initializes the trainable and reference policies; GA-GRPO samples outputs, computes rewards and group-relative advantages, and updates the policy with KL regularization.Source: §2.1 · p. 1, Figure 1 · p. 2

2.2. Task Formulation and Grounding Supervision

Controlled-error pairs add grounding targets to preference prediction: replacement errors specify corrupted spans and repairs, while modality omissions specify missing evidence.The targets follow directly from the construction process; natural pairs retain preference supervision only.Source: §2.2 · p. 2

2.3. Grounded Supervised Fine-Tuning (Grounded SFT)

Grounded SFT trains preference decisions alongside error localization and repair, or missing-evidence recovery, using structured targets and autoregressive cross-entropy.Binary and structured examples share the objective but differ in target format; LoRA initializes both the trainable and frozen reference policies for GA-GRPO.Source: §2.3 · p. 2

2.4.1. Verifiable Preference and Grounding Rewards

Verifiable rewards score replacement-error localization and repair with token-level F1, and score omission recovery with F1 plus frozen-encoder semantic similarity.Grounding rewards apply to controlled-error examples only when the preference prediction is correct; natural pairs receive preference-only rewards.Source: §2.4.1 · pp. 2–3

2.4.2. Dynamic Zero-Variance Rescue

When every sampled binary preference output is wrong, dynamic rescue adds the ground-truth output, suppresses incorrect outputs, and reinforces the correct label.Other groups retain the original GRPO advantages; the update uses KL regularization to the frozen Grounded SFT reference policy.Source: §2.4.2 · p. 3

3. EXPERIMENTS

EmoGavel performs strongly on natural and controlled-error preference evaluation, while explicit grounding supervision improves localization, repair, and recovery. Ablations show distinct contributions from grounding reward, dynamic rescue, and preference gating.

3.1. Experimental Setup

Grounded SFT uses EmoPrefer-Data-V2 for development, a video-ID 90/10 train-validation split, and EmoPrefer-Data plus its EAPO extension for final evaluation.The evaluation covers natural and controlled-error preference, swap consistency, and error grounding; the base model is Qwen2.5-Omni-7B.Source: §3.1 · p. 3

3.2. Analysis of Preference Robustness

79.75% WAF on natural pairs, 88.20% 4-Error Avg, and 81.25% Swap Cons make EmoGavel the best single-model judge and strongest on controlled-error robustness and order consistency.EAPO remains slightly higher on natural pairs by fusing three adapted judges.Source: §3.2 · p. 3, Table 1 · p. 4

3.3. Analysis of Error Grounding

Explicit grounding supervision closes the preference-only gap, and GA-GRPO raises Grounding to 77.27%, though Modality Omission Evid. Recovery falls below Grounded SFT.Table 2 reports localization, repair, missing-evidence recovery, parse rate, preference, and macro Grounding across training strategies.Source: §3.3 · p. 4, Table 2 · p. 4

3.4. Ablation Studies on GA-GRPO

A grounding reward raises Grounding from 72.73% to 76.32%, with little change in preference performance; vanilla GRPO offers only modest improvement over Grounded SFT, indicating preference reward alone is insufficient to substantially enhance error grounding.Source: §3.4 · p. 4
Combining both components achieves 79.75% Natural and 88.20% 4-Error Avg while maintaining 77.27% Grounding, indicating complementary benefits.Table 3 presents the GA-GRPO ablation results.Source: §3.4 · p. 4, Table 3 · p. 4

4. CONCLUSION

EmoGavel extends multimodal emotion preference learning with verifiable error grounding through Grounded SFT and GA-GRPO. Experiments report strong natural and controlled-error preference results alongside grounding capability, while future work targets less explicitly aligned errors and varied multimodal evidence.

EmoGavel extends preference learning beyond selecting a preferred description to verifiable error grounding.The framework combines Grounded SFT with GA-GRPO to learn preference prediction, localization, repair, and missing-evidence recovery.Source: §4 · p. 4
A single Qwen2.5-Omni-7B judge achieves 79.75% WAF on natural pairs, 88.20% average WAF across four controlled-error types, and 72.27% Grounding.These results are reported as a balance of preference prediction, controlled-error robustness, and grounding capability.Source: §4 · p. 4
Future work will extend controlled errors to more naturally occurring, less explicitly aligned errors and examine grounding under different multimodal evidence conditions.Source: §4 · p. 4