Source-linked AI summary
Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
Junjie Wu, Xuan Kan, Zihao He, Shunwen Tan, Bo Pan, Kaitai Zhang
TL;DR
Existing MLLM judges are difficult to scale across diverse tasks because many are specialized and can overfit to surface-level instruction formats. MT-RL-Judge jointly optimizes multiple evaluation tasks with reinforcement learning, combining format and accuracy rewards while encouraging reasoning-first judgments. Across six tasks it consistently outperforms strong baselines and generalizes robustly to unseen pairwise formats, although SFT can generalize poorly when prompt templates change.
Problem
Existing MLLM-as-a-Judge models are often optimized for narrow tasks and may overfit specific prompt templates, limiting generalization across diverse evaluation scenarios.
Method
MT-RL-Judge jointly trains one judge across diverse evaluation tasks using reinforcement learning with format and accuracy rewards.
Results
Across six distinct tasks, MT-RL-Judge consistently outperforms strong baselines and shows robust out-of-domain generalization on unseen pairwise formats.
Takeaways & Limitations
Unified multi-task RL is a promising direction for scalable, reliable MLLM-based evaluation across heterogeneous and distribution-shifted tasks.
Takeaways & Limitations
SFT-based judges can overfit specific prompt templates because standard SFT encourages surface-level input-output correlations rather than underlying judgment logic.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have been widely adopted as MLLM-as-a-Judges due to their strong alignment with human judgment across various visual tasks. However, most existing judge models are optimized for single-task scenarios and struggle to generalize to diverse contexts, which is a critical requirement for reliable evaluation. To address this limitation, we propose Multi-Task Reinforcement Learning for MLLM-as-a-Judge (MT-RL-Judge), a framework that jointly optimizes the judge model across multiple tasks, leveraging the generalization capabilities of RL. Experimental results against several strong baselines demonstrate that MT-RL-Judge outperforms strong baselines in both judgment consistency and correlation with human preferences. Furthermore, our approach exhibits robust generalization on out-of-distribution tasks, further validating its effectiveness.
1 Introduction
MLLM-as-a-Judge addresses the costly, hard-to-scale evaluation of multimodal content, but existing judges struggle with task diversity and prompt-format generalization. MT-RL-Judge uses unified multi-task reinforcement learning to improve efficiency, judgment quality, explainability, and out-of-distribution robustness.
- Human evaluation is reliable but prohibitively expensive and difficult to scale, motivating automated MLLM-based evaluation for multimodal outputs.
- Existing judges often require task-specific training, specialize in narrow domains, and overfit to particular instruction formats.
- MT-RL-Judge jointly trains one judge across diverse tasks and input-output formats using reinforcement learning rather than relying on specialized single-task models.GRPO encourages the model to internalize evaluation logic and generate reasoning before its final verdict.
- Efficiency: A unified judge eliminates switching among specialized models, streamlining inference and reducing deployment costs.
- Effectiveness: Joint training does not compromise performance against single-task specialists and instead yields superior judgment results, improving automated quality-assurance reliability.
- Generalization: On unseen pairwise formats in MJ-Bench, MT-RL-Judge significantly outperforms SFT counterparts without retraining, demonstrating strong generalization to novel evaluation scenarios.
- The authors describe MT-RL-Judge as, to their knowledge, the first unified RL-based MLLM judge designed to generalize across diverse evaluation tasks.
2 Related Works
Prior work includes prompt-based judges that rely on off-the-shelf MLLMs and fine-tuned judges that update model parameters. Prompt-based methods use rubrics, reasoning paths, and demonstrations to guide evaluation.
- LLM-as-a-Judge emerged partly because BLEU, ROUGE, and BERTScore can correlate weakly with human preferences for open-ended outputs.
- Prompt-based Judges: Prompt-based judges use off-the-shelf MLLMs without parameter updates, relying solely on prompt engineering for evaluation guidance.
- Prompt-based Judges: Their prompts may include detailed rubrics, Chain-of-Thought reasoning paths, and in-context demonstrations.
- Judge Taxonomy: The related-work taxonomy distinguishes prompt-based judges from fine-tuned judges that update model parameters.
3 Our Method
MT-RL-Judge unifies diverse evaluation datasets and jointly trains an MLLM judge with reinforcement learning to improve scalability, generalization, and judgment reliability. Its reward design combines output-format compliance with judgment accuracy, while GRPO optimizes expected reward across tasks.
- Problem Formulation: Existing unified judges aggregate multiple evaluation datasets, but SFT can encourage surface-level input-output memorization and overfitting to training prompt templates.These limitations constrain generalization beyond specific dataset distributions and prompt formats.
- MT-RL-Judge: MT-RL-Judge jointly optimizes one judge across diverse tasks and varying input-output formats using multi-task reinforcement learning.The framework targets a global policy whose expected reward is maximized across the unified dataset rather than on an isolated task.
- Reward Function: The reward function combines a format reward enforcing reasoning-first outputs with an accuracy reward for reasoning traces that culminate in correct judgments.The total reward is a weighted combination of these complementary components, controlled by α.
- Training Objective: GRPO generates multiple outputs for each prompt and uses their average reward to optimize the judge without a separate value function.This training objective computes expected reward across the entire unified dataset.
- Training Objective: Unified optimization produces high-quality evaluations across diverse tasks, with explicit reasoning traces supporting interpretability, reliability, and alignment with human preferences.The stated deployment scope includes industrial applications.
4 Experiments
The experiments evaluate MT-RL-Judge across six datasets covering alignment, safety compliance, and visual quality, comparing zero-shot, SFT, and RL-based judges. MT-RL-Judge performs strongly on in-domain tasks and generalizes to unseen pairwise formats on MJ-Bench.
- Experimental Setup: The evaluation spans six benchmark datasets covering text-image alignment, safety compliance, and visual quality assessment.SeeTRUE and ImageReward assess semantic consistency; UnsafeBench assesses harmful visual content; AGIN provides Naturalness, Rationality, and Technical Quality subsets.
- Experimental Setup: Macro-F1 is the primary metric, and baselines include an off-the-shelf MLLM, single-task SFT judges, and unified SFT judges.The models use the same foundational backbone, with task-specific fine-tuning comparisons included.
- Main Results: RL-Single surpasses SFT-Single on 5 of 6 benchmarks, including gains of +3.0 on SeeTrue and +4.63 on AGIN-Rationality.The paper attributes these improvements to the reasoning-intensive nature of the tasks and RL’s incentive for logical deduction.
- Main Results: SFT-Unified outperforms SFT-Single on most tasks, reaching 81.75 versus 78.64 on AGIN-Nat.The paper links this pattern to shared evaluation criteria and latent correlations across domains.
- Main Results: MT-RL-Judge achieves the best overall performance across diverse benchmarks, including 83.67 on SeeTrue, while remaining competitive on UnsafeBench.SFT-Single is marginally higher on UnsafeBench, which the paper associates with memorization of dataset-specific safety patterns.
- Out-of-Domain Generalization: On MJ-Bench’s unseen pairwise format, MT-RL-Judge achieves 60.59% on Alignment and 82.23% on Safety, whereas SFT-Unified’s Safety performance falls to 49.40%.MJ-Bench tests pairwise comparison despite training focused exclusively on pointwise evaluation; the zero-shot Off-the-shelf baseline reaches 73.07% on Safety.
5 Conclusion
The paper proposes MT-RL-Judge, a unified multi-task reinforcement learning framework for MLLM-as-a-Judge evaluators. Across six tasks, it outperforms strong baselines and generalizes to unseen pairwise formats.
- 5 Conclusion: MT-RL-Judge jointly optimizes diverse evaluation tasks with a composite reward for structural format and prediction accuracy.The framework is designed to internalize judging logic rather than overfit to surface-level instruction formats.
- 5 Conclusion: Across six distinct tasks, MT-RL-Judge consistently outperforms various strong baselines.
- 5 Conclusion: MT-RL-Judge shows robust out-of-domain generalization on unseen pairwise formats, indicating resilience to distribution shifts.
6 Ethical Considerations
The approach targets industrial deployment, but its experiments use only publicly available datasets. The manuscript therefore reports no additional ethical considerations.
- 6 Ethical Considerations: The proposed approach is targeted for deployment in an industrial context.
- 6 Ethical Considerations: Because the experiments rely solely on publicly available datasets, the manuscript reports no additional ethical considerations.
A Training Configurations
The training configuration uses full-parameter SFT and GRPO-based reinforcement learning with Qwen3-VL-30B-A3B-Instruct initialization. Training selects checkpoints using validation performance or reward plateaus.
- Initialization: All MLLM-as-a-Judge models are initialized from Qwen/Qwen3-VL-30B-A3B-Instruct.
- SFT: SFT uses full-parameter fine-tuning with AdamW, a cosine learning-rate schedule, and Flash Attention 2 for high-resolution visual inputs.Training continues until validation performance plateaus, after which the highest-validation-score checkpoint is selected.
- RL: The RL stage primarily uses Group Relative Policy Optimization with 20 generations per prompt to provide sufficient exploration.Training continues until validation reward plateaus, and the checkpoint with the highest accuracy reward is selected.
- RL: The reported configuration uses bfloat16 precision, AdamW, global batch size 256, rollout batch size 512, and maximum image resolution 4,194,304 pixels.
B Prompt for all the Tasks
The appendix presents the prompts used for six evaluation tasks, distinguishing supervised fine-tuning (SFT) prompts from reinforcement-learning (RL) prompts. The tasks cover safety, image rationality, text-image alignment, technical quality, and naturalness evaluation.
- Prompt organization: The experiments use separate SFT and RL prompts for six tasks listed in Table 1.Figures 1–6 show SFT prompts, while Figures 7–12 show the corresponding RL prompts.
- Task prompts: Unsafe Bench prompts ask whether an image contains safe or unsafe content.This task is represented in the SFT and RL prompt sets.
- Task prompts: AGIN-Rat prompts ask whether an image is rational.The prompt frames the task as evaluating logical consistency and rationality in images.
- Task prompts: SeeTrue prompts evaluate whether a text description matches an image, while Image Reward prompts evaluate technical image quality and naturalness.The supplied prompts associate these tasks with visual-linguistic alignment, technical quality, and distinguishing natural-looking images from AI-generated images.
- Task prompts: AGIN-Tech prompts ask whether a text description matches a given image.The same visual-linguistic evaluation framing is used for this text-image alignment task.