PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

Junjie Wu, Xuan Kan, Zihao He, Shunwen Tan, Bo Pan, Kaitai Zhang

arXiv:2603.11665v2cs.CL

TL;DR

Many trainable MLLM judges specialize in narrow domains, limiting generalization across evaluation scenarios. MT-RL-Judge jointly trains across tasks with reinforcement learning, outperforming strong baselines across six tasks and generalizing to unseen pairwise formats.

  • Problem

    Trainable MLLM judges are often specialized for narrow domains, limiting their generalization across diverse evaluation scenarios.

    Source: §1 · p. 1

  • Method

    MT-RL-Judge jointly trains a judge across diverse tasks with reinforcement learning and rewards for output format and prediction accuracy.

    Source: §3.2 · p. 3, §5 · p. 6

  • Results

    Across six tasks, MT-RL-Judge consistently outperforms strong baselines and generalizes robustly to unseen pairwise formats.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    The authors identify unified multi-task RL judging as a promising direction for scalable automated quality assurance.

    Source: §1 · p. 2, §5 · p. 6

    The paper notes that SFT can mimic surface correlations rather than judging logic, constraining generalization and encouraging overfitting to specific prompt templates.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have been widely adopted as MLLM-as-a-Judges due to their strong alignment with human judgment across various visual tasks. However, most existing judge models are optimized for single-task scenarios and struggle to generalize to diverse contexts, which is a critical requirement for reliable evaluation. To address this limitation, we propose Multi-Task Reinforcement Learning for MLLM-as-a-Judge (MT-RL-Judge), a framework that jointly optimizes the judge model across multiple tasks, leveraging the generalization capabilities of RL. Experimental results against several strong baselines demonstrate that MT-RL-Judge outperforms strong baselines in both judgment consistency and correlation with human preferences. Furthermore, our approach exhibits robust generalization on out-of-distribution tasks, further validating its effectiveness.

1 Introduction

MT-RL-Judge is a unified multi-task reinforcement-learning framework for MLLM-as-a-Judge, designed to address the scalability and generalization limits of specialized judges. The paper reports stronger performance across evaluation tasks and on unseen formats.

  • Human evaluation is reliable but costly to scale, motivating MLLM-as-a-Judge for large-scale assessment of synthetic multimodal content.The introduction frames content quality and safety as important industrial needs, including customer service and advertisement generation (Liu et al., 2023; Fu et al., 2024), and describes MLLM judges as an automated alternative (Chen et al., 2024a; Wang et al., 2025; Pu et al., 2025).Source: §1 · p. 1
  • Existing judges often need task-specific training, specialize in narrow domains, and SFT models can overfit instruction formats, limiting generalization.The introduction notes that an SFT judge trained on pointwise alignment struggles with pairwise comparisons, making it brittle as requirements change (Chen et al., 2024a; Zhou et al., 2025; Luera et al., 2025; Wang et al., 2025; Gu et al., 2025).Source: §1 · p. 1
  • MT-RL-Judge uses multi-task RL and GRPO to train one judge across diverse tasks and formats, generating reasoning steps before its verdict.The paper says this approach is intended to internalize evaluation logic rather than memorize superficial input-label mappings, improving judgment quality and explainability.Source: §1 · p. 1
  • A unified judge eliminates switching among specialized models for heterogeneous inputs, streamlining inference and reducing deployment costs.Source: §1 · p. 2
  • MT-RL-Judge yields superior results to single-task specialists and significantly outperforms SFT counterparts on MJ-Bench formats unseen during training.The paper presents this as evidence that joint training preserves effectiveness while improving generalization to novel scenarios without retraining.Source: §1 · p. 2
  • The authors present MT-RL-Judge as the first unified RL-based MLLM-as-a-Judge framework intended to generalize across diverse evaluation tasks.Source: §1 · p. 2

2 Related Works

Because reference-based metrics often correlate weakly with human preferences, evaluation shifted toward LLM-as-a-Judge and its multimodal extension, MLLM-as-a-Judge. Existing approaches use prompts or task-specific fine-tuning, but fine-tuned judges often generalize poorly and remain specialized for single tasks.

Related Works

  • Reference-based metrics often correlate weakly with human preferences, motivating LLM-as-a-Judge and its multimodal extension, MLLM-as-a-Judge.The paradigm prompts capable LLMs to evaluate outputs using task-specific rubrics; MLLMs extend it to diverse sensory inputs.Source: §2 · p. 2
  • Prompt-based judges use rubrics, CoT, and demonstrations, while fine-tuned judges address task complexity but often generalize poorly and remain single-task specialists.Fine-tuning uses SFT or RL; specialized judges can impose high maintenance and inference costs in large-scale commercial systems.Source: §2 · p. 2

3 Our Method

MT-RL-Judge jointly optimizes an MLLM judge across aggregated evaluation datasets, using reinforcement learning to encourage reasoning-first outputs and reliable judgments. Its training combines format and accuracy rewards with Group Relative Policy Optimization.

3.1 Problem Formulation

  • The unified formulation aggregates multiple evaluation datasets and jointly optimizes the judge across them to address single-task models’ limited scalability and generalization.The task is to predict human-annotated labels from multimodal inputs and prompts.Source: §3.1 · pp. 2–3
  • Unified SFT can overfit training prompt templates because maximum-likelihood training may capture surface correlations instead of reasoning logic.This motivates the framework’s emphasis on reasoning-based generalization.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 MT-RL-Judge

  • MT-RL-Judge optimizes a global policy for expected composite reward across diverse judging tasks, aiming for reasoning that transfers across contexts.This shifts the objective beyond fitting individual dataset distributions.Source: §3.2 · p. 3
  • MT-RL-Judge combines format reward for reasoning-first structure with accuracy reward for reasoning traces that lead to correct judgments.A weighting hyperparameter sets the relative importance of the two reward components.Source: §3.2 · pp. 3–4
  • GRPO optimizes the judge without a separate value function, using average rewards from multiple outputs sampled for each unified-dataset prompt.The method seeks to maximize expected reward across the entire unified dataset rather than a single task.Source: §3.2 · p. 4
  • The authors report that unified optimization yields high-quality evaluations across diverse tasks, with interpretable reasoning traces and alignment with human preferences.They characterize the combination of accuracy and transparency as supporting reliable industrial deployment.Source: §3.2 · p. 4

4 Experiments

MT-RL-Judge combines multi-task training with reinforcement learning and achieves the best overall performance across diverse benchmarks. It also generalizes to held-out pairwise evaluations, where RL-based reasoning helps it apply learned criteria in a new input format.

4.1 Datasets

  • The evaluation uses six datasets across text-image alignment, safety compliance, and visual quality assessment.The datasets are SeeTRUE and ImageReward for alignment, UnsafeBench for safety, and AGIN’s Naturalness, Rationality, and Technical Quality subsets for visual quality.Source: §4.1 · p. 4

4.2 Setting

  • Experiments compare MT-RL-Judge with zero-shot, single-task SFT, unified SFT, and single-task RL judges using Qwen3-VL-30B-A3B-Instruct and Macro-F1.All baselines use the same foundational backbone; Macro-F1 is the primary metric because the evaluation tasks are mainly binary classification problems.Source: §4.2 · pp. 4–5

4.3 Main Results

  • MT-RL-Judge achieves the best overall performance across diverse benchmarks, including 83.67 on SeeTrue, while SFT-Single is marginally higher only on UnsafeBench.The authors attribute the UnsafeBench exception to possible memorization of dataset-specific safety patterns shared between training and test splits.Source: Table 2 · p. 5, §4.3 · p. 5
  • RL-Single surpasses SFT-Single on 5 of 6 benchmarks, with gains of +3.0 on SeeTrue and +4.63 on AGIN-Rationality.The authors attribute the gains to RL incentivizing rigorous logical deduction before prediction, particularly for reasoning-intensive tasks.Source: §4.3 · p. 5
  • SFT-Unified outperforms SFT-Single on most tasks, scoring 81.75 versus 78.64 on AGIN-Nat.The authors suggest that multi-task exposure helps capture shared evaluation criteria and latent cross-domain correlations, reducing overfitting to task-specific prompts.Source: §4.3 · p. 5

4.4 MT-RL-Judge Enhances Generalizability

  • MJ-Bench tests generalization to pairwise image comparisons using image-text matching and safety tasks held out from all evaluated judges’ training corpora.The models were trained on pointwise binary judgments; the pairwise setup probes whether they learned task semantics rather than memorized training prompts.Source: §4.4 · pp. 5–6
  • On MJ-Bench, MT-RL-Judge scores 60.59% on Alignment and 82.23% on Safety in an unseen pairwise format.The authors attribute this generalization to RL-driven reasoning that abstracts evaluation criteria and applies them to novel task formulations.Source: Table 3 · p. 6, §4.4 · p. 6
  • SFT-Unified scores 49.40% on the MJ-Bench Safety Judge task, below the zero-shot Off-the-shelf baseline at 73.07%.The authors interpret this degradation as overfitting to the single-image input structure used in training.Source: Table 3 · p. 6, §4.4 · p. 6

5 Conclusion

MT-RL-Judge jointly optimizes diverse evaluation tasks using a composite reward for structural format and prediction accuracy, aiming to learn underlying judging logic rather than overfit to instruction formats. Across six tasks it outperforms strong baselines and generalizes robustly to unseen pairwise formats.

5 Conclusion

  • MT-RL-Judge jointly optimizes diverse evaluation tasks with a composite reward for structural format and prediction accuracy.The framework is designed to encourage learning underlying judging logic rather than overfitting to surface-level instruction formats.Source: §5 · p. 6
  • Across six distinct tasks, MT-RL-Judge consistently outperforms various strong baselines.Source: §5 · p. 6
  • MT-RL-Judge generalizes robustly to unseen pairwise formats, indicating resilience to distribution shifts.The paper presents unified multi-task RL judging as a promising direction for scalable and reliable multimodal evaluation in industrial applications.Source: §5 · p. 6

6 Ethical Considerations

Although the approach targets industrial deployment, the experiments use only publicly available datasets, and the work introduces no additional ethical considerations.

  • The approach targets industrial deployment, but its experiments use only publicly available datasets and introduce no additional ethical considerations.Source: §6 · p. 6

Appendix · A Training Configurations

The appendix specifies the software, base model, and selection procedures used for supervised fine-tuning (SFT) and reinforcement learning (RL). SFT uses full-parameter fine-tuning, while RL primarily uses GRPO with 20 generations per prompt.

A Training Configurations

  • All judge models start from Qwen/Qwen3-VL-30B-A3B-Instruct, with LLaMA-Factory used for SFT and EasyR1 for RL.The paper identifies Zheng et al. (2024) for LLaMA-Factory and Yaowei Zheng (2025) for EasyR1.Source: A Training Configurations · p. 9
  • SFT uses full-parameter fine-tuning with AdamW and a cosine learning-rate schedule, plus Flash Attention 2 for high-resolution visual inputs.SFT hyperparameters are listed in Table 4.Source: A Training Configurations · p. 9, Table 4 · p. 9
  • SFT continues until validation performance plateaus, and the checkpoint with the highest validation score is selected for downstream experiments.Unlisted SFT hyperparameters follow LLaMA-Factory defaults.Source: A Training Configurations · p. 9
  • The SFT hyperparameter configuration is provided in Table 4.Source: Table 4 · p. 9
  • RL primarily uses Group Relative Policy Optimization (GRPO), with 20 generations per prompt to support exploration.Its specific configuration is provided in Table 5.Source: A Training Configurations · p. 9, Table 5 · p. 9
  • RL training continues until validation reward plateaus, after which the checkpoint with the highest accuracy reward is selected.Unlisted RL hyperparameters follow Easy-R1 defaults.Source: A Training Configurations · p. 9
  • The RL hyperparameter configuration is provided in Table 5.Source: Table 5 · p. 9

B Prompt for all the Tasks

This appendix presents the SFT and RL prompts used for six experimental tasks, with separate prompt figures for each task and training stage. The prompt descriptions cover judgments about image safety, rationality, technical quality, naturalness, and text–image alignment.

Loading 2603.11665v2…