Source-linked AI summary
When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning
Zhengxian Wu, Kai Shi, Chuanrui Zhang, Zirui Liao, Jun Yang, Ni Yang, Qiuying Peng, Luyuan Zhang, Hangrui Xu, Tianhuang Su, Zhenyu Yang, Haonan Lu, Haoqian Wang
TL;DR
The paper addresses the costly, difficult-to-scale reliance on annotated answers and teacher-model supervision for multimodal reasoning. It proposes unsupervised self-evolution that combines Actor self-consistency, bounded Judge modulation, and group-relative optimization, achieving stable improvements across mathematical reasoning benchmarks without human-annotated answers or external supervision.
Problem
Multimodal self-evolution lacks reliable supervision, so noisy consistency signals can amplify dominant errors, collapse response lengths, and destabilize long-term training.
Method
The framework samples multiple trajectories with an Actor, applies bounded Judge modulation, models group-wise reward distributions, and optimizes relative advantages with GRPO.
Results
The method consistently improves reasoning performance across multiple mathematical benchmarks without human-annotated answers, including a 3.3-point average-accuracy gain on Geo3K-trained models.
Takeaways & Limitations
Jointly modeling trajectories and their relative quality yields stable and sustained multimodal reasoning improvements in unsupervised training.
Takeaways & Limitations
The system remains bounded by the Judge’s capability and does not yet determine how to raise evaluation standards or update the Judge autonomously.
Abstract
from arXiv · showhide
Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.
1 Introduction
Multimodal self-evolution seeks to reduce costly dependence on annotations and teacher models, but consistency-based signals can amplify incorrect dominant answers and destabilize training. The proposed Actor–Judge framework reshapes trajectory rewards and improves reasoning performance through collaborative modeling.
- Motivation: High-quality annotations, reasoning traces, and teacher-model distillation make multimodal supervision costly and difficult to scale.The paper motivates self-evolving post-training as a way to reduce reliance on these supervision sources.
- Limitations of Existing Self-Evolution: Consistency-based self-training can reinforce systematic errors because the most frequent answer is not necessarily the highest-quality reasoning path.Majority voting discards structural information about secondary modes and can suppress exploration.
- Limitations of Existing Self-Evolution: Reward concentration can cause early optimization saturation, response-length collapse, and a low-entropy output distribution during long-term training.The paper identifies these effects as consequences of coarse training signals that fail to capture fine-grained differences between candidates.
- Proposed Framework: The framework uses an Actor to sample multiple reasoning trajectories and a Judge to apply bounded, continuously differentiable modulation based on trajectory quality.This modulation reshapes the Actor’s initial self-consistency distribution before optimization.
- Results: On MathVision, unsupervised post-training improves accuracy by up to +5.9 absolute points, reaching 30.9% versus 25.0%.The paper also reports healthier entropy trajectories and reduced response-length collapse across benchmarks.
- Proposed Framework: The method models within-input relative structure among candidate solutions to mitigate common failure modes in unsupervised self-evolution.The paper presents collaborative modeling as a way to improve training stability and preserve more informative differences among trajectories.
2 Related work
Related work explores verifiable rewards, reflection, critics, and test-time scaling as alternatives to human-provided supervision. Unsupervised self-evolution applies these ideas to multimodal reasoning, with majority voting serving as a relative correctness signal despite its limitations.
- Verifiable Rewards and Reflection: Multimodal reinforcement-learning work increasingly uses objectively verifiable signals, structured reflection, or explicit critics instead of subjective human preferences.These approaches aim to support more stable reasoning optimization and evaluation.
- Unsupervised Self-Evolution: Unsupervised self-evolution uses test-time scaling strategies such as majority voting to obtain relative correctness signals without ground-truth answers.The related work identifies this strategy in both large language models and vision-language models.
3 Method
The framework combines Actor self-consistency, bounded Judge modulation, and group-wise distributional optimization to produce more stable unsupervised multimodal reasoning updates. It uses GRPO to reallocate probability within trajectory groups rather than simply pursuing absolute scores.
- Consistency-Based Initial Reward: The Actor samples multiple reasoning trajectories for each input, using answer-frequency consistency as an initial reward signal.Agreement among trajectories raises the empirical probability and reward assigned to their shared answer.
- Distributional Reward Modeling: The complete framework jointly models repeated rollouts, bounded Judge modulation, and group-wise reward distributions to support stable policy updates without external supervision.The group-relative advantage can be interpreted as matching the policy to a reward-induced target distribution.
- Consistency-Based Initial Reward: Majority voting provides denoising but discards distributional structure, allowing early dominant answers to suppress exploration and drive low-entropy policies.Consistency-based rewards retain relative empirical strengths, producing a smoother training signal.
- Judge Calibration: A frozen Judge evaluates correctness, reasoning quality, and visual grounding, then continuously modulates rather than replaces the Actor’s consistency reward.The Judge is initialized as a structurally identical copy of the Actor and remains fixed during training.
- Judge Calibration: The modulation function is bounded and continuously differentiable, encouraging high-scoring trajectories while suppressing low-scoring ones without amplifying Judge noise.Its thresholds, smoothness parameters, and amplification or suppression limits control the modulation behavior.
- Judge Calibration: Using raw Judge scores directly can destabilize training because input-dependent score scales let a few trajectories dominate and amplify Judge bias.This can prematurely steer the policy toward the Judge’s preference.
- Distributional Reward Modeling: GRPO converts modulated rewards into group-relative advantages, implicitly defining a reward-induced target distribution over trajectories for probability reallocation within each input group.This shifts optimization away from absolute high scores and reduces self-reinforcement of early dominant modes.
4 Experiments
Experiments evaluate the framework across multimodal mathematical benchmarks, training dynamics, ablations, generalization tasks, model scales, and computational cost. Without human-annotated answers, the method improves accuracy, stabilizes training, generalizes beyond mathematics, and incurs moderate overhead.
- Datasets and baselines: Experiments use Geometry3k, GeoQA, and MMR1 for training, and MathVision, MathVerse, WeMath, LogicVista, and DynaMath for evaluation.The study compares against unsupervised self-evolving, supervised, and strong-model-distillation baselines.
- Training setup: The actor is trained with GRPO while the Judge remains frozen, using eight sampled trajectories per question.Both models are initialized from Qwen2.5-VL-7B-Instruct.
- Main results: Without human-annotated answers, Geo3K training raises average accuracy from 34.6 to 37.9 (+3.3) across benchmarks.On MathVision, the method achieves up to 30.9 versus 25.0, an absolute improvement of 5.9 points.
- Training dynamics: The method maintains healthier entropy and response-length trajectories than supervised RL, avoiding repeated reinforcement of early dominant patterns.Majority voting rapidly reduces policy entropy during MMR1 training.
- Ablation study: Self-Consistency alone cannot reliably distinguish high- and low-quality trajectories, whereas Judge-only training can become unstable and increase response length.The full method redistributes probability mass within each candidate set and achieves the best ablation performance.
- Generalization and cost: The method remains effective across model sizes, outperforms majority voting on pass@10, improves a teacher-distilled baseline, and generalizes to ChartQA and MMVP.Its relative training time is 1.4× supervised GRPO, whose training time is approximately 10.5 hours.
5 Conclusion
The paper concludes that its unsupervised self-evolution framework stabilizes multimodal training by jointly modeling multiple trajectories with self-consistency, Judge modulation, and group-wise reward modeling. Experiments across mathematical reasoning benchmarks show stable performance improvements.
- Conclusion: The framework jointly models multiple reasoning trajectories using Actor self-consistency, Judge-based modulation, and group-wise distributional reward modeling.The design aims to reduce mode collapse during long-term training.
- Conclusion: Experiments on multiple mathematical reasoning benchmarks show stable performance improvements.
Limitations
The work focuses on constructing stable training signals and does not yet address improvement beyond the Judge’s capability limit. Future self-evolution requires progressively raising evaluation standards and deciding when to update the Judge.
- Scope boundary: The framework does not yet improve the self-evolving system beyond the Judge’s capability limit.The authors leave this problem for future study.
- Future direction: Sustained self-evolution requires the Judge to raise its evaluation standards and autonomously determine when it should be updated.
A Why Group-wise Distributional Modeling Prevents Policy Collapse
The method replaces single-candidate selection with a soft, reward-induced distribution over trajectories sampled for each input. This preserves probability mass across comparable candidates and supports gradual within-group updates rather than early mode collapse.
- Group-wise reward modeling: For each input, candidate trajectories are sampled from the behavior policy and assigned scalar rewards before group-relative modeling.The framework operates on a candidate set T(x) generated under πθold(· | x).
- Group-wise reward modeling: Energy scaling and a group-wise log-sum-exp baseline convert trajectory rewards into group-relative advantages.The temperature parameter α controls the energy scaling used before defining the baseline and advantages.
- Distributional target: The induced target qα(· | x) makes the learning target a soft distribution over candidates instead of a single selected trajectory.The group-relative advantage equals the log-probability under the reward-induced distribution.
- Distributional target: When one candidate dominates in reward, qα becomes deterministic; otherwise, it remains non-degenerate and assigns mass to several candidates.Sharpness depends on α and reward gaps within the group.
- Policy update: Matching the policy to qα continuously reshapes probability mass within each group rather than concentrating all mass on one candidate early in training.This contrasts with one-hot targets, which directly encourage deterministic mappings.
B Training Algorithm
The training algorithm combines multi-trajectory sampling, self-consistency rewards, Judge-based modulation, group-wise reward shaping, and GRPO policy optimization.
- Training Algorithm: Algorithm 1 summarizes the overall unsupervised self-evolution training procedure.The procedure is provided to clarify implementation and facilitate reproducibility.
- Training Algorithm: The procedure includes multi-trajectory sampling, self-consistency reward initialization, Judge-based score modulation, group-wise distributional shaping, and GRPO optimization.
C.1 Training Data
The experiments use established multimodal reasoning resources and an unlabeled QA training split, alongside zero-human-supervision baselines based on self-generated signals.
- Training Data: Geo3K contains 3,002 diagram-grounded geometry questions split into 2,101 training, 300 validation, and 601 test examples.Each example pairs a natural-language problem, geometry diagram, and ground-truth answer.
- Training Data: GeoQA contains 5,010 planar-geometry problems with a 7:1.5:1.5 train-validation-test split and three problem categories.The benchmark requires joint interpretation of problem text and diagrams.
- Training Data: The RL-stage QA split of MMR1 is used as the unlabeled training set for unsupervised self-evolution.MMR1 also contains a cold-start set with long chain-of-thought annotations.
- Baselines: Vision-Zero generates training signals through self-play interactions on pairs of original and edited images.The comparisons include variants trained on synthetic CLEVR scenes and real-world edited image pairs.
- Baselines: EvoLMM uses raw images and a closed-loop Proposer–Solver process to generate questions, answers, and rewards without human-annotated QA pairs or metadata.Its training pool contains roughly 6k images sampled from visual reasoning, chart, and geometry datasets.
C.3 Evaluation Benchmarks
Evaluation spans visual mathematical, diagnostic, robustness, and logical reasoning benchmarks, with standardized Judge prompts and qualitative analyses of training behavior and failure cases.
- Evaluation Benchmarks: MathVision evaluates mathematical reasoning under visual context using the official 304-problem testmini split.
- Evaluation Benchmarks: MathVerse measures both final-answer accuracy and whether models genuinely use diagram information in visual math problems.It addresses the risk that textual cues duplicate visual content.
- Evaluation Benchmarks: WeMath diagnoses visual-math errors through concept-centered categories that distinguish insufficient knowledge from inadequate generalization.
- Evaluation Benchmarks: LogicVista evaluates multimodal logical reasoning with 448 multiple-choice questions.
- Evaluation Benchmarks: DynaMath tests robustness by generating variants from 501 seed questions across multiple topics.Each seed is instantiated into 10 variants.
- Evaluation and Analysis: The Judge scores answer correctness, reasoning quality, and visual grounding, while strict JSON and final-answer formatting constrain reward outputs.Invalid final-answer formatting causes all scores to be set to 0.0.
- Case Study: Majority voting compresses outputs toward a near-deterministic mapping, whereas the proposed method exhibits more stable learning behavior overall.The case study describes how incorrect frequency advantages can be repeatedly reinforced under majority voting.
- Case Study: An incorrect consensus can still mislead the method when self-consistency favors a wrong answer and the frozen Judge scores it highly.The resulting sharper distribution reduces exploration and can lower pass@10.
G Experiments and Analysis
The experiments analyze training stability, distributional modeling, and the interaction between self-consistency and Judge preferences. The method improves optimization stability and trajectory quality, while distributional contraction can reduce pass@10 in some cases.
- Training stability: The method improves training stability through group-relative reward modeling and Judge-based modulation, while reducing response-length collapse.The paper reports healthier entropy trajectories and more stable behavior across benchmarks.
- Limitations: Pass@10 can decrease when self-consistency and Judge scores favor the same incorrect answer, producing incorrect consensus and reduced sampling diversity.Because pass@10 depends on obtaining at least one correct solution among multiple samples, distributional contraction weakens its coverage benefit.
- Distributional modeling: Distributional modeling consistently improves performance across self-consistency, Judge-only, and final formulations on both benchmarks.It converts trajectory-level signals into relative group-wise advantages, smoothing training signals and stabilizing optimization.
- Signal relationship: Top-1 agreement between self-consistency and the Judge increases after training while remaining unsaturated, preserving room for exploration.The shift indicates greater alignment with reasoning quality without deterministic dependence on one signal.
- Signal relationship: Judge-selected trajectory accuracy improves even though the Judge is not trained, indicating that Judge guidance encourages higher-quality reasoning trajectories.Self-consistency supplies a stable distributional prior, while Judge modulation steers learning toward higher-quality candidates under ambiguity.