Source-linked AI summary
First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training
Lai Wei, Yuting Li, Chen Wang, Yue Wang, Linghe Kong, Weiran Huang, Lichao Sun
TL;DR
MLLM post-training commonly depends on costly annotated data or external rewards, limiting scalable continual improvement. MM-UPT addresses this gap with majority-voted self-rewards inside GRPO and extends the approach to model-generated data. The method improves multimodal reasoning without labeled data while exposing competence, diversity, and deployment-cost trade-offs.
Problem
SFT and RL for MLLMs rely on expensive annotated multimodal data or external supervision, motivating an unsupervised third post-training stage.
Method
MM-UPT adapts GRPO by using majority voting over sampled responses as pseudo-rewards and trains on either existing unlabeled or self-generated multimodal data.
Results
MM-UPT improves reasoning across benchmarks without labels, remains effective for models already trained with supervised GRPO, and synthetic-data training is competitive with original unlabeled-data training.
Takeaways & Limitations
Unsupervised post-training offers a simple path toward continual and scalable MLLM self-improvement without external supervision.
Takeaways & Limitations
MM-UPT can amplify errors when the model lacks sufficient prior knowledge, reduce pass@n diversity, and shift computation from inference to one-time training.
Abstract
from arXiv · showhide
Improving Multi-modal Large Language Models (MLLMs) in the post-training stage typically relies on supervised fine-tuning (SFT) or reinforcement learning (RL), which require expensive and manually annotated multi-modal data--an ultimately unsustainable resource. This limitation has motivated a growing interest in unsupervised paradigms as a third stage of post-training after SFT and RL. While recent efforts have explored this direction, their methods are complex and difficult to iterate. To address this, we propose MM-UPT, a simple yet effective framework for unsupervised post-training of MLLMs, enabling continual self-improvement without any external supervision. The training method of MM-UPT builds upon GRPO, replacing traditional reward signals with a self-rewarding mechanism based on majority voting over multiple sampled responses. Our experiments demonstrate that such training method effectively improves the reasoning ability of Qwen2.5-VL-7B (e.g., 66.3\%$\rightarrow$72.9\% on MathVista, 62.9\%$\rightarrow$68.7\% on We-Math), using standard dataset without ground truth labels. To further explore scalability, we extend our framework to a data self-generation setting, designing two strategies that prompt the MLLM to synthesize new training samples on its own. Additional experiments show that combining these synthetic data with the unsupervised training method can also boost performance, highlighting a promising approach for scalable self-improvement. Overall, MM-UPT offers a new paradigm for autonomous enhancement of MLLMs, serving as a critical third step after initial SFT and RL in the absence of external supervision. Our code is available at https://github.com/waltonfuture/MM-UPT.
2 Zhongguancun Academy
This passage lists affiliations: Shanghai Innovation Institute and Lehigh University.
- The listed affiliations are Shanghai Innovation Institute and Lehigh University.
1 Introduction
The paper proposes unsupervised post-training as a third stage for MLLMs, addressing the dependence of SFT and RL on costly annotated data. MM-UPT uses majority-voted pseudo-rewards within GRPO and extends to self-generated data for scalable self-improvement.
- MM-UPT: MM-UPT adapts GRPO by using majority voting over sampled responses as a dynamic pseudo-label and self-reward signal.Responses agreeing with the consensus receive positive reward, while divergent responses are penalized.
- Self-Generated Data: The framework also supports self-generation through in-context and direct synthesis of new multimodal questions.These synthetic samples are trained with the same unsupervised reinforcement-learning method.
- Findings: MM-UPT improves reasoning on existing unlabeled data and achieves competitive performance with supervised GRPO, while synthetic-data training offers a scalable self-improvement path.The paper also reports a trade-off: accuracy gains accompany reduced response diversity and require sufficient initial competence to avoid error amplification.
- Motivation and Contributions: MM-UPT formalizes unsupervised post-training as a third stage after SFT and RL, enabling continual improvement without external supervision.The motivation is the growing impracticality of curating and annotating high-quality multimodal data at scale.
2 Related Works
The related work situates MM-UPT within self-improvement for language and multimodal models, emphasizing the limits of externally annotated data and the complexity of existing synthetic-data pipelines.
- Self Improvement: Self-improvement research uses model-generated data without external supervision, including emerging applications in multimodal settings.
- Multi-modal Reasoning: Multimodal reasoning is more complex than text-only reasoning because models must process and interpret visual inputs for tasks such as geometry and chart understanding.
3 The Framework of Multi-Modal Unsupervised Post-Training
MM-UPT formulates post-training on unlabeled multimodal data and replaces supervised rewards with majority-vote pseudo-rewards in GRPO. It also trains on model-generated questions to create an autonomous self-improvement loop.
- 3.1 Problem Formulation: MM-UPT targets improvement of a trained MLLM using unlabeled image-question pairs without ground-truth answers or external supervision.Unlike SFT, RLVR, and RLHF, it relies only on the model’s own responses to generate training signals.
- 3.2 Training Method: Majority voting converts sampled responses into pseudo-labels, rewarding responses that match the most frequent extracted answer.The method applies standard GRPO to these majority-based rewards and updates the policy while retaining its reference-model constraint.
- 3.2 Training Method: The GRPO procedure samples response groups, extracts answers, computes majority-agreement rewards and normalized advantages, then updates the policy parameters.
- 3.3 Synthetic Data: MM-UPT generates additional training data through in-context synthesis from image-question-answer examples or direct synthesis from images alone.Both synthetic-data strategies reuse majority voting to define pseudo-rewards during unsupervised post-training.
- 3.3 Synthetic Data: Using synthetic samples expands the training corpus without human annotations, forming a fully autonomous self-improvement loop.
4 Experiments
MM-UPT is evaluated on unlabeled standard and synthetic multi-modal reasoning data across four benchmarks, multiple generation strategies, and several backbone models. It improves over the base model and unsupervised baselines, remains competitive with supervised methods, and supports scalable self-improvement.
- 4 Experiments: The experiments evaluate unlabeled human-created questions and model-generated synthetic questions using multimodal mathematical reasoning benchmarks and datasets.The setup includes four benchmarks and standard datasets covering geometric diagrams, charts, tables, multiple-choice questions, and fill-in-the-blank formats.
- 4.3 Scenario 2: Unsupervised Training on Synthetic Datasets: Both in-context and direct synthetic-question strategies significantly improve the base model and perform competitively with training on original questions on average.Table 2 compares the two synthetic-data strategies with original questions in Scenario 2.
- 4.2 Scenario 1: Unsupervised Training on Standard Datasets: MM-UPT improves the Qwen2.5-VL-7B average from 49.47 to 53.17 across four benchmarks, outperforming unsupervised baselines and remaining competitive with supervised post-training.Table 1 reports accuracy on MathVision, MathVerse, MathVista, and We-Math; prior baselines provide marginal gains or degrade performance on some benchmarks.
- 4.3 Scenario 2: Unsupervised Training on Synthetic Datasets: Direct synthesizing produces more diverse and novel questions than in-context synthesizing, although some directly generated questions still contain hallucinations.Manual inspection finds that in-context synthesis often rephrases original questions, whereas many direct-synthesis examples are high quality and useful for unsupervised post-training.
- 4.4 Ablation Study: MM-UPT consistently improves models of different sizes and architectures, including a 7.4% average gain for Qwen2.5-VL-3B and a 54.07 average for ThinkLite-VL-7B.The backbone ablation applies MM-UPT on Geometry3K without labels and reports 74.70 on MathVista for ThinkLite-VL-7B.
- 4.4 Ablation Study: Applying MM-UPT after supervised GRPO further improves models, raising MM-Eureka-7B from 53.10 to 53.78 and producing a 54.07 average for ThinkLite-VL-7B.These results support using MM-UPT as a lightweight refinement step with new unlabeled data.
5 Deeper Analysis
MM-UPT improves multiple backbones without labels, but its majority-voting reward depends on prior knowledge and introduces accuracy–diversity and training–inference cost trade-offs.
- Why majority voting works: Majority voting can outperform individual responses when per-response correctness exceeds 0.5, supporting its use as an unsupervised pseudo-reward.For p = 0.7 and n = 10, the cited analysis gives P(E) ≈ 0.849.
- When MM-UPT fails: MM-UPT degrades performance on ThinkLite-11K because difficulty-aware samples leave the model more likely wrong than right, causing majority voting to amplify errors.Table 4 reports decreased performance across all benchmarks in this limited-prior-knowledge setting.
- Accuracy–diversity trade-off: MM-UPT improves pass@1 but consistently reduces pass@n for large n, reflecting lower response diversity as training favors high-consensus answers.The paper identifies reduced exploration and suppression of occasionally correct minority answers as associated effects.
- Training–inference trade-off: MM-UPT shifts repeated-sampling costs from inference to one-time training, favoring deployment scenarios where inference efficiency and scalability matter.Inference-time ensembling remains preferable when training resources are limited.
- Implications: The paper identifies balancing accuracy with diversity and choosing between training-time and inference-time costs as open directions for MM-UPT.These trade-offs define important considerations for future exploration.
6 Conclusion
The paper formalizes Unsupervised Post-Training as a third post-training stage and instantiates it with MM-UPT for self-improving MLLMs without external supervision.
- MM-UPT uses majority voting as a self-rewarding mechanism within GRPO to guide multi-modal reasoning models toward consistent, high-confidence responses.
- The framework improves reasoning across multiple benchmarks without labeled data or external reward models.
- Synthetic questions generated by the model can be combined with MM-UPT to further boost performance, indicating a scalable path toward autonomous self-improvement.
A Implementation Details
The implementation compares several self-training baselines, evaluates standard visual-reasoning benchmarks, and uses answer extraction with an external evaluator to compute benchmark accuracy.
- Baseline methods: LMSI trains on majority-voted responses, while SRLM selects positive and negative examples using self-judged scores.
- Baseline methods: Genixer is adapted by removing instruction collection and template design, then generating 16 responses per question with Qwen2.5-VL for filtering.
- Baseline methods: STIC uses a two-stage self-training process involving generated image descriptions, distorted images, and recycled SFT data.
- Benchmarks: The evaluation covers MathVision, MathVista, and MathVerse, which assess visual mathematical reasoning across diverse problem types and difficulty levels.
- Evaluation: Models place final answers in a designated box, and Qwen2.5-32B-Instruct evaluates correctness against ground-truth answers; scores may differ from original reports because protocols vary.
- Training data: The training datasets are Geometry3K, GeoQA, and MMR1, containing geometric, fill-in-the-blank, multiple-choice, and other reasoning samples.
B.1 Training Dynamics
Training diagnostics show that MM-UPT increases majority-voting reward and benchmark accuracy while reducing semantic entropy, with improvements also transferring beyond mathematical visual reasoning.
- B.1 Training Dynamics: The majority-voting reward consistently increases while semantic entropy steadily decreases during MM-UPT training on Qwen2.5-VL-7B with MMR1.These trends indicate increasingly consistent, confident, and stable predictions.
- B.1 Training Dynamics: Average benchmark accuracy rises during training, while effective rank also increases as the model’s internal knowledge is exploited.
- B.2 Generalization Beyond Multimodal Mathematical Reasoning: MM-UPT improves both ChartQA and IconQA accuracy after training on MMR1, avoiding degradation on non-mathematical visual question answering tasks.The authors attribute this transfer to reinforcing reliable and consistent answers.
- B.3 Adaptability to Language Tasks: Applying the same self-rewarding mechanism to Qwen2.5-MATH-7B produces substantial improvements on MATH and Omni-MATH.The results support extending MM-UPT beyond multi-modal settings when the base model has sufficient initial competency.
C Compute Resources
The experiments use NVIDIA H100-80G and A800-40G GPUs, with one Geometry3K GRPO training run taking about 10 hours on eight A800 GPUs.
- Training Qwen2.5-VL-7B on Geometry3K with GRPO takes around 10 hours using eight A800 GPUs.The experiments use NVIDIA H100-80G and A800-40G GPUs.