Source-linked AI summary
VisPlay: Self-Evolving Vision-Language Models from Images
Yicheng He, Chengsong Huang, Zongxia Li, Jiaxin Huang, Yonghui Yang
TL;DR
Existing VLM reinforcement learning methods depend on costly human annotations or task-specific reward heuristics, while self-evolution remains underexplored for visual models. VisPlay addresses this gap by jointly training a questioner and reasoner from unlabeled images with GRPO, yielding consistent improvements in reasoning, compositional generalization, and hallucination reduction across benchmarks.
Problem
VLM reinforcement learning often relies on costly, difficult-to-scale human annotation, while self-evolution remains underexplored for vision-language models.
Method
VisPlay jointly trains an Image-Conditioned Questioner and a Multimodal Reasoner from raw images using GRPO and rewards balancing question difficulty with answer quality.
Results
VisPlay consistently improves visual reasoning, compositional generalization, and hallucination reduction across multiple benchmarks and VLMs.
Takeaways & Limitations
Iterative co-improvement from self-generated questions and responses supports scalable self-evolving multimodal intelligence without human supervision.
Takeaways & Limitations
Experiments cover only the Qwen2.5-VL and MiMo-VL families, and the framework lacks definitive verification for self-generated data.
Abstract
from arXiv · showhide
Reinforcement learning (RL) provides a principled framework for improving Vision-Language Models (VLMs) on complex reasoning tasks. However, existing RL approaches often rely on human-annotated labels or task-specific heuristics to define verifiable rewards, both of which are costly and difficult to scale. We introduce VisPlay, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning abilities using large amounts of unlabeled image data. Starting from a single base VLM, VisPlay assigns the model into two interacting roles: an Image-Conditioned Questioner that formulates challenging yet answerable visual questions, and a Multimodal Reasoner that generates silver responses. These roles are jointly trained with Group Relative Policy Optimization (GRPO), which incorporates diversity and difficulty rewards to balance the complexity of generated questions with the quality of the silver answers. VisPlay scales efficiently across two model families. When trained on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements in visual reasoning, compositional generalization, and hallucination reduction across eight benchmarks, including MM-Vet and MMMU, demonstrating a scalable path toward self-evolving multimodal intelligence. The project page is available at https://bruno686.github.io/VisPlay/
1. Introduction
VisPlay addresses the costly, hard-to-scale dependence on human annotation by enabling VLMs to self-evolve from raw images. It uses interacting questioner and reasoner roles, achieving consistent gains across models and visual reasoning benchmarks.
- 1. Introduction: Average accuracy improves through successive evolutions from Evo 1 to Evo 5 on seven datasets compared with a Vision-47K GRPO baseline.Figure 1 presents this progression for Qwen2.5-VL-3B-Instruct.
- 1. Introduction: Self-evolution is established for LLM reasoning and coding but remains comparatively underexplored for VLMs because visual inputs create additional challenges.The paper positions abundant unlabeled visual data as a basis for continued VLM improvement.
- 1. Introduction: VisPlay enables VLMs to improve visual reasoning using raw, unannotated images through a self-evolving reinforcement-learning framework.The framework is designed to reduce reliance on human-annotated data.
- 1. Introduction: A single base VLM alternates between an Image-Conditioned Questioner generating challenging visual questions and a Multimodal Reasoner producing silver responses.Both roles are jointly optimized with GRPO using rewards that balance question difficulty and answer quality.
- 1. Introduction: VisPlay produces consistent performance gains across three VLMs and diverse visual reasoning benchmarks, including general understanding, visual mathematics, and hallucination detection.The evaluated models are Qwen2.5-VL-3B, Qwen2.5-VL-7B, and MiMo-VL-7B.
2. Method
VisPlay uses GRPO to co-evolve an image-conditioned Questioner and a Multimodal Reasoner from unlabeled images. The Questioner generates diverse, difficult questions, while the Reasoner creates pseudo-labels and learns from curated samples.
- GRPO trains the Questioner with relative rewards, avoiding a value function while constraining policy updates with clipped objectives and KL regularization.
- VisPlay alternates two agents initialized from one pretrained backbone: a Questioner generates visual queries, and a Reasoner answers them.
- Multimodal Reasoner Training: The Reasoner samples responses, derives pseudo-labels by majority voting, and assigns confidence from answer frequency without ground-truth labels.
- Image-Conditioned Questioner Training: The Questioner’s uncertainty reward targets questions whose Reasoner confidence is near 0.5, treating model uncertainty as a proxy for difficulty.
- Image-Conditioned Questioner Training: Diversity regularization clusters similar questions, while a hard <question> tag filter removes malformed outputs before combining rewards.
- Multimodal Reasoner Training: Training retains generated question–pseudo-label pairs with confidence between 0.25 and 0.75, excluding certain or highly unstable samples before Reasoner GRPO updates.
3. Experiments
VisPlay is evaluated on multiple visual reasoning domains and model backbones, with results showing consistent gains, improving co-evolution dynamics, and competitive performance against human-labeled training.
- Main Results: 47.27 average score after the third iteration, up from 30.61 at baseline for Qwen2.5-VL-3B, while Qwen2.5-VL-7B and MiMo-VL-7B also improve.
- Main Results: VisPlay improves performance across general visual understanding, mathematical reasoning, and hallucination detection benchmarks.For Qwen2.5-VL-3B, the Hallucination score rises from 32.81 to 94.95 by the second iteration.
- Human-Annotated Data Comparison: VisPlay achieves competitive average accuracy with standard GRPO trained on human-curated Vision-47K pairs while reducing dependence on manual supervision.The comparison covers Qwen2.5-VL-3B and 7B after one training epoch.
- Co-Evolution Dynamics: Across three VLMs, question difficulty and problem-solving accuracy both trend upward during training, with minor accuracy fluctuations.Figure 3 tracks difficulty on the left axis and accuracy on the right axis across the two interacting roles.
- Co-Evolution Dynamics: Reasoner accuracy increases from 44.0 to 49.0 on first-iteration questions, while estimated pseudo-label accuracy declines from 72 to 61 as questions become harder.The results indicate simultaneous improvement in reasoning ability and growth in data complexity across iterations.
- Question Difficulty Evolution: Generated questions progress from direct observation to relational reasoning and then multi-step inference across three training iterations.Examples include counting and identification, spatial or comparative judgments, and precise localization or causal relationships.
4. Related Work
Prior VLM post-training work has increasingly adopted reinforcement learning and self-play, while self-evolving learning remains more developed for LLMs than for VLMs.
- VLM post-training has shifted from supervised fine-tuning toward reinforcement-learning paradigms as pretrained VLM capabilities strengthen.
- Existing VLM self-play methods use simulated game data but often depend on external models or tools for training-data generation.
- LLM self-evolution has explored data-free or zero-human-supervision training, whereas extending these dynamics to VLMs is harder because of the visual modality.
5. Limitation
VisPlay’s evaluation is limited to two VLM families and lacks definitive verification for self-generated data. Larger-model scalability and faithfulness verification therefore remain open research questions.
- 5. Limitation: Experiments cover only the Qwen2.5-VL and MiMo-VL families, leaving effectiveness on significantly larger VLMs such as models with ≥10B parameters unresolved.The limitation is attributed to computational constraints.
- 5. Limitation: VisPlay lacks a definitive verification method for self-generated data, creating an open need for automated faithfulness checks that prevent error accumulation.GRPO indirectly optimizes quality, but the authors identify more robust verification as future work.
6. Conclusion
VisPlay enables VLMs to improve from unlabeled images by jointly training an Image-Conditioned Questioner and Multimodal Reasoner with GRPO. The framework yields consistent gains in reasoning, compositional generalization, and hallucination reduction across multiple benchmarks.
- 6. Conclusion: VisPlay improves VLM reasoning from unlabeled images by decomposing one model into an Image-Conditioned Questioner and a Multimodal Reasoner optimized with GRPO.The two roles balance question challenge with response accuracy without human supervision.
- 6. Conclusion: Experiments show consistent gains in visual reasoning, compositional generalization, and hallucination reduction across multiple benchmarks.The conclusion frames these gains as evidence for self-evolving multimodal intelligence.
- 6. Conclusion: Iteratively generating and learning from self-produced experiences supports a path toward multimodal systems that continually refine their capabilities without human-labeled data.The authors connect this process to richer multimodal interactions and cross-domain adaptation.
A.1 Detailed Training Dataset and Benchmarks
The experiments use unlabeled images from Vision-47K, three VLM backbones, and benchmarks spanning visual understanding, mathematical reasoning, and hallucination detection. These datasets cover diverse image types and reasoning demands.
- A.1 Detailed Training Dataset and Benchmarks: Vision-47K provides 47,000 web-sourced images across charts, medical images, educational materials, and driving simulation frames, used without associated questions or answers.Only the images are retained for training.
- A.1 Detailed Training Dataset and Benchmarks: VisPlay is trained on Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, and MiMo-VL-7B-SFT backbone models.The backbones span 3B and 7B parameter scales and use multimodal training configurations.
- A.1 Detailed Training Dataset and Benchmarks: General visual understanding is evaluated with MM-Vet, MMMU, RealWorldQA, and VisNumBench, covering recognition, OCR, cross-modal reasoning, contextual understanding, and visual number sense.The benchmarks include more than 5,000 MM-Vet test samples, 11.5K MMMU questions, and roughly 1.9K VisNumBench questions.
- A.1 Detailed Training Dataset and Benchmarks: MathVerse and MATH-Vision evaluate multimodal mathematical reasoning using diagram-centric and competition-level problems across geometry, algebra, and multiple subject areas.MathVerse contains 2.6K questions, while MATH-Vision has approximately 3K problems across 16 subjects and five difficulty tiers.
- A.1 Detailed Training Dataset and Benchmarks: HallusionBench measures visual hallucination errors and distinguishes language-only hallucinations from visual illusions through yes/no evaluation.The format is intended to measure hallucination rates and error types precisely.
A.2 Training Configuration
Training alternates between an image-conditioned Questioner and a multimodal Reasoner using controlled prompts, rollout settings, and answer evaluation. Both roles use matched training steps and conservative micro-batching for stable learning.
- A.2 Training Configuration: The Questioner generates eight candidate image-conditioned questions per input using an 8192-token context window and is evaluated on MMStar.It is trained on 47K multimodal samples, with training duration varying by model scale.
- A.2 Training Configuration: The Reasoner uses chain-of-thought reinforcement learning with outputs capped at 4096 tokens and a template requiring the final answer inside boxed notation.The prompt enforces a consistent reasoning and answer format.
- A.2 Training Configuration: Questioner prompts require exactly one image-related question in a selected multiple-choice, numerical, or regression format.The template constrains the generated question type and answerability requirements.
- A.2 Training Configuration: An LLM-as-Judge evaluates whether two answers are substantially equivalent while ignoring superficial differences in formatting, spacing, punctuation, and case.The judgment focuses on core content, logical meaning, and information expression.
- A.2 Training Configuration: Both Questioner and Solver updates use micro-batches of one, support up to 20K batched tokens per forward pass, and run for the same number of training steps.These settings are described as stabilizing learning under long sequences.
A.3 Prompt Templates
Three prompt templates define the core interaction structure for a self-evolving Vision-Language Model, coordinating specialized roles in an autonomous learning loop.
- A.3 Prompt Templates: The templates orchestrate specialized roles, including a question generator, multimodal reasoner, and evaluator, within an autonomous learning loop.The passage identifies these roles as components of the interaction structure.
- A.3 Prompt Templates: Each template imposes behavioral constraints for generating tasks, solving them with step-by-step reasoning, and assessing answer consistency.These constraints specify the intended behavior of the model components.
- A.3 Prompt Templates: Together, the templates establish a coordinated framework for self-evolving multimodal learning.