PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play

Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wan, Wentian Zhao

arXiv:2509.25541v2cs.CVcs.AI

TL;DR

Human-curated data and verification limit scalable VLM self-improvement. Vision-Zero addresses this with label-free strategic self-play from diverse images and Iterative-SPO, achieving broad gains over annotation-based methods across reasoning, chart/OCR, and vision-centric tasks.

  • Problem

    Current VLM training depends heavily on costly human-curated data, expert-designed question-answer pairs, and verification, limiting scalable multimodal self-improvement.

  • Method

    Vision-Zero uses label-free domain-agnostic “Who Is the Spy?” self-play games from visual inputs, with Iterative-SPO alternating self-play and RLVR.

  • Results

    Vision-Zero improves reasoning, chart/OCR, and vision-centric performance and surpasses strong baselines trained on costly human-annotated datasets.

  • Takeaways & Limitations

    Strategic visual self-play can provide an economical, flexible route to sustained VLM post-training without human annotations.

  • Takeaways & Limitations

    The current framework relies on edited image pairs and single-image observations, limiting specialized domains and richer video, multi-image, or 3D settings.

Abstract

from arXiv · show

Although reinforcement learning (RL) has emerged as a promising approach for improving vision-language models (VLMs) and multimodal large language models (MLLMs), current methods rely heavily on manually curated datasets and costly human verification, which limits scalable self-improvement in multimodal systems. To address this challenge, we propose Vision-Zero, a label-free, domain-agnostic multi-agent self-play framework for self-evolving VLMs through competitive visual games generated from arbitrary image inputs. Specifically, Vision-Zero encompasses three main attributes: (1) Strategic Self-Play Framework: Vision-Zero trains VLMs in "Who Is the Spy"-style games, where the models engage in strategic reasoning and actions across multiple roles. Through interactive gameplay, models autonomously generate their training data without human annotation. (2) Gameplay from Arbitrary Images: Unlike existing gamified frameworks, Vision-Zero can generate games from arbitrary images, thereby enhancing the model's reasoning ability across diverse domains and showing strong generalization to different tasks. We demonstrate this versatility using three distinct types of image datasets: CLEVR-based synthetic scenes, charts, and real-world images. (3) Sustainable Performance Gain: We introduce Iterative Self-Play Policy Optimization (Iterative-SPO), a novel training algorithm that alternates between Self-Play and reinforcement learning with verifiable rewards (RLVR), mitigating the performance plateau often seen in self-play-only training and achieving sustained long-term improvements. Despite using label-free data, Vision-Zero achieves state-of-the-art performance on reasoning, chart question answering, and vision-centric understanding tasks, surpassing other annotation-based methods. Models and code have been released at https://github.com/wangqinsi1/Vision-Zero.

1 INTRODUCTION

Vision-Zero addresses the scalability limits of human-dependent VLM training with label-free strategic self-play generated from diverse images. Its Iterative-SPO algorithm alternates self-play and RLVR, while experiments report broad gains over human-annotated baselines.

  • Motivation: The framework targets scalability bottlenecks caused by costly human-curated multimodal data and expert-designed question-answer pairs.These constraints limit the scale and diversity of training data.
  • Strategic self-play: Vision-Zero’s strategic “Who Is the Spy?” game trains visual understanding, inference, and communication through asymmetric multi-role interaction.Civilians observe the image, while the spy infers its content from civilian clues.
  • Vision-Zero: Vision-Zero is a label-free, domain-agnostic self-play framework that generates VLM training data from visual games rather than human annotations.It uses image pairs with visual differences and supports diverse inputs.
  • Optimization: Iterative-SPO alternates self-play with RLVR to stabilize training and prevent premature convergence to equilibrium states.The algorithm incorporates verifiable supervision into the self-play process.
  • Results: Vision-Zero improves performance across reasoning, chart/OCR, and vision-centric tasks while surpassing baselines trained on costly human-labeled datasets.The comparison is summarized in Figure 2 across multiple tasks.

2 VISION-ZERO: A GENERALIZABLE GAMIFICATION TRAINING FRAMEWORK

Vision-Zero is a generalizable, label-free VLM post-training framework built around strategic visual self-play with arbitrary images. Its Iterative-SPO algorithm alternates self-play and RLVR to support sustained improvement while combining visual reasoning, decision-making, and domain-agnostic data use.

  • Strategic Environment: Vision-Zero embeds VLMs in a Who Is the Spy game where civilians view an image, the spy sees a blank input, and both roles exchange clues and votes.The asymmetric setup requires the spy to infer hidden visual content while civilians balance informative clues against information leakage.
  • Label-Free and Domain-Agnostic Data Input: The framework accepts arbitrary label-free images, demonstrated with CLEVR scenes, charts, and real-world data to support diverse training inputs.The image is provided to civilians while the spy receives a blank visual input.
  • Iterative-SPO: Iterative-SPO alternates self-play in the clue stage with RLVR in the decision stage, using strategic rewards and vote-correctness rewards for the two modules.Its objective is to combine competitive clue generation with verifiable decision supervision.
  • Iterative-SPO: Clue-stage optimization uses zero-sum rewards and role-advantage estimation to address asymmetric information between the spy and civilians.The reward design penalizes suspicion while RAE adjusts advantages for the unequal role information.
  • Iterative-SPO: Alternating self-play with RLVR is designed to prevent strategic stagnation, knowledge plateaus, role collapse, and divergence during training.The training stage switches when performance signals indicate that one stage has saturated.
  • Advantages: Vision-Zero uses diverse image inputs to jointly strengthen reasoning, visual comprehension, spatial understanding, and OCR while reducing shortcut bias and negative capability transfer.The framework is presented as a cost-efficient way to curate training data directly from existing image datasets.

3 EXPERIMENTS

Vision-Zero is evaluated across diverse reasoning, chart, and vision-centric tasks, with experiments measuring generalization, efficiency, cost, and Iterative-SPO’s contribution.

  • Sustainable Performance Growth: Win rates increased during training, with Qwen2.5-VL-7B rising from 50% to 71%, while decision-stage token lengths grew from 250 to approximately 400 for InternVL models.These trends were reported as evidence of sustained performance growth and enhanced reasoning under Iterative-SPO.
  • Strong Task Generalization Capability: Vision-Zero models consistently outperformed state-of-the-art baselines across six reasoning and mathematics benchmarks, despite using no mathematics-specific task training.VisionZero-Qwen-7B variants achieved approximately 3% gains over the base model, while the strongest baseline improved by approximately 1.9%.
  • Cross-Capability Negative Transfer Mitigation: Vision-Zero mitigated cross-capability negative transfer, including improving MMVP accuracy from 76.8% to 79.5% and chart understanding by an average of 3.9%.The results span CLEVR, chart, and real-world training inputs across chart and vision-centric evaluations.
  • Low Costs: Vision-Zero required zero labeling cost and 127 A100-hours, reducing dataset construction and training costs relative to prior RLVR methods.The cost comparison covers dataset construction, training costs, and model performance across methods.
  • High Training Efficiency: 3.3× and 6.4× improvements in overall training efficiency were achieved over original GRPO for Qwen2.5-VL-7B and InternVL3-8B, respectively.The comparison used identical hardware and iteration settings; Vision-Zero also achieved higher final validation performance.
  • Model Generalizability: Vision-Zero improved InternVL3-8B and InternVL3-14B reasoning accuracy by 1.8% and 1.6%, respectively, demonstrating generalization across model families.Against MM-Eureka-trained GRPO baselines, reasoning capabilities improved by 1.3% and 2%, respectively.
  • Superiority of Iterative-SPO: Iterative-SPO outperformed pure self-play and pure RLVR, improving final LogicVista accuracy by 2% over pure self-play and 1% over pure RLVR.The method alternates self-play with RLVR to address the premature equilibrium observed in pure self-play.

4 CONCLUSION

Vision-Zero provides zero-human-in-the-loop VLM post-training through strategic, domain-agnostic self-play and Iterative-SPO. Experiments show improvements across reasoning, chart/OCR, and vision-centric tasks while reducing dataset construction costs.

  • 4 CONCLUSION: Vision-Zero combines domain-agnostic gamified self-play with Iterative-SPO to improve VLM capabilities without human-in-the-loop post-training.Iterative-SPO alternates self-play with RLVR to stabilize training and avoid suboptimal equilibria.

A APPENDIX

The appendix supplies additional material, design details, experimental settings, and supplementary results for Vision-Zero.

  • A APPENDIX: The appendix covers related work, Vision-Zero design details, experimental settings, and supplementary experimental results.Listed subsections include prompts, dataset preparation, Iterative-SPO, baselines, training costs, chart/OCR tasks, and vision-centric tasks.

A.1 RELATED WORK

Prior work studies self-play, undercover games, VLM game evaluation, visual discrimination, multi-agent communication, and contrastive RLVR. Vision-Zero builds on these directions while targeting self-improving VLM training through strategic visual gameplay.

  • Multi-Agent RL and Undercover Games: Prior VLM self-play work uses dialog games, foundation-model self-play, and undercover-style interactions to study iterative improvement, strategy, deception, and multi-round reasoning.Existing social-deduction studies primarily evaluate social capabilities or rely on prompt engineering for human-like behavior.
  • Game-Based Training and Evaluation: Game-based VLM benchmarks expose deficiencies in vision-based decision-making and evaluate perception, question answering, rule following, and end-to-end playing.These tasks align with different phases of undercover-style gameplay.
  • Visual Description and Communication: Visual description, odd-one-out reasoning, and multi-agent communication provide related paradigms for image discrimination, spatial reasoning, complementary information access, and emergent conventions.These paradigms connect descriptive communication with the information-sharing demands of visual games.
  • Contrastive RLVR for VLM Post-Training: Contrastive RLVR methods construct automatically verifiable visual tasks for multi-image comparison, hallucination localization, and geometry reasoning.Vision-Zero is positioned alongside these scalable post-training approaches while using strategic visual gameplay.

A.2 VISION-ZERO DESIGN DETAILS

This section presents the implementation details of Vision-Zero, covering prompt design, cross-domain training-data construction, and Iterative-SPO.

  • The implementation discussion covers prompt design, domain-specific training-data construction, and the formal Iterative-SPO algorithm.

A.2.1 PROMPT SETTING

Vision-Zero uses role-aware prompts that separate SPY and NOT SPY behavior, with minimal domain-specific changes for charts and real-world images. Dynamic templating enables batched multi-agent gameplay with limited manual intervention.

  • Prompt design enforces behavioral separation between SPY and NOT SPY players for role-consistent clue generation and inconsistency detection.
  • Chart prompts model SPY behavior as swapping two chart attributes, while real-world prompts describe differences in objects or scene parts.
  • Dynamic templates fill episode-specific tokens and role instructions, supporting batched multi-agent simulation across thousands of games with minimal manual intervention.

CLUE STAGE PROMPT

The clue-stage prompt assigns players distinct SPY or NOT SPY roles and instructs them to describe image features while reasoning about prior clues. SPY players infer the unseen chart or table from others’ descriptions, whereas NOT SPY players provide distinctive non-overlapping clues.

  • Players describe object features or relationships in one sentence while following role-specific instructions for SPY and NOT SPY behavior.
  • Only one player sees a blank image, while the remaining players share the same chart or table and try to avoid being identified as the odd player.
  • NOT SPY players provide clear, specific features that do not overlap previous clues, while SPY players infer the normal chart’s topic, attributes, and value relationships.
  • The prompt requires private step-by-step reasoning followed by a one-sentence picture description enclosed in \boxed{}.

DECISION STAGE PROMPT

The decision-stage prompt asks the model to identify the SPY from players’ clues by comparing them with the original image. It emphasizes inconsistency, vagueness, misleading behavior, and uncertainty, and requires a boxed player number or N/A.

  • The model identifies which player is most likely the SPY based on clues about their pictures.
  • The decision setup contrasts one SPY with non-SPY players who share the original picture and provide clues while attempting to blend in or expose deception.
  • The model compares clues against the original image and checks inconsistency, generic wording, misleading intent, deception, and uncertainty.
  • The required output is private step-by-step reasoning followed by a boxed PLAYER_NUMBER or N/A when uncertain.

A.2.2 DATASET PREPARATION

The dataset pipeline creates visual game inputs from synthetic scenes and charts using structured representations, controlled edits, and automated rendering. This avoids unreliable direct image editing while preserving diverse visual attributes.

  • Dataset preparation: Vision-Zero prepares CLEVR and chart data by modifying structured scene or chart representations before rerendering images.CLEVR uses scene JSON and change logs, while charts encode numerical values and metadata in JSON before Python-based rendering.
  • CLEVR-based data: 2,000 CLEVR image pairs are automatically rendered from scenes containing 4–6 objects with controlled attribute changes.The pipeline edits object shape and color in JSON, then rerenders the modified scene with Blender.
  • Chart-based data: Direct chart-image editing was unreliable for exact graphical attributes, so the authors adopted structured JSON editing followed by deterministic plotting.The reported failures included axis ticks, bar widths, label alignment, color scales, and visual artifacts.
  • Chart-based data: The structured chart pipeline is robust to typical AI-editor failures and leverages multimodal models’ captioning and scene-parsing abilities.
  • Chart-based data: 1,000 ChartQA training charts span line, bar, and pie formats, providing diverse chart styles and complexity for derived pairs.The sampling process costs only tens of US dollars through a fully automated pipeline.

A.2.3 ITERATIVE-SPO ALGORITHM

Iterative-SPO alternates clue and decision stages in the Who Is the Spy game, using role-aware rewards and KL-regularized policy updates. Stage switching uses prediction accuracy and “n/a” rates to determine when training should change focus.

  • Iterative-SPO algorithm: Iterative-SPO alternates between clue and decision stages to sustain self-play training rather than following a single fixed interaction phase.The algorithm switches stages based on average decision accuracy and “n/a” rates tracked with exponential moving averages.
  • Clue stage: During the clue stage, players generate clues from dialogue history and input images, then receive zero-sum rewards based on voting outcomes.The clue reward compares the spy’s vote with the civilians’ average vote and adjusts civilian rewards relative to that average.
  • Policy optimization: Role-advantage estimation supplies stage-specific advantages before policy optimization, addressing the asymmetric roles in the game.The algorithm applies RAE-based advantages and group-normalized advantages before updating policies.
  • Decision stage: During the decision stage, each citizen casts a vote from the clue history and image information to identify the spy.
  • Policy optimization: Policy updates use KL-regularized policy gradients, while stage switching monitors accuracy and “n/a” rates over training rounds.

A.3 EXPERIMENTS SETTING

The experiments evaluate Vision-Zero across multiple VLMs, benchmarks, and annotation-based baselines. Results report gains in reasoning, chart understanding, and vision-centric tasks, alongside robustness analyses and explicit scope limitations.

  • Results: Vision-Zero-Qwen-7B (CLEVR) surpasses state-of-the-art baselines by 1.1% on average across six vision-centric task categories.The CLEVR-trained model performs better than the chart-trained variant on these vision-centric tasks.
  • Results: Vision-Zero consistently outperforms MiCo across all six general reasoning benchmarks under the reported evaluation.The paper attributes this comparison to the strategically constructed self-play environment and strategic interaction mechanism.
  • Results: VisionZero-Qwen-7B (Chart) improves ChartXIV_QA by 4.1% while matching or exceeding models trained on large-scale chart-specific data without QA supervision.
  • Stability analysis: With failed image edits, Vision-Zero still achieves an average accuracy increase of 2.8% over baseline, although the gain is slightly lower than without noise.The authors report that failed-edit samples remain useful because the model can reason from alternative available information.
  • Ablation studies: Two or three clue rounds yield average gains of 3% and 4.1% across six benchmarks, whereas one round provides only a small improvement.The paper links the larger gains to more information and multi-step integration during training.
  • Limitations: Vision-Zero is limited by reliance on image editors and currently targets single-image, pairwise-edit settings rather than richer modalities.The authors identify medical imaging, scientific charts, remote sensing, extended videos, multi-image contexts, and interactive 3D environments as boundary cases requiring further work.
Loading 2509.25541v2…