PaperlayerView exact revision on arXiv

Source-linked AI summary

PyVision-RL: Forging Open Agentic Vision Models via RL

Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, Chen Wei

arXiv:2602.20739v1cs.AIcs.CV

TL;DR

Agentic multimodal RL can collapse into short, low-tool-use behavior, limiting sustained interaction. PyVision-RL stabilizes rollout selection and rewards continued tool use, producing image and video models with strong benchmark results and more efficient video reasoning. Its video model selectively constructs visual context from task-relevant frames, while deployment requires care because Python may access the host file system.

  • Problem

    RL fine-tuning can cause agentic multimodal models to reduce tool usage and converge to short, low-interaction behaviors.

  • Method

    PyVision-RL combines Python-based dynamic tooling with oversampling–filtering–ranking rollouts, accumulative tool rewards, and on-demand video context construction.

  • Results

    PyVision-Image achieves state-of-the-art results, while PyVision-Video reaches 44.0% accuracy using 5K average visual tokens per sample versus Qwen2.5-VL-7B’s 38.0% accuracy and 45K tokens.

  • Takeaways & Limitations

    Sustained interaction and dynamic, on-demand visual processing support effective and efficient multimodal agentic reasoning.

  • Takeaways & Limitations

    Because Python can access the host file system, deploying PyVision-Image and PyVision-Video requires careful consideration of potential impacts.

Abstract

from arXiv · show

Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.

1. Introduction

PyVision-RL addresses interaction collapse in agentic multimodal reinforcement learning with stabilized rollout selection, sustained tool-use incentives, and unified image/video scaffolds. Its models achieve strong benchmark performance, while PyVision-Video improves visual-token efficiency through on-demand frame processing.

  • Agentic multimodal models may reduce tool usage after RL fine-tuning, converging to short, low-interaction behaviors.
  • PyVision-RL uses Python as a dynamic primitive tool for image and video understanding within a unified agentic training framework.
  • Its oversampling–filtering–ranking rollout strategy stabilizes agent–environment interaction, while accumulative tool rewards incentivize sustained multi-turn behavior.
  • PyVision-Video loads video into the runtime and selectively samples task-relevant frames during reasoning, reducing visual-token consumption compared with uniform sampling.
  • +2.2% on VSI-Bench and 5K versus 45K average visual tokens per sample accompany PyVision-Video’s 44.0% versus 38.0% accuracy against Qwen2.5-VL-7B.
  • PyVision-Image and PyVision-Video achieve state-of-the-art performance while substantially improving token efficiency, particularly for video understanding.

2. Related Work

Prior multimodal tool-use research includes fixed task-specific tools and reinforcement-learning methods targeting training and optimization challenges. PyVision-RL instead uses Python as a primitive tool, supporting flexible multimodal interaction.

  • Static multimodal toolsets predefine task-specific operations such as cropping, zooming, and video clipping.
  • Dynamic tooling treats Python as a primitive tool rather than exposing a fixed collection of manually engineered operations.
  • Conventional video MLLMs uniformly sample frames and inject them into context, whereas on-demand construction selectively samples and plots relevant frames through Python code.
  • Reinforcement-learning approaches for multimodal models include critic-free algorithms, improved advantage estimation, modified PPO clipping, and methods addressing training–inference mismatch.

3. Method: PyVision-RL

PyVision-RL trains open-weight multimodal agents with Python-based dynamic tooling, combining sustained tool-use rewards with rollout selection to stabilize reinforcement learning. The unified framework supports image and video understanding, including on-demand video frame selection, and reports broad benchmark gains with reduced visual-token use.

  • Agentic Scaffold: PyVision-RL uses Python as a primitive tool within a unified scaffold for image and video understanding.
  • Agentic Scaffold: PyVision-Video loads the full video into Python and selectively samples task-relevant frames during reasoning instead of using uniform sampling.This design improves performance while reducing visual token usage.
  • Results: PyVision-Image achieves state-of-the-art results across visual search, multimodal reasoning, and agentic reasoning benchmarks.Reported gains include +10.2% on V*, +9.6% on WeMath, and +7.3% on TIR-Bench over the cited baselines.
  • Results: PyVision-Video achieves a 7.3% absolute improvement over the Qwen2.5-VL-7B baseline on VSI-Bench.The comparison includes Video-R1 and VITAL, with all methods based on Qwen2.5-VL-7B and trained with RL.
  • Accumulative Tool Reward: The accumulative tool reward adds 0.1·ntc to the reward only when the final answer is correct.Here, ntc is the total number of tool calls in the rollout, so incorrect or unproductive calls receive no such bonus.
  • Oversampling–Filtering–Ranking Rollouts: Rollout generation oversamples candidates, filters zero-variance groups and broken interactions, then ranks groups by reward standard deviation.The method retains informative, moderately difficult groups to improve training stability and efficiency.

4. Experiments

Experiments show strong image and video performance, with PyVision-Video reducing visual-token use through on-demand frame selection. Ablations indicate that sustained tool use and training components supporting stable interaction improve later-stage performance.

  • Ablation Study: At step 600, a maximum turn budget of 4 outperforms a budget of 2 by 1.93%.The advantage of a larger turn budget emerges later in RL training, suggesting a higher performance upper bound.
  • Main Results: PyVision-Image outperforms competing methods across visual search, multimodal reasoning, and agentic reasoning benchmarks.It improves over Qwen2.5-VL-7B by +10.2% on V*, +6.5% on HRBench-4K, and +6.4% on HRBench-8K, and over the base model by +3.8% on agentic reasoning.
  • Main Results: On-demand context construction selectively retrieves task-relevant video frames through Python instead of uniformly sampling the input video.The method dynamically queries the full video and includes informative frames in the multimodal context, reducing redundant visual tokens.
  • Main Results: PyVision-Video achieves 44.0% performance on VSI-Bench using approximately 5K visual tokens per sample.Qwen2.5-VL-7B reaches 38.0% with approximately 45K visual tokens, while SpaceR reaches 45.6% with around 25K.
  • Ablation Study: Removing the accumulative tool reward reduces tool usage, while retaining standard deviation normalization causes persistent performance fluctuations.The accumulative reward and larger turn budgets produce larger gains in later training stages; standard deviation sorting reduces negative advantages for correct trajectories.
  • Ablation Study: RL training remains stable while entropy loss and gradient norm decrease and tool calls, accuracy reward, and response length increase.These dynamics indicate sustained multi-turn interaction within episodes under the proposed training algorithm.

5. Conclusion

PyVision-RL is a unified framework for image and video understanding that uses Python for dynamic tooling and training strategies designed to sustain interaction. Its models show strong benchmark performance, while PyVision-Video substantially reduces visual-token usage and achieves a favorable accuracy–efficiency trade-off.

  • PyVision-RL uses Python for dynamic tooling across image and video understanding and introduces rollout strategies for stable tool-use reinforcement learning.The framework combines oversampling–filtering–ranking rollout generation with sustained interaction objectives.
  • PyVision-Video reduces visual token usage while maintaining effective spatial reasoning on VSI-Bench.The conclusion characterizes its accuracy–efficiency trade-off as favorable and connects the result to dynamic tooling and sustained interaction.

Impact Statement

The paper identifies a deployment risk for PyVision-Image and PyVision-Video: Python-based tooling may access the host file system and cause damage. Deployment therefore requires careful consideration of these impacts.

  • Python as the primitive tool may access the host file system and cause damage during deployment.The stated boundary concerns the operational safety of deploying both agentic vision models.

B. More Details of Training Pipeline and Training Data

The training pipeline generates multiple rollouts per prompt, removes broken or unusable groups, and ranks the remaining samples by rollout difficulty. It then selects moderately difficult rollouts for policy updates.

  • Oversampling: The framework oversamples αB prompts from the prompt pool and generates G rollouts for each prompt using the current policy.Rollouts execute code blocks in an environment and receive observations when available.
  • Rollout evaluation: Each rollout receives a reward from the reward model, while trajectories encountering timeouts, runtime death, or execution errors are marked broken.Rewards are computed as rj,i = R(xj, oj,i).
  • Filtering: Groups in which all rollouts are broken are filtered out before ranking the remaining rollout groups.The filtered set is initialized as F = ∅ before processing the oversampled prompts.
  • Ranking: The remaining rollouts are sorted by group-level standard deviation σj,i in descending order, using variation as a difficulty-ranking signal.The framework labels this stage ranking via difficulty.
  • Selection: The top B ∗G samples from the sorted set are selected as Dtrain for the policy update, targeting moderately difficult samples.The complete oversampling-filtering-ranking pipeline is described in Algorithm 1 and illustrated in Fig. 10.

B.2. Training Data Distribution

The paper presents the supervised fine-tuning and reinforcement-learning data distributions for PyVision-Image and PyVision-Video in separate figures.

  • Training Data Distribution: Figure 11 and Figure 12 illustrate the SFT and RL data distributions for PyVision-Image and PyVision-Video.

C.1. Ablation Results Plot on Different Benchmarks

The paper evaluates training-component ablations across multiple benchmarks and settings, with detailed numerical results provided in Table 3. One reported effect concerns the maximum turn budget during later training.

  • C.1. Ablation Results Plot on Different Benchmarks: Results are plotted across different benchmarks under different training settings.The paper refers to these comparisons as ablation results across training configurations.
  • C.1. Ablation Results Plot on Different Benchmarks: A maximum turn budget of 4 outperforms a budget of 2 at training step 600 by +1.77% on V* and +4.65% on MathVerse.The comparison is reported for the later training stage.

C.2. Ablation Results Detail

The paper provides exact numerical results for the ablation of training components in Table 3.

  • C.2. Ablation Results Detail: Table 3 lists the exact ablation result numbers for the training components.The ablated components are accumulative tool reward, standard deviation ranking, removing standard deviation normalization in advantage estimation, and maximum turn budget.

D.1. Training Dynamics of PyVision-Video

PyVision-Video’s RL training remains stable while validation performance, accuracy reward, response length, and tool calls increase steadily. The observed tool-use dynamics and advantage distribution support sustained, long-horizon interaction during training.

  • Tool-use dynamics: The average number of tool calls negatively correlates with the ratio of correct samples receiving negative advantage.The authors hypothesize that negative signals on correct samples using relatively fewer tools drive increasing tool-call means.
  • Training dynamics: Validation score, accuracy reward, response length, and average tool-call numbers increase steadily during PyVision-Video’s RL training.Entropy loss remains moderate and gradient norm decreases steadily, indicating stable RL optimization.
  • Advantage estimation: Removing the standard-deviation normalization term produces lower-variance advantages and more stable performance improvement.The comparison directly evaluates advantage distributions with and without this normalization term.
  • Interaction length: PyVision-Image trained with a maximum turn budget of 4 uses more than 3 turns on most samples after 700 training steps.Models with larger maximum turn budgets exhibit more tool calls across all reported benchmarks.
  • Tooling behavior: On VSI-Bench, 87.4% of PyVision-Video tool calls are fetch_frames_and_plot, while the model also uses diverse tool categories.The concentration reflects on-demand context construction, alongside flexibility in dynamic tooling.
Loading 2602.20739v1…