PaperlayerView exact revision on arXiv

Source-linked AI summary

PyVision: Agentic Vision with Dynamic Tooling

Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, Chen Wei

arXiv:2507.07998v3cs.CLcs.AIcs.CV

TL;DR

Visual reasoning systems often depend on predefined workflows and static toolsets, limiting dynamic adaptation. PyVision enables MLLMs to generate, execute, and refine Python-based tools interactively, and it reports consistent benchmark improvements, including +7.8% on V* and +31.1% on VLMsAreBlind-mini. The results support dynamic tooling as a path toward more agentic visual reasoning.

  • Problem

    Prior visual-reasoning approaches largely rely on predefined workflows and static toolsets, while the process of dynamically creating tools has limited visibility.

  • Method

    PyVision is an interactive multi-turn framework that lets MLLMs generate, execute, and iteratively refine Python-based tools for multimodal queries.

  • Results

    PyVision consistently improves strong backend models across benchmarks, including +7.8% on V* with GPT-4.1 and +31.1% on VLMsAreBlind-mini with Claude-4.0-Sonnet.

  • Takeaways & Limitations

    Dynamic tool generation adapts strategies to task and domain demands and moves multimodal reasoning toward more versatile, autonomous systems.

  • Takeaways & Limitations

    A visual-difference example shows that PyVision’s reasoning pipeline can still produce incorrect final answers and hallucinations.

Abstract

from arXiv · show

LLMs are increasingly deployed as agents, systems capable of planning, reasoning, and dynamically calling external tools. However, in visual reasoning, prior approaches largely remain limited by predefined workflows and static toolsets. In this report, we present PyVision, an interactive, multi-turn framework that enables MLLMs to autonomously generate, execute, and refine Python-based tools tailored to the task at hand, unlocking flexible and interpretable problem-solving. We develop a taxonomy of the tools created by PyVision and analyze their usage across a diverse set of benchmarks. Quantitatively, PyVision achieves consistent performance gains, boosting GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini. These results point to a broader shift: dynamic tooling allows models not just to use tools, but to invent them, advancing toward more agentic visual reasoning.

1. Introduction

PyVision addresses the limits of predefined workflows and static toolsets by enabling MLLMs to generate and refine task-specific Python tools interactively. Across benchmarks, this dynamic tooling improves performance and supports adaptive visual reasoning.

  • Prior visual-reasoning systems commonly use predefined workflows and static toolsets within single-turn frameworks.
  • PyVision lets MLLMs autonomously generate, execute, and iteratively refine Python code in response to multimodal queries.
  • Its tool taxonomy covers basic and advanced image processing, visual prompting and sketching, numerical and statistical analysis, and creative task-specific operations.
  • Dynamic tool generation adapts the model’s strategy to task and domain demands, while the authors frame it as progress toward versatile autonomous visual reasoning.
  • +7.8% on V* and +31.1% on VLMsAreBlind-mini are reported gains from PyVision with GPT-4.1 and Claude-4.0-Sonnet, respectively.The VLMsAreBlind-mini result improves from 48.1% to 79.2%.

2. PyVision

PyVision is an interactive framework in which MLLMs generate and execute Python code, receive multimodal outputs, and refine reasoning over multiple turns. Its runtime and prompt design support flexible, persistent, and isolated tool-generation loops.

  • Each inference turn generates Python code, executes it in an isolated runtime, and feeds textual or visual outputs back into the MLLM.
  • PyVision provides Python as the building block for dynamically generated tools rather than a fixed set of visual modules.It leverages libraries including OpenCV, Pillow, NumPy, Pandas, Scikit-learn, and Scikit-image.
  • System prompts standardize image access, code formatting, output functions, parsing tags, and final-answer tags.
  • The multi-turn loop continues until the MLLM produces a final answer, allowing iterative refinement of its reasoning.
  • Process isolation limits the impact of crashes or side effects, while cross-turn persistence preserves variables and state for later turns.

3. Dynamically Generated Tools

PyVision dynamically generates diverse Python-based tools tailored to visual tasks and domains, ranging from image processing and sketching to numerical analysis. Its tooling patterns vary by benchmark, while case studies illustrate adaptive workflows for search, medical imaging, symbolic puzzles, video, and visual comparison.

  • Tooling Taxonomy: PyVision’s taxonomy includes basic image processing, advanced image processing, visual prompting and sketching, and numerical and statistical analysis.The taxonomy is constructed from generated code across tasks and domains.
  • Basic Image Processing: Basic image-processing tools crop, rotate, and enhance inputs to focus attention, normalize orientation, or improve subtle visual content.Cropping can function as soft object detection by selecting regions of interest through reasoning.
  • Advanced Image Processing: Advanced tools perform segmentation, detection, and OCR to isolate regions, localize objects, and extract text without external APIs.These operations support follow-up tasks such as counting, measuring, and hybrid visual-linguistic reasoning.
  • Visual Prompting and Sketching: Visual prompting and sketching add marks or lines that act as visual notes for counting, enumeration, geometric reasoning, and spatial tasks.Numerical and statistical tools analyze histograms or compute visual quantities such as areas and lengths.
  • Case Studies: Case studies show PyVision adapting tools to task structure, including iterative cropping and OCR, medical contrast and histogram analysis, pixel differencing, and selective video-frame processing.The spot-the-difference example demonstrates creative tool synthesis, although its final answers are not completely correct.
  • Tooling Patterns Across Tasks and Domains: Over 83% of tools in V* visual search are cropping operations, while mathematical tasks favor numerical tools, symbolic visual tasks favor advanced processing, and medical images favor contrast enhancement.Remote-sensing tasks more commonly use segmentation tools.
  • Conclusion: These task- and domain-specific patterns underscore dynamic tool generation as a way to tailor the model’s strategy to the problem at hand.The reported conclusion links flexible tool generation with adaptive visual reasoning.

4. Results on Versatile Benchmarks

Across diverse benchmarks, PyVision consistently improves strong backend models, with gains varying by task and backend capability. The results also show that dynamic tooling amplifies models’ existing reasoning or perception strengths and that Claude generates code more frequently than GPT-4.1.

  • Benchmark performance: PyVision yields uniform GPT-4.1 gains across datasets, including +1.8% on MathVista, +2.4% on MMMU, and +7.8% on V*.The largest listed GPT-4.1 improvement is on the fine-grained visual-search benchmark V*.
  • Backend strengths: Claude gains more on MathVision-mini while GPT-4.1 gains more on V*, matching their stronger abstract-reasoning and perceptual performance, respectively.Claude’s gains are +3.3% versus GPT-4.1’s +2.3% on MathVision-mini; GPT-4.1 gains +7.8% versus Claude’s +0.3% on V*.
  • Backend strengths: Qwen2.5-VL-72B shows the same pattern: +1.7% improvement on MathVision-mini with weaker reasoning and +10.0% on V* with stronger perception.Its corresponding base-model performances are 18.4% on MathVision-mini and 67.0% on V*.
  • Interaction patterns: Claude-4.0-Sonnet consistently generates more code than GPT-4.1 across six benchmarks, often using longer toolchains and reaching 100% code coverage.Figure 10 compares code-block counts per query and the percentage of sessions containing any code generation.

5. Related Work

Prior multimodal reasoning systems commonly use heuristic or learned program generation, predefined workflows, or static toolsets. PyVision instead uses Python-based dynamic tool creation during multi-turn reasoning, while related work also explores thinking with images through predefined tools.

  • Multi-Modal Tool Using: Early compositional VQA systems used heuristic orchestration, LSTM program generation, or pretrained LLMs to produce programs.Neural Module Networks and IEP are identified as examples of earlier approaches.
  • Multi-Modal Tool Using: Visual ChatGPT, MM-REACT, HuggingGPT, Image-of-Thought, and VAT design workflows for processing VQA inputs and producing answers.VisProg and ViperGPT additionally use predefined static toolsets for specific vision tasks.
  • Thinking with Images: Several recent approaches encourage MLLMs to think with images using predefined tools, whereas PyVision uses Python for tool creation during inference.The cited examples include CoGCoM, DeepEyes, Pixel Reasoner, OpenThinkIMG, and Chain-of-Focus.

6. Conclusion

PyVision is presented as an agentic framework that lets MLLMs generate and execute Python code on the fly for multimodal reasoning. Its evaluation across varied visual reasoning scenarios reports significant performance improvement.

  • Conclusion: PyVision enables MLLMs to generate and execute Python code on the fly as an agentic multimodal-reasoning framework.The contribution is framed around dynamically created computational tools rather than a fixed toolset.
  • Conclusion: PyVision dynamically creates tools from specific queries and visual inputs without visual parsers or predefined static toolsets.The framework is evaluated on medical, multimodal mathematics, remote sensing, and visual-puzzle scenarios.
  • Conclusion: The reported evaluations show significant performance improvement across versatile benchmarks and visual reasoning scenarios.The conclusion connects this improvement to PyVision’s evaluation across diverse application settings.

Appendix Contents

The appendix contains evaluation and system-prompt details alongside examples of generated tools. The listed examples span image processing, detection, segmentation, rendering, visualization, OCR, and numerical analysis.

  • Evaluation details: Appendix A lists additional evaluation details, system-prompt details, and evaluation-parameter details.These materials appear in sections A, A.1, and A.2.
  • Generated tools: Appendix B presents examples of generated tools, beginning with crop, rotate, enhance-contrast, segmentation, and detection snippets.These examples are listed in sections B.1 through B.5.
  • Generated tools: Additional generated-tool snippets cover OCR, rendering marks, auxiliary-line rendering, image-histogram visualization, and numerical analysis.These examples are listed in sections B.6 through B.10.

A. Additional Evaluation Details

PyVision’s evaluation prompt configures MLLMs to solve visual questions with executable Python assistance and structured outputs. The runtime exposes preloaded images and returns textual or visual results for iterative reasoning.

  • Prompt and runtime setup: The system prompt tells the agent to continue until the user’s query is completely resolved before ending its turn.It also asks for extensive planning and reflection around function calls.
  • Prompt and runtime setup: PyVision instructs the MLLM to solve visual problems step by step using selectively generated executable Python code.The external sandbox executes complete code snippets to support reasoning.
  • Prompt and runtime setup: Input images are preloaded as image_clue_i variables, with image resolution supplied to support operations such as cropping.The model can reference the images directly without adding loading code.
  • Prompt and runtime setup: Python outputs are returned through print() for textual results and plt.show() for visualizations, with code enclosed in <code> tags.These conventions enable reliable parsing of generated code and its outputs.
  • Prompt and runtime setup: Final answers are placed in an <answer> block using a boxed-answer format.The template repeats this requirement to standardize evaluation outputs.

A.2. Evaluation Details

The evaluation details specify a chain-of-thought prompting setup and document model-specific evaluation procedures. GPT-4.1 receives additional self-evaluation on MathVista and MMMU, while Qwen2.5-VL-72B results may be harmed by the prompt.

  • Evaluation setup: Inference uses temperature 0.6 together with a chain-of-thought prompt template for visual question answering.The template requests step-by-step solving and a boxed final answer.
  • Model-specific procedures: GPT-4.1 was evaluated on MathVista and MMMU using the chain-of-thought prompt to match Claude-4.0-Sonnet’s evaluation setting.These results were collected by the authors.
  • Model-specific procedures: Qwen2.5-VL-72B was evaluated on V* and MathVision-testmini with the same chain-of-thought prompt used for other models.Because Qwen2.5-VL-72B was not specifically optimized for chain-of-thought reasoning, the prompt may damage its results.

B.1. Code Snippet of C R O P Tool

The appendix illustrates how PyVision generates task-specific visual tools, ranging from simple cropping and rotation to enhancement, segmentation, OCR, marking, sketching, histograms, and numerical analysis. These examples show Python code being assembled into concrete operations for different visual questions.

  • Image processing and localization: PyVision generates a CROP tool to focus on a relevant advertising board before identifying the shop type.The example estimates image coordinates, crops the region, and displays it for inspection.
  • Image processing and localization: PyVision generates ROTATE and ENHANCE CONTRAST tools for orientation correction and improved inspection of subtle retinal abnormalities.The examples rotate an upside-down image by 180 degrees and enhance contrast in a fundus image.
  • Image processing and localization: A generated SEGMENTATION tool thresholds, morphologically cleans, labels, and area-filters regions to count buildings in satellite imagery.The pipeline uses Otsu thresholding, connected components, and a minimum building-area criterion.
  • Recognition and annotation: PyVision generates DETECTION and OCR tools to crop around a small cup and recognize text using EasyOCR.The OCR example converts the rotated image to an array before calling the reader.
  • Recognition and annotation: Generated RENDER MARKS and RENDER AUXILIARY LINES tools annotate people or spatial relationships to support counting and positional reasoning.The examples mark people in red shirts and sketch a directional move on a grid.
  • Analysis tools: PyVision also creates histogram-visualization and numerical-analysis tools to inspect patterns and calculate mean RGB values.These tools visualize pixel-intensity distributions or quantify the color of a selected image region.
Loading 2507.07998v3…