Source-linked AI summary
Thinking with Programming Vision: Towards a Unified View for Thinking with Images
Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, Tao Jin
TL;DR
MLLMs remain brittle to orientation changes, while existing visual-tool methods are narrow, manually specified, and weakly suited to multi-turn composition. CodeVision uses generated code as a universal image-tool interface, trained with SFT and dense-reward RL, and significantly improves benchmark performance while enabling emergent tool use, efficient chaining, and error recovery. Its demonstrated scope is centered primarily on orientation correction and cropping, with broader general-purpose visual agency left for future work.
Problem
Existing visual-tool methods provide limited gains, depend on fixed tool interfaces, and do not robustly support diverse multi-turn, multi-tool reasoning despite MLLMs’ brittleness to orientation changes.
Method
CodeVision treats generated code as a universal interface for image operations and trains it through SFT on multi-turn tool data followed by RL with dense process rewards.
Results
The approach significantly improves performance on challenging benchmarks and enables emergent unseen-tool use, efficient tool chaining, and robust error recovery.
Takeaways & Limitations
Code-as-tool reasoning provides a flexible alternative to fixed registries for robust, compositional visual problem solving.
Takeaways & Limitations
The demonstrated framework focuses primarily on orientation correction and cropping, while a truly general-purpose visual agent remains future work.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) that think with images can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tools with limited real-world necessity and scalability. In this work, we first reveal a critical and previously overlooked weakness: even state-of-the-art MLLMs are surprisingly brittle, showing significant performance degradation on images with simple orientation changes or natural corruptions, underscoring the need for more robust tool-based reasoning. To address this, we propose CodeVision, a flexible and scalable code-as-tool framework where the model generates code as a universal interface to invoke any image operation, moving beyond fixed tool registries. We train our model using a two-stage methodology, beginning with Supervised Fine-Tuning (SFT) on a high-quality dataset curated for complex, multi-turn tool composition and error recovery, followed by Reinforcement Learning (RL) with a novel and dense process reward function to encourage strategic and efficient tool use. To facilitate this research, we construct new SFT and RL datasets and introduce a challenging new benchmark suite designed to rigorously evaluate robustness to orientation changes and multi-tool reasoning. Experiments on Qwen2.5-VL and Qwen3-VL series show that our approach significantly improves model performance and fosters emergent capabilities such as flexible tool composition, efficient chained execution, and robust error recovery from runtime feedback. Code is available at https://github.com/ByteDance-BandAI/CodeVision.
1. Introduction
The paper identifies brittleness and limited scalability in image-tool reasoning, then proposes CodeVision, which uses code as a universal interface and is trained for multi-turn, multi-tool problem solving.
- Motivation: Existing methods often yield only 2–5% gains from tools, rely on manually specified interfaces, and emphasize repeated cropping instead of diverse multi-turn tool composition.These limitations constrain tool necessity, scalability, and realistic task coverage.
- Motivation: State-of-the-art MLLMs perform poorly on identifying five simple orientation transformations, exposing brittleness that motivates more robust visual tool reasoning.The diagnostic applies rotations and flips to 200 images; humans achieve 100% accuracy.
- CodeVision: CodeVision treats code as a unified tool interface, allowing models to invoke an effectively unbounded set of image operations without fixed tool registries.The framework targets flexibility and generalization to new tools and argument schemas.
- CodeVision: CodeVision exhibits emergent tool use, single-execution chaining, and runtime-feedback-based error recovery, including tools absent from RL training data.These advantages are attributed to the code-as-tool paradigm together with principled training and dataset construction.
- Evaluation and training: The work constructs datasets and benchmarks for multi-turn, multi-tool composition and error handling to evaluate robustness beyond single-tool scenarios.The stated contributions include high-quality SFT and RL data plus three benchmarks covering single- and multi-tool settings.
- Evaluation and training: Experiments significantly improve performance on the challenging benchmarks while validating efficient tool chaining and robust error recovery.The reported improvements cover the proposed robustness and complex tool-use evaluation setting.
2. Related Work
Prior work equips MLLMs with visual tools, but image-centric systems remain centered on narrow, pre-registered interfaces and limited tool composition. CodeVision addresses these open challenges with broader code-mediated tool use and multi-turn evaluation.
- Thinking with Images: Image reasoning research has largely centered on crop or zoom tools, whose benefits can be marginal or difficult to observe on many tasks.Broader suites including OCR, line drawing, and segmentation have also been explored, but remain less central.
- Tool Integration: Tool-integrated reasoning systems use capabilities such as search, code execution, diffusion, and web search, yet image-centric systems mostly retain hand-specified tool sets.This contrast motivates a more flexible interface for visual operations.
- MLLM Reasoning: MLLM reasoning research increasingly uses reinforcement learning and explicit visual inspection, while robust multi-turn composition and generalization to unseen tools remain open challenges.The unresolved scope concerns combining different tools across multiple steps rather than merely repeating one operation.
- Positioning: The paper introduces code-mediated tool invocation and datasets designed for multi-turn, multi-tool composition to address these limitations.Its approach targets flexible tool use beyond fixed registries and repeated cropping.
3. Methodology
CodeVision trains robust visual tool use through a cold-start SFT stage followed by RL with dense rewards for necessary, strategic, and efficient tool use. Its data pipeline covers multi-turn composition, localization, error recovery, and cached tool interactions before policy optimization.
- Two-stage training: SFT followed by RL teaches tool-use syntax first, then deliberate, efficient, and robust strategy selection.SFT covers multi-tool sequences, error handling, and coarse-to-fine localization; RL uses a dense multi-component reward.
- SFT data construction: The SFT corpus spans diverse domains and task types, including single-tool, multi-tool, multi-crop, error-handling, and no-tool cases.Sources include handwriting, in-the-wild OCR/VQA, chart understanding, and math reasoning datasets; crop and multi-crop examples target fine-grained localization.
- SFT data construction: Metadata-conditioned transformations make selected tool calls necessary by presenting altered images that must be restored to a canonical view.For example, a rotate-180 requirement is created by rotating the original image before providing it as the initial input.
- SFT data construction: Approximately 5,000 verified SFT examples align multi-turn reasoning, action sequences, tool returns, and final answers.GPT-5-generated trajectories are iteratively updated with metadata and checked against canonical untransformed references; failures may be discarded or corrected.
- SFT training: Masked causal training optimizes only assistant reasoning and tool-call tokens while retaining multi-turn context and cached tool returns.User and tool-return tokens are masked, and tools are not executed online during SFT.
- RL reward design: The RL reward combines terminal accuracy and formatting, process-level strategy shaping, and penalties for inefficient trajectories or reward hacking.Required tools receive structured rewards, while optional beneficial tools receive rollout-based necessity bonuses and additional successful-use bonuses.
4. Experiments
CodeVision is evaluated on orientation robustness, single- and multi-tool benchmarks, qualitative tool composition, error recovery, and reward-design ablations. It improves transformed-image and multi-tool performance while revealing remaining challenges in precise localization and efficient behavior.
- Main Results: 73.4 on transformed OCRBench is a +17.4 improvement over CodeVision-7B’s base model, demonstrating robust orientation correction.The evaluation applies rotations and flips to OCRBench and ChartQAPro to test perception-critical and reasoning-heavy scenarios.
- Main Results: 60.1 on MVToolBench nearly doubles Gemini2.5-Pro’s 32.6 while CodeVision remains competitive on single-tool cropping benchmarks.MVToolBench evaluates multi-tool reasoning, whereas V*, HRBench4k, and HRBench8k evaluate cropping.
- Qualitative Capabilities: The model chains novel image operations, recovers from incorrect tool calls, and performs iterative cropping through multi-turn reasoning.Examples include contrast-plus-grayscale chaining, correction from flip-horizontal to rotate-90, and a second crop after recognizing an incomplete first crop.
- Limitations: Fine-grained localization remains a limitation: the model can reason about a target but sometimes produces loose or slightly misplaced crop coordinates.One case uses an overly broad crop, while another misses the target entirely despite successful orientation correction and coarse localization.
- Ablations: Removing strategy rewards lowers MVToolBench from 60.1 to 50.7, showing that dense process signals guide effective tool-use strategies.The ablation reports substantial degradation across all benchmarks after removing the strategy reward.
- Ablations: Constraint penalties prevent reward hacking, where unnecessary orientation operations corrupt an already-correct image and cause failure.Without these penalties, the model may continue rotating or cropping after solving the task, producing inefficient trajectories.
- Ablations: SFT is necessary before RL because direct RL on the base model fails to achieve meaningful improvement in code’s vast action space.The two-stage comparison supports using SFT as a cold start for subsequent strategic RL training.
5. Conclusion
The paper identifies MLLM brittleness to orientation changes and introduces CodeVision, a code-as-tool framework trained with SFT and RL to improve visual reasoning and tool use.
- CodeVision combines a flexible code-as-tool framework with SFT and dense-reward RL to address MLLM brittleness to orientation changes.The framework treats code as a universal interface for visual operations and trains models to use tools strategically.
6. Limitations and Future Work
The paper’s current scope centers on orientation correction and cropping, leaving broader tool diversity, compositionality, process supervision, and scaling as future directions.
- The current framework focuses primarily on orientation correction and cropping, limiting its demonstrated scope.The authors identify scaling and expansion as necessary for a truly general-purpose visual agent.
- Expanding Tool Diversity and Compositionality: Future work should expand tool diversity, complex compositions, and multi-image operations to improve generalization.Suggested directions include comparing, merging, or analyzing multiple images and extending code-as-tool beyond standard Python functionality.
- Refining Process Supervision with Beneficial Tools: Process supervision currently relies on a must-use tool list, which could be extended with beneficial tools that improve performance without being necessary.Examples include contrast enhancement for low-light images and preliminary crops; rewarding such tools may support more nuanced strategies.
- Scaling Data, Tasks, and Model Exploration: Benchmark performance continues increasing without a plateau, while policy entropy indicates room for exploration, motivating larger data, task, tool, and model scales.The authors view broader data and task diversity as a path toward more capable and robust visual agents.