Source-linked AI summary
TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang
TL;DR
Existing benchmarks largely assess visual search, leaving broader tool-dependent image reasoning insufficiently measured. TIR-Bench introduces 13 diverse tasks and evaluates 22 MLLMs, finding that strong performance requires agentic thinking-with-images capabilities while revealing task-specific limitations.
Problem
Existing visual-search benchmarks mainly test localization and cropping, leaving broader thinking-with-images abilities unevaluated.
Method
TIR-Bench evaluates agentic thinking-with-images across 13 diverse tool-dependent tasks and compares 22 MLLMs, including tool-using agents.
Results
TIR-Bench is challenging: o3-TU achieves the highest average accuracy at 46%, outperforming Gemini-2.5-Pro by nearly 17% and winning 10 of 14 categories.
Takeaways & Limitations
Successful completion of TIR-Bench requires thinking-with-images capabilities, while traditional non-agentic models perform poorly across tasks.
Takeaways & Limitations
Performance improvements are not uniform, and o3-TU lacks the ability to call segmentation models needed for some proportion-estimation tasks.
Abstract
from arXiv · showhide
The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking-\textit{with}-images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking-\textit{with}-images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce \textbf{TIR-Bench}, a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning.
1 INTRODUCTION
TIR-Bench addresses the narrow scope of existing visual-search evaluations by testing agentic thinking-with-images across diverse tool-dependent tasks. Evaluation of 22 MLLMs shows that the benchmark is challenging and that tool-use capabilities are associated with substantially stronger performance.
- Thinking-with-images extends textual reasoning by letting models create and execute image-processing tools whose transformed outputs inform subsequent reasoning.
- Existing benchmarks mainly test visual search through localization and cropping, leaving broader thinking-with-images abilities unevaluated.
- TIR-Bench introduces 13 tasks requiring diverse image manipulations, including zooming, rotation, contrast enhancement, auxiliary-line drawing, and puzzle reassembly.
- TIR-Bench evaluates 22 MLLMs across open-source, proprietary, and tool-using categories, with the best overall performance reaching only 46%.
- Non-agentic models perform poorly, while tool-using models such as o3, o4-mini, and PyVision achieve much higher performance on TIR-Bench.
- The paper also compares function-calling proficiency and direct versus agentic fine-tuning for image-operation tasks.
2 RELATED WORKS
Prior work develops multimodal reasoning systems with predefined visual tools, while benchmark research remains specialized and does not fully cover broad agentic image manipulation.
- Existing MLLM benchmarks are typically specialized for particular domains or evaluation settings rather than broadly assessing agentic thinking-with-images.
- Earlier methods generate tool calls from predefined visual-specific toolsets and reinject processed images into multimodal reasoning.
3 TIR-BENCH
TIR-Bench is a 13-task benchmark designed to test active, tool-based visual reasoning beyond static image analysis. It combines newly created, sourced, and programmatically generated data across reasoning and perception tasks.
- 3.1 TASK DESIGN: TIR-Bench contains 13 tasks designed to evaluate active tool-based visual reasoning beyond static image analysis.
- 3.1 TASK DESIGN: The task design includes programmatic analysis, external object-segmentation calls, and image enhancement for challenging visual conditions.
- 3.1 TASK DESIGN: The benchmark includes reasoning and perception tasks spanning math, symbolic reasoning, low-light VQA, instrument reading, color analysis, jigsaw, maze, rotation, word search, visual search, and proportion VQA.
- 3.2 DATA COLLECTION: Data collection combines newly annotated internet or generated images, curated ColorBench and RefCOCO examples, and procedurally generated puzzles, mazes, rotations, and word-search challenges.
- 3.2 DATA COLLECTION: Rotated-image OCR uses 60 OCRBench images with rotations distributed as 25% at 90°, 25% at 270°, and 50% at 180°.
- 3.3 BENCHMARK SUMMARY: The dataset contains 1,215 examples, with 665 multiple-choice questions and 550 free-form problems.
4 EXPERIMENTS
The experiments evaluate 22 MLLMs on TIR-Bench, compare agentic and non-agentic reasoning, and examine function calling and fine-tuning strategies. Agentic tool use substantially improves performance, but the benchmark remains difficult and gains are uneven across tasks.
- 4.2 EXPERIMENTAL RESULTS: 46% is the highest overall accuracy, showing that TIR-Bench challenges even advanced tool-using models such as o3-TU.The benchmark evaluates models across diverse tasks requiring thinking with images.
- 4.2 EXPERIMENTAL RESULTS: Non-agentic models perform poorly across tasks, with most open-source models near random-guess performance and Gemini-2.5-Pro surpassing it by only 15%.These results indicate that ordinary multimodal reasoning is insufficient for TIR-Bench.
- 4.2 EXPERIMENTAL RESULTS: o3-TU achieves 46% average accuracy, outperforming Gemini-2.5-Pro by nearly 17% and o3 without a code interpreter by 19%.It achieves state-of-the-art results on 10 of 14 categories.
- 4.2 EXPERIMENTAL RESULTS: Tool-use gains are uneven: o3-TU remains weak on Jigsaw Game, while Proportion VQA accuracy drops from 34.2% to 31.7%.Proportion VQA requires external segmentation models, which o3-TU cannot currently call directly.
- 4.4 FUNCTION CALL EXPERIMENT RESULTS: Prompt 3, which guides models to check each degree choice, produces the best function-calling performance for most models.Recent models such as o3 also make more function calls on average than earlier models such as GPT-4o.
- 4.5 FINE-TUNING COMPARISON EXPERIMENT RESULTS: Tool-use SFT significantly outperforms Direct SFT, scales positively with training-data size, and reduces loss faster on rotated-image OCR.Direct SFT shows no comparable data-scaling trend, while restoring image orientation may avoid forgetting.
5 CONCLUSION
The paper proposes TIR-Bench to evaluate agentic MLLMs’ thinking-with-images abilities across diverse tool-assisted reasoning tasks. It finds the benchmark challenging for models and compares direct with agentic fine-tuning while also evaluating function calling.
- TIR-Bench evaluates thinking-with-images through 13 tasks spanning diverse tool-assisted reasoning skills.
- The evaluation covers standard MLLMs and models augmented with tool-use capabilities, testing whether successful completion requires thinking with images.
- The study also compares direct and agentic fine-tuning for image-operation tasks and evaluates MLLMs’ function-calling abilities.
B.1 TASK DESIGN
TIR-Bench’s task design targets active, iterative visual reasoning rather than static image analysis. Its tasks require models to manipulate images, apply programmatic analysis, and visually assess intermediate results.
- The 13 tasks evaluate active tool-based visual reasoning, including programmatic analysis, image enhancement, orientation correction, and external object segmentation.
- Maze solving combines image processing, pathfinding, and drawing the solution path back onto the image.
- Other tasks test color measurement, geometric augmentation, fine-grained character comparison, low-light enhancement, instrument reading, and programmatic difference detection.
- Jigsaw Puzzle requires repeated segmentation, reassembly, and self-correction based on evaluating each candidate configuration.
- Rotation Game requires testing multiple angles and visually evaluating each transformed image until the orientation is corrected.
C.1 COLOR
The supplied materials provide example figures for the benchmark’s task categories and implementation prompts for iterative image processing and rotation. They do not state outcome comparisons for these figures or the table.
- Additional examples cover Color, Low-Light, Instrument Reading, Jigsaw, Math, Maze, Rotated OCR, Proportion, Rotation, Spot the Difference, Symbolic Reasoning, Visual Search, and Word Search.
- The code-writing prompt strategy asks models to process the image and continue from returned processed images or text.
- The rotation prompt strategies ask models to call or write rotation functions, try multiple degrees, and proceed after inspecting the rotated image.
- Table 3 reports average function-calling counts across prompt and calling strategies.
E IMPLEMENTATION DETAILS
The evaluations use zero-shot settings with deterministic decoding for open models and official APIs for proprietary models.
- All evaluations are zero-shot, with open-model experiments on NVIDIA A100 GPUs and proprietary-model experiments through official APIs.
- Open models use do sample = False, temperature = 0, and a maximum of 2048 new tokens.
F EXAMPLES OF MODEL RESPONSES
The examples show models solving image-based distance questions with varying accuracy and levels of measurement detail. One response explicitly converts pixel distances using the scale before computing travel time.
- The distance question has ground truth 4, while o3-TU and o4-mini-TU answer 3 and o3 and o4-mini answer 3 and 2.
- 4 hours is the final answer produced by Gemini-2.5-flash after converting measured pixel distances into kilometers.It computes AC and CB from pixel coordinates, sums them to 261.88 km, and divides by 60 km/h.
- Several figures provide additional examples of model responses, including proportion estimation and word searching cases.