PaperlayerView exact revision on arXiv

Source-linked AI summary

Thyme: Think Beyond Images

Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Haonan Fan, Kaibing Chen, Jiankang Chen, Haojie Ding, Kaiyu Tang, Zhang Zhang, Liang Wang, Fan Yang, Tingting Gao, Guorui Zhou

arXiv:2508.11630v1cs.CV

TL;DR

Existing open-source “think with images” approaches do not provide the breadth of image manipulation and code-based reasoning described for proprietary systems. Thyme addresses this gap by training MLLMs to autonomously generate and execute image-processing and computational code, using SFT followed by RL with GRPO-ATS. Across nearly 20 benchmarks, it reports significant and consistent gains, while its evaluation and capabilities remain bounded by benchmark coverage and base-model limitations.

  • Problem

    Existing open-source “think with images” work lacks a comparably rich combination of diverse image manipulation and code-based reasoning capabilities.

  • Method

    Thyme uses two-stage training: SFT on a curated 500K-sample dataset, followed by RL with GRPO-ATS, which applies different temperatures to text and code generation.

  • Results

    Thyme achieves significant and consistent performance gains across perception, reasoning, and general tasks on nearly 20 multimodal benchmarks.

  • Takeaways & Limitations

    Thyme provides an autonomous framework for combining image manipulation and mathematical computation through executable code in multimodal reasoning.

  • Takeaways & Limitations

    Performance is constrained by base-model limitations, while existing benchmarks underrepresent rotated and low-contrast images needed to evaluate some enabled capabilities.

Abstract

from arXiv · show

Following OpenAI's introduction of the ``thinking with images'' concept, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning tasks. However, to the best of our knowledge, no open-source work currently offers a feature set as rich as proprietary models (O3), which can perform diverse image manipulations and simultaneously enhance logical reasoning capabilities through code. In this paper, we make a preliminary attempt in this direction by introducing Thyme (Think Beyond Images), a novel paradigm for enabling MLLMs to transcend existing ``think with images'' approaches by autonomously generating and executing diverse image processing and computational operations via executable code. This approach not only facilitates a rich, on-the-fly set of image manipulations (e.g., cropping, rotation, contrast enhancement) but also allows for mathematical computations, all while maintaining high autonomy in deciding when and how to apply these operations. We activate this capability through a two-stage training strategy: an initial SFT on a curated dataset of 500K samples to teach code generation, followed by a RL phase to refine decision-making. For the RL stage, we manually collect and design high-resolution question-answer pairs to increase the learning difficulty, and we propose GRPO-ATS (Group Relative Policy Optimization with Adaptive Temperature Sampling), an algorithm that applies distinct temperatures to text and code generation to balance reasoning exploration with code execution precision. We conduct extensive experimental analysis and ablation studies. Comprehensive evaluations on nearly 20 benchmarks show that Thyme yields significant and consistent performance gains, particularly in challenging high-resolution perception and complex reasoning tasks.

1 Introduction

Thyme enables MLLMs to autonomously generate and execute code for diverse image manipulations and mathematical computations, extending “think with images” through a two-stage SFT+RL training approach. Across nearly 20 benchmarks, it reports consistent gains across perception, reasoning, and general tasks.

  • The paradigm addresses limitations of existing “think with images” methods by incorporating visual inputs into reasoning while supporting executable image processing and computation.Existing approaches include generating images to guide subsequent reasoning, whereas Thyme generates and executes code for operations.
  • Thyme autonomously decides whether image-tool use is needed and dynamically generates code to perform selected operations without human intervention.Supported operations include cropping, scaling, rotating, contrast enhancement, and mathematical computation.
  • A two-stage SFT+RL strategy teaches foundational coding abilities and then refines decision-making and execution precision.The SFT stage uses a curated dataset of approximately 500K samples, while RL further strengthens the capabilities.
  • 200 GPU hours are required for SFT to activate the listed image-manipulation and computation capabilities, with RL further strengthening them.The authors present this as favorable computational efficiency.
  • Thyme achieves consistent and substantial performance improvements across perception, reasoning, and general tasks on nearly 20 multimodal benchmarks.The system can also combine multiple operations, such as cropping, zooming, and rotating, in one execution.

2 Preliminary and Overall Pipeline

Thyme’s pipeline lets the model decide whether code is needed, execute generated code in a sandbox, and use returned results for further reasoning. The sandbox adds security and robustness while correcting minor coding issues that could otherwise cause execution failures.

  • Overall Pipeline: The model first reasons about the problem and decides whether code generation is necessary; simple cases can be answered directly.When code is needed, the model autonomously generates it, potentially combining cropping, zooming, rotation, contrast enhancement, and computation.
  • Overall Pipeline: Generated code runs in an external sandbox, which returns execution results to the model for the next dialogue round.This supports iterative interaction and continued reasoning based on tool outputs.
  • SFT Data Construction Pipeline: The SFT data pipeline filters generated samples through sandbox execution, MLLM verification, and manual review before forming the cold-start dataset.Samples that fail execution, mismatch the reasoning or answer, or are low quality are removed.
  • Sandbox Building: The sandbox is designed to complete operations within reasonable time limits, raise errors appropriately, and prevent unauthorized system modifications.Security and robustness are essential properties of the execution environment.
  • Sandbox Building: Sandbox preprocessing reduces coding errors by formatting code, correcting variable and boundary issues, presetting inputs, and preserving execution context across code segments.These measures target indentation, cropping bounds, input/output handling, imports, and multi-round dependencies.
  • Sandbox Building: Dangerous operations such as remove, unlink, move, and rename are blocked before execution, while runs exceeding 10 seconds produce timeout errors.These controls isolate generated code from local system data and configurations.

3 Thyme-SFT Cold Start

Thyme-SFT cold start constructs diverse training data spanning direct answers, image-processing and computational code, and multi-round interaction. The pipeline filters and verifies executable examples while teaching models to revise image-manipulation errors, although jointly training the relatively small math subset poses a challenge.

  • Data Composition: The cold-start datasets are organized into direct-answer, code-based image/computation, and multi-round interaction tasks of increasing complexity.The source data includes multiple open datasets spanning perception, reasoning, and iterative manipulation scenarios.
  • Data Composition: 100k no-tool samples train the model to decide when coding is unnecessary and answer straightforward image questions directly.Qwen2.5-VL-72B assesses whether code generation is needed before sample selection.
  • Data Quality: Code snippets are sandbox-checked for executability, and execution outputs plus analyses and answers are used for further verification.The verification process discards non-executable code and checks whether outputs achieve intended goals.
  • Data Composition: The code-focused data covers cropping, rotation, contrast enhancement, complex computations, and multi-turn refinement of failed operations.Examples include rotated images, low-contrast text recognition, mathematical code, and iterative cropping or correction.
  • Multi-Round Training: Multi-round examples teach the model to revise incorrect crops and adjust bounding boxes after earlier manipulation errors.The training data explicitly includes further enhancement and error-correction interactions, improving operation quality.
  • Training Challenges: Joint training struggles with computation-related code because math data are relatively scarce compared with image-manipulation data.The paper also reports first-round analysis and code becoming ineffective when correction is deferred to the second round.

4 Thyme-RL

Thyme-RL constructs challenging high-resolution visual question-answer data and trains the model with on-policy GRPO, adaptive sampling, and structured rewards. The design improves code usability during rollouts while reducing code avoidance and supports diverse perception and reasoning tasks.

  • RL Data Construction and Annotation: The RL dataset combines public perception and reasoning sources with manually constructed high-resolution visual questions targeting small, difficult objects.The manually collected images exceed 2048 pixels in width or height, and target objects occupy no more than 5% of image resolution.
  • RL Data Construction and Annotation: Questions cover OCR, attributes, position, quantity, object recognition, and chart understanding to diversify the visual reasoning demands.The task set includes identifying text, colors or shapes, object locations, object counts, categories, and chart or table values and trends.
  • Reinforcement Learning Algorithm: GRPO samples complete multi-round interaction trajectories and updates the policy using group-relative advantages while excluding sandbox observations from token-level training calculations.Trajectory rewards evaluate code generation, sandbox use, and final-answer correctness; sandbox content is treated as external observation.
  • GRPO with Adaptive Temperature Sampling: GRPO-ATS sets τ=0 for code generation to prioritize accuracy and τ=1 for reasoning generation to encourage exploration.The sampling design reflects the different requirements of predictable code execution and exploratory textual reasoning.
  • GRPO with Adaptive Temperature Sampling: Lower code temperature improves sample efficiency by recovering valid rollouts and mitigates training collapse toward avoiding code generation.Invalid or repetitive code can otherwise exhaust interaction steps, reduce usable samples, and make the model overly conservative about writing code.
  • Reward Function Design: The reward design combines formatting, result, and consistency signals while granting the consistency bonus only when the final answer is correct.Result evaluation uses rule-based matching first and an auxiliary MLLM when direct matching fails.

5 Experiments

Thyme is evaluated across perception, reasoning, and general-task benchmarks against open- and closed-source multimodal baselines. Results and ablations show broad gains, especially on difficult perception tasks, while targeted SFT and reward strategies shape code-use behavior.

  • 5.1 Benchmarks and Baselines: Thyme is compared with Qwen2.5-VL-7B, InternVL3-8B, Qwen2.5-VL-32B, and GPT-4o using VLMEvalKit.The evaluation covers perception, reasoning, and general tasks.
  • 5.3 Evaluation Results: Thyme achieves consistent and substantial improvements across nearly 20 benchmarks spanning perception, reasoning, and general-task categories.The reported gains include a substantial reduction in hallucination on many general tasks.
  • 5.3 Evaluation Results: Over 25% improvement appears in both perception and reasoning tasks for MME-Realworld monitoring and autonomous-driving domains.These are challenging domains where the Qwen2.5-VL-7B baseline performs relatively poorly; gains are limited on already-strong OCR, Diagram, and Table tasks.
  • 5.4.1 The Impact of Training Strategies on the SFT Process: Directly mixing all SFT data produces chaotic outputs, whereas masking sandbox content and training only on the final interaction round significantly improves performance.Final-round training also prevents the model from learning a pattern of initially generating suboptimal code and correcting it later.
  • 5.4.1 The Impact of Training Strategies on the SFT Process: Removing code comments produces unsatisfactory results, while math-data annealing more effectively introduces code-based complex-calculation behavior despite limited additional performance gains.The authors associate comments with greater logicality during code writing and use annealing because mixed training rarely elicits mathematical computation code.
  • 5.4.2 The Influence of RL Reward Design on the RL Process: Consistency rewards improve performance, but process-quality rewards can hurt performance because subjective reasoning scores are easy to hack.Code rewards encourage more code generation, yet unnecessary code does not improve performance.
  • 5.4.3 Analysis of the RL Learning Process: During RL, response length rapidly decreases before converging as the model learns that direct textual responses are more efficient for most tasks.The result reward trends upward, while the initial gap between result reward and consistency reward indicates that some correct answers lack coherent reasoning.

5.5 Case studies

Thyme improves visual understanding by autonomously applying targeted image operations such as cropping, zooming, rotation, and contrast enhancement. The case studies show these operations helping recover small, misoriented, faint, or otherwise difficult-to-read visual information.

  • Cropping & Zooming: Cropping and zooming enabled Thyme to locate a small street sign and correctly answer Michigan.The model identified the sign’s position, enlarged the relevant region, and selected option D.
  • Cropping & Zooming: Cropping and zooming also revealed the phone number 206 441 5000 on a foreground building.The vertically arranged digits became readable after processing the relevant building region, yielding option A.
  • Cropping & Zooming: Cropping and zooming established that the image contained no awning-tricycles, producing option E.Thyme focused on the lower-left region before concluding that no target objects were visible.
  • Rotation & Contrast Enhancement: Rotation made a slanted formula legible, allowing Thyme to represent the repeating decimal 0.642857 with a bar over the repeating digits.The model rotated the image before interpreting the mathematical expression.
  • Rotation & Contrast Enhancement: Contrast enhancement improved readability of faint text, enabling Thyme to identify the word communities.The original text blended into the background, while the enhanced image made it significantly more legible.

5.6 Bad Cases

The bad cases show that Thyme’s autonomous tool use can be poorly calibrated. It may omit a necessary crop, invoke code unnecessarily, confuse variables, or crop an irrelevant region.

  • Complex Problems Without Coding: Telehandlers were selected as having the highest occupancy rate, although the listed percentages were internally inconsistent.The passage reports Telehandlers at 72.2% while also listing Excavators at 78.8%.
  • Complex Problems Without Coding: Thyme answered directly without the necessary crop despite a high-resolution image requiring focus on the relevant area.This failure illustrates that direct answering can be inappropriate for complex visual questions.
  • Complex Problems With Coding: Thyme correctly solved the triangle problem by computing a as 4.5.The response used the provided calculation and returned the numeric answer directly.
  • Complex Problems With Coding: A trivial calculation caused unnecessary code generation and an incorrect result after Thyme confused variable a with variable b.The failure combines excessive tool use with a variable-identification error.
  • Complex Problems With Coding: Thyme correctly identified the train model as 2001 after cropping the front-panel region.The cropped image confirmed option D.
  • Complex Problems With Coding: Thyme sometimes reaches the correct answer while cropping an irrelevant region during its reasoning process.The operation may therefore be unnecessary even when the final answer is correct.

6 Conclusion and Limitations

Thyme enables MLLMs to generate and execute code for image manipulation and computation through two-stage training and GRPO-ATS. Evaluations across nearly 20 benchmarks report consistent gains, while the method remains constrained by base-model capability and benchmark coverage.

  • Conclusion: Thyme autonomously generates and executes code for image manipulations and complex computations using SFT followed by RL with GRPO-ATS.The training begins with 500K curated samples, and GRPO-ATS uses different temperatures for text and code generation.
  • Conclusion: Evaluations across nearly 20 benchmarks show significant and consistent improvements, especially on high-resolution perception and complex reasoning tasks.The reported gains are described relative to the baseline.
  • Limitations: Thyme’s reliability is constrained by the base model’s limited object localization and code-generation capabilities.These limitations can produce incorrect cropping operations or non-standard, unexecutable code.
  • Limitations: Existing benchmarks underrepresent rotated and low-contrast images, limiting numerical evaluation of those manipulation capabilities.The authors call for benchmarks designed specifically to assess advanced image-processing operations.

A Annotation Requirements

The annotation requirements document is intended to ensure that each image accurately describes target-object characteristics and provides clear annotation information to users.

  • Annotation Requirements: The annotation process requires images to accurately describe target-object characteristics and provide clear information to users.The document frames these as its central annotation goals.

A.1 Task Description

The task requires annotators to create question-answer pairs for high-resolution images, focusing on small, difficult-to-recognize objects.

  • Each high-resolution image receives an annotated question targeting small, difficult-to-recognize objects.
  • Each image also receives an accurate answer derived from analyzing the relevant image regions.

A.2 Annotation Requirements

Annotation requirements constrain target objects and bounding boxes while requiring questions that test varied visual recognition abilities.

  • Target objects must occupy no more than 5% of the image area and require zooming for clear identification.
  • Bounding boxes need not be pixel-level but must fully contain the object and clearly display it after cropping.
  • Questions should cover OCR, attribute, location, and quantity recognition.

A.3 Annotation Steps

The annotation workflow proceeds from reviewing the image to localizing a small target, drawing its bounding box, and designing a question.

  • Annotators first inspect the high-resolution image, emphasizing small and hard-to-recognize objects.
  • They then localize an object occupying no more than 5% of the image area and draw a box that fully contains it.
  • The workflow concludes by designing a question for the selected target.

B Related Work

The related work and training templates frame multimodal reasoning as increasingly capable but still limited in how actively visual inputs participate in reasoning. Thyme-style training instead requires models to decide whether image processing is needed and generate executable operations when appropriate.

  • Related work: Modern MLLMs support high-resolution image processing and complex dialogue, while related work has explored multimodal reasoning and “think with images” paradigms.
  • Related work: Existing visual-reasoning approaches are described as limited to functions such as cropping or generating an auxiliary image.
  • Training templates: The training prompt first asks whether the image can answer the question directly or requires processing.
  • Training templates: When processing is needed, the prompt requires diagnosing image issues, selecting an operation category, and generating executable Python code.
  • Training templates: Combined image issues are handled through multi-step processing reflected in the generated code.
  • Training templates: The templates specify output conventions for processed-image paths, code delimiters, tool identifiers, and direct-answer cases.
Loading 2508.11630v1…