PaperlayerView exact revision on arXiv

Source-linked AI summary

Thyme: Think Beyond Images

Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Haonan Fan, Kaibing Chen, Jiankang Chen, Haojie Ding, Kaiyu Tang, Zhang Zhang, Liang Wang, Fan Yang, Tingting Gao, Guorui Zhou

arXiv:2508.11630v1cs.CV

TL;DR

As MLLMs explore incorporating visual inputs into reasoning, open-source approaches still lack the breadth of image manipulation and code-based computation offered by proprietary models. Thyme uses executable code for image operations and computations, and reports consistent, substantial gains across perception, reasoning, and general tasks.

  • Problem

    Existing MLLM approaches explore incorporating visual inputs into reasoning, including using generated images to guide subsequent reasoning.

    Source: §1 · p. 3

  • Method

    Thyme lets MLLMs decide when code is needed and generate executable code for image manipulations and computations, with SFT followed by RL using GRPO-ATS.

    Source: §2.1 · p. 4, §6 · p. 24

  • Results

    Thyme achieves consistent and substantial performance improvements across perception, reasoning, and general-task categories.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Reported gains across perception, reasoning, and general tasks indicate that Thyme’s performance is not confined to a single task category.

    Source: §1 · p. 3

    Existing benchmarks underrepresent rotated and low-contrast images, limiting numerical evaluation of Thyme’s rotation-correction and contrast-enhancement capabilities.

    Source: §6 · p. 24

Abstract

from arXiv · show

Following OpenAI's introduction of the ``thinking with images'' concept, recent efforts have explored stimulating the use of visual information in the reasoning process to enhance model performance in perception and reasoning tasks. However, to the best of our knowledge, no open-source work currently offers a feature set as rich as proprietary models (O3), which can perform diverse image manipulations and simultaneously enhance logical reasoning capabilities through code. In this paper, we make a preliminary attempt in this direction by introducing Thyme (Think Beyond Images), a novel paradigm for enabling MLLMs to transcend existing ``think with images'' approaches by autonomously generating and executing diverse image processing and computational operations via executable code. This approach not only facilitates a rich, on-the-fly set of image manipulations (e.g., cropping, rotation, contrast enhancement) but also allows for mathematical computations, all while maintaining high autonomy in deciding when and how to apply these operations. We activate this capability through a two-stage training strategy: an initial SFT on a curated dataset of 500K samples to teach code generation, followed by a RL phase to refine decision-making. For the RL stage, we manually collect and design high-resolution question-answer pairs to increase the learning difficulty, and we propose GRPO-ATS (Group Relative Policy Optimization with Adaptive Temperature Sampling), an algorithm that applies distinct temperatures to text and code generation to balance reasoning exploration with code execution precision. We conduct extensive experimental analysis and ablation studies. Comprehensive evaluations on nearly 20 benchmarks show that Thyme yields significant and consistent performance gains, particularly in challenging high-resolution perception and complex reasoning tasks.

1 Introduction

Thyme enables MLLMs to autonomously generate code for image manipulations and mathematical computations, extending image-based reasoning beyond generated-image or cropping-only approaches. It is trained with SFT followed by RL and reports consistent gains across perception, reasoning, and general tasks.

1 Introduction

  • Generated-image methods can lose fine-grained details and add inference latency, while cropping-based methods enhance perception but are limited to bounding-box-selected regions (Chern et al., 2025; Zhang et al., 2025a; Wu et al., 2025; Shen et al., 2024; Zhang et al., 2025b; Huang et al., 2025a; Shao et al., 2024a; Zheng et al., 2025; Team et al., 2025).The introduction contrasts these limits with O3's broader operations, including rotation, contrast enhancement, and coding.Source: §1 · p. 3
  • Thyme autonomously decides whether and how to use code for image operations and computations, including cropping, scaling, rotation, and contrast enhancement.It can combine multiple operations, such as cropping, zooming, and rotating, in one execution.Source: §1 · pp. 3–4
  • Thyme reports consistent, substantial gains across perception, reasoning, and general tasks on nearly 20 benchmarks.The introduction characterizes these results as significant and consistent across the evaluated categories.Source: §1 · p. 3
  • A curated SFT dataset of approximately 500K samples spans image operations, high-resolution cropping, image correction, code-based computation, and multi-turn interaction.The SFT phase requires about 200 GPU hours to activate fundamental image-manipulation and computation abilities; training masks sandbox feedback and uses only the final response in multi-turn dialogues for loss.Source: §1 · pp. 3–4
  • For RL, Thyme adds 10,000 manually collected, high-resolution challenging images and uses GRPO-ATS with separate temperatures for text and code generation.Text temperature 1.0 encourages exploration, while code temperature 0.0 reduces randomness and runtime errors.Source: §1 · p. 4

2 Preliminary and Overall Pipeline

Thyme’s model decides whether a problem needs code, generates image-processing or computational operations autonomously, and uses a sandbox to execute them and return results for further reasoning. The sandbox is designed to protect the system while reducing errors and coding burden.

2.1 Overal Pipeline of Thyme

  • The model analyzes each problem and either answers directly or generates code, choosing operations and parameters such as crops, zooms, rotations, contrast changes, and computations.The decision depends on problem type and difficulty, and multiple operations may be combined in one code sequence.Source: §2.1 · p. 4
  • After execution, results return to the model, which reasons from them and continues the interaction in the next dialogue round.Sandbox handling of formatting and input-output properties is intended to avoid errors from minor code details and improve code usability.Source: §2.1 · p. 4

2.2 Sandbox Building

  • To isolate execution from the local system, the sandbox rejects code containing dangerous operations such as remove, unlink, move, or rename and stops runs exceeding 10 seconds.These safeguards aim to prevent unauthorized system changes and ensure operations finish within a reasonable time or produce errors.Source: §2.2 · p. 5
  • The sandbox formats code, repairs common input and output issues, adjusts inferred crop coordinates to image bounds, and preserves variables and context across code segments.It uses autopep8 and AST-based checks, presets variables such as image_path and imports such as cv2, and identifies processed outputs even when code omits explicit output variables.Source: §2.1 · p. 4, §2.2 · p. 5

3 Thyme-SFT Cold Start

Thyme-SFT builds training data spanning direct answers, code-assisted image and math tasks, and multi-round refinement. Its training strategy addresses weak first-round behavior and underrepresented math data.

3.1 Training Data Construction Pipeline

  • Training data spans no-code answers, code-based image or math operations, and multi-round tasks, drawing on diverse open datasets.The three categories increase in complexity, from direct answers to operations such as cropping, rotation, and contrast enhancement, then multi-round refinement and reasoning.Source: §3.1 · pp. 5–6
  • The pipeline selects 100k no-code samples and 50k high-quality executable real-world QA pairs after code, outcome, and expert screening.Qwen2.5-VL-72B assesses whether code is needed for the no-code subset; real-world samples are drawn from more than 4 million instances.Source: §3.1 · p. 6
  • Manually constructed data adds 28k cropping, 14k rotation, approximately 10k contrast-enhancement, and 15k computational-code samples.The datasets use ground-truth boxes or rotation angles for image operations; computational examples transform long-chain solutions into executable code.Source: §3.1 · p. 7
  • Multi-round data teaches the model to use execution history to refine manipulations and correct earlier errors, addressing failures in single-turn interactions.Examples include refining an initially broad crop or correcting an incorrect bounding box.Source: §3.1 · p. 7

3.2 Training Strategy

  • Training masks earlier rounds in multi-round samples and excludes sandbox outputs from targets, while math data is trained separately after image data.These strategies address ineffective first-round outputs and the tendency for image-operation data to overwhelm the smaller math dataset; math fine-tuning uses a lower learning rate and learns all rounds jointly.Source: §3.2 · p. 8

4 Thyme-RL

Thyme-RL combines curated public prompts with manually constructed high-resolution visual questions, then trains through on-policy GRPO with adaptive sampling and rewards for answer quality and reasoning consistency.

4.1 RL Data Construction and Annotation

  • RL data combines manually verified prompts from V*, arXivQA, and ThinkLite-VL with manually annotated high-resolution images targeting difficult small-object questions.Because public datasets often lack resolution and perceptual complexity, the team collected 30,000 images with a dimension over 2048 pixels; 15 annotators wrote questions about objects occupying no more than 5% of the image.Source: §4.1 · p. 9

4.2 Preliminary of Reinforcement Learning Algorithm

  • The on-policy GRPO variant samples multi-round interaction trajectories and scores code generation, sandbox use, and final-answer correctness, while excluding sandbox observations from policy-token accounting.A KL penalty constrains policy deviation from a reference policy, and advantages are computed from the final rewards of trajectories in each group.Source: §4.2 · p. 10

4.3 GRPO with Adaptive Temperature Sampling

  • Code generation uses temperature 0 while natural-language reasoning uses 1.0, improving sample usability and helping prevent the model from avoiding code.The temperature switches when the <code> token appears; lower-temperature code generation helps recover from invalid-code rollouts and reduces code aversion. Figure 6 also notes sandbox outputs are excluded from training and advantage estimation.Source: §4.3 · p. 10, Figure 6 · p. 10
  • Training halts a sample when fixed-length substrings repeat beyond 50% of its output, and truncates dialogues exceeding the maximum turn limit to reduce wasted computation.The repetition detector uses Rabin-Karp rolling hashes.Source: §4.3 · p. 11

4.4 Reward Function Design:

  • The reward design combines strict answer-formatting and answer-correctness rewards, using rule-based matching first and Qwen-2.5-VL-72B for semantic comparison when needed.Outputs use <think> and <answer> tags; the auxiliary MLLM supports open-ended answer evaluation.Source: §4.4 · p. 11
  • A consistency reward evaluates whether the final answer follows logically from the reasoning (Team et al., 2025; Zhang et al., 2025d), but its bonus applies only when the answer is correct.This condition prevents consistency from outweighing answer accuracy.Source: §4.4 · p. 11

5 Experiments … 5.4 Ablation and Analysis

Thyme’s experiments report gains in perception, reasoning, and general tasks, then analyze how SFT strategies and RL reward design shape code use and reasoning consistency. The ablations find benefits from targeted SFT and consistency rewards, while RL reduces unnecessary code generation.

5.1 Experimental Setting

  • Training uses Qwen 2.5 VL 7B (Bai et al., 2025a), with image-manipulation SFT followed by math-code annealing, then one RL epoch.SFT runs for three epochs with batch size 128; RL uses learning rate 5 × 10^-7, batch size 256, four rollouts, and KL coefficient 0.001.Source: §5.1 · p. 11

5.2 Benchmarks and baselines

  • Evaluation covers perception, reasoning, and general benchmarks, using Qwen-2.5-VL-7B (Bai et al., 2025b) as the primary baseline and comparing against InternVL3-8B (Zhu et al., 2025), Qwen-2.5-VL-32B, and GPT-4o (OpenAI, 2024).The perception suite includes MME-RealWorld, HR Bench, and V*; reasoning includes MathVision and MathVista; general tasks include Hallucination bench, MMStar, and OCR Bench. VLMEvalKit is the evaluation pipeline.Source: §5.2 · pp. 11–12

5.4.1 The Impact of Training Strategies on the SFT Process

  • Masking sandbox labels and outputs improves performance, while training multi-round execution on only the final turn prevents learning suboptimal-code-then-correction patterns.Naive SFT yields no significant gains and chaotic output, including self-predicted sandbox contents and overwritten code or outputs.Source: §5.4.1 · p. 12, Table 5 · p. 14
  • Removing code comments produces unsatisfactory results, while math-data annealing teaches calculation-code behavior more effectively despite no significant gain over Only Last Round.The authors suggest writing comments supports code understanding and logic; annealing addresses the small amount of mathematical code data that is otherwise rarely observed during inference.Source: §5.4.1 · pp. 12, 14

5.3 Evaluation Results

  • Thyme outperforms larger models on perception, improves reasoning through executable-code computation, and gains on general tasks, particularly by reducing hallucination.The authors attribute perception gains to test-time scaling; they note model scaling benefits reasoning more, where inherent knowledge matters.Source: §5.3 · p. 12, Table 3 · p. 13
  • Over 25% improvement appears in both perception and reasoning on Monitoring and Autonomous Driving, while gains are limited on already-strong OCR, Diagram, and Table tasks.The baseline exceeds 60% and approaches 90% accuracy on OCR, Diagram, and Table.Source: §5.3 · p. 12, Table 4 · p. 14

5.4.2 The Influence of RL Reward Design on the RL Process

  • Consistency rewards improve performance, whereas process rewards hurt it and code rewards yield no positive gain; additional rewards apply only when the answer is correct.Process scoring adds overhead and is subjective and easy to hack; code rewards can encourage unnecessary code, such as simple arithmetic or uncomplicated graph cropping.Source: Table 6 · p. 14, §5.4.2 · p. 15

5.4.3 Analysis of the RL Learning Process

  • Result reward rises from approximately 0.5 to approximately 0.7, while consistency reward increases from 0.15 to nearly 0.35 during RL training.Average response length initially falls rapidly and then converges as the model learns that direct text is more efficient than unnecessary code on many tasks.Source: Figure 7 · p. 15, §5.4.3 · p. 15

5.5 Case studies · 5.6 Bad Cases

Case studies show Thyme using image operations and code for visual inspection and calculations, while failure cases expose missed or unnecessary coding and irrelevant cropping. The examples demonstrate both successful task completion and decision-making errors.

5.5.1 Cropping & Zooming

5.5.2 Rotation & Contrast Enhancement

  • Thyme corrects image orientation with Python code to identify an expression; the contrast enhancement case caption describes cropping and zooming to answer a phone-number question.Source: Figure 11 · p. 19, Figure 12 · p. 20

5.5.3 Complex Calculations

  • After deducing expressions for x and y, Thyme writes code to calculate their accurate results.Source: Figure 13 · p. 21

5.6.1 Complex Problems Without Coding

  • On a high-resolution image, Thyme fails to crop the relevant area and answers directly instead.Source: Figure 14 · p. 22

5.6.2 Unuseful Coding

  • For a trivial calculation, Thyme adds unnecessary coding overhead and confuses variables a and b, producing an incorrect result.Source: Figure 15 · p. 22

5.6.3 Inaccurate Cropping

  • Thyme eventually gives the correct answer despite cropping a region irrelevant to the problem during its reasoning.Source: Figure 16 · p. 23

6 Conclusion and Limitations

Thyme trains MLLMs to generate and execute code for image manipulation and computation, with reported gains across nearly 20 benchmarks. The conclusion notes limitations in base-model capability and benchmark coverage.

Conclusion

  • Across nearly 20 benchmarks, Thyme achieves significant and consistent gains over the baseline, particularly on high-resolution perception and complex reasoning tasks.It uses SFT on a curated 500K-sample dataset followed by RL with GRPO-ATS, which applies different sampling temperatures to text and code generation.Source: §6 · p. 24

Model Capability

  • Thyme’s performance is constrained by its base model’s limited object localization and code-generation capabilities, which can produce incorrect crops or unexecutable code.The authors suggest that stronger foundation models could improve the reliability and accuracy of tool use.Source: §6 · p. 24

Evaluation Scope

  • Existing benchmarks underrepresent rotated and low-contrast images, leaving advanced manipulation capabilities without robust numerical evaluation.The authors identify new benchmarks designed for advanced image-processing operations as a path toward more comprehensive evaluation.Source: §6 · p. 24

A Annotation Requirements

The annotation requirements aim to ensure each image accurately describes its target objects and provides users with clear annotation information.

A Annotation Requirements

  • The annotation document aims to ensure images accurately describe target-object characteristics and provide users with clear annotation information.It introduces detailed steps and requirements for the annotation process.Source: A Annotation Requirements · p. 28

A.1 Task Description

The annotation task builds high-resolution image QA pairs around small, hard-to-recognize targets and records their locations and question categories.

A.1 Task Description

  • Questions target difficult-to-recognize objects occupying no more than 5% of the image, which users must zoom in on to identify.Each question has an accurate answer derived from analyzing specific image regions.Source: §A.1 · p. 28
  • Annotators draw a bounding box for each target using (x1, y1, x2, y2) coordinates, with the crop clearly showing the object.Source: §A.1 · p. 28
  • Each question is assigned one category selected from the five categories defined in the question design.Source: §A.1 · p. 28

A.2 Annotation Requirements

Annotation targets are small objects that require zooming for clear identification, with approximate boxes that keep the object visible in the crop. Questions test multiple image-understanding capabilities, including chart interpretation.

A.2 Annotation Requirements

  • Select objects occupying no more than 5% of the image; they should require zooming to identify clearly.Source: §A.2 · p. 28
  • Bounding boxes need not be pixel-accurate, but the crop must clearly display the object and roughly match its image features.Source: §A.2 · p. 28
  • Questions should test varied visual skills, from recognizing text and object properties to interpreting chart data.Coverage includes OCR, attributes, locations, quantities, object identity, and chart values, extrema, or trends.Source: §A.2 · p. 28

A.3 Annotation Steps

The annotation process targets small, hard-to-recognize image details, pairing precise bounding boxes with concise, image-specific questions and answers. Each image’s annotations are saved independently with coordinates.

A.3 Annotation Steps

  • Annotators inspect high-resolution images, identify target objects occupying no more than 5% of image area, and draw boxes that fully contain them.Source: §A.3 · p. 28
  • For each target, annotators create concise questions and appropriate answers directly related to the image content.Source: §A.3 · p. 28
  • They save each image’s questions, answers, and bounding-box coordinates as independent, meaningful annotations.Source: §A.3 · p. 28

B Related Work

Related work traces rapid, increasingly versatile MLLM development, while identifying a gap in approaches that make visual inputs active in reasoning. Thyme addresses this gap by enabling autonomous code-based image manipulation and computation.

Multimodal Large Language Models

  • MLLM research spans high-resolution dialogue, longer contexts, efficiency, hallucination mitigation, conversation, alignment, and integrated multimodal input and output.The passage cites Qwen2.5-VL Bai et al. (2025a), GPT-4o OpenAI. (2024), and LLaVA Li et al. (2024a), among other work.Source: B Related Work · p. 29

Multimodal Reasoning

  • RL has advanced multimodal reasoning, but many approaches emphasize textual reasoning, treat images as static input, or support only cropping or auxiliary-image generation; Thyme enables autonomous code-based image manipulation and computation.The passage contrasts single-function “think with images” approaches—cropping Zheng et al. (2025) and auxiliary-image generation Chern et al. (2025)—with Thyme’s broader operations.Source: B Related Work · p. 29

Training data prompt

  • The training-data prompt asks the model to decide whether an image needs processing before answering, and to provide an answer without code when it does not.For an image that can be used directly, the prompt specifies including -1 in the answer box.Source: B Related Work · p. 30
  • When processing is needed, the prompt calls for identifying image issues, choosing direction, lighting/contrast, scaling/ROI, or combined operations, and generating executable Python code.Examples include rotation or flipping, brightness or contrast adjustment, and cropping or resizing.Source: B Related Work · p. 30

Related papers

Loading 2508.11630v1…