PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning

Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, Wenhu Chen

arXiv:2505.15966v3cs.CVcs.AIcs.CL

TL;DR

Pixel-space reasoning addresses the limitation of text-only intermediate reasoning for visually intensive VLM tasks by letting models apply operations directly to visual inputs. The paper combines warm-start instruction tuning with curiosity-driven reinforcement learning, achieving best known open-source performance across several visual reasoning benchmarks.

  • Problem

    Text-only intermediate reasoning limits VLM performance on visually intensive tasks because models cannot directly interact with information-rich visual inputs.

  • Method

    The approach combines instruction tuning on synthesized reasoning traces with curiosity-driven reinforcement learning to cultivate visual operations and balance pixel-space and textual reasoning.

  • Results

    Pixel-Reasoner achieves best known open-source performance across several visual reasoning benchmarks and exceeds proprietary models such as Gemini-2.5-Pro and GPT-4o.

  • Takeaways & Limitations

    Pixel-space reasoning, supported by warm-start instruction tuning and curiosity-driven RL, enables VLMs to inspect and infer from visual evidence with enhanced fidelity.

  • Takeaways & Limitations

    The framework is currently limited to two primary visual operations, which is insufficient for broader tasks.

Abstract

from arXiv · show

Chain-of-thought reasoning has significantly improved the performance of Large Language Models (LLMs) across various domains. However, this reasoning process has been confined exclusively to textual space, limiting its effectiveness in visually intensive tasks. To address this limitation, we introduce the concept of reasoning in the pixel-space. Within this novel framework, Vision-Language Models (VLMs) are equipped with a suite of visual reasoning operations, such as zoom-in and select-frame. These operations enable VLMs to directly inspect, interrogate, and infer from visual evidences, thereby enhancing reasoning fidelity for visual tasks. Cultivating such pixel-space reasoning capabilities in VLMs presents notable challenges, including the model's initially imbalanced competence and its reluctance to adopt the newly introduced pixel-space operations. We address these challenges through a two-phase training approach. The first phase employs instruction tuning on synthesized reasoning traces to familiarize the model with the novel visual operations. Following this, a reinforcement learning (RL) phase leverages a curiosity-driven reward scheme to balance exploration between pixel-space reasoning and textual reasoning. With these visual operations, VLMs can interact with complex visual inputs, such as information-rich images or videos to proactively gather necessary information. We demonstrate that this approach significantly improves VLM performance across diverse visual reasoning benchmarks. Our 7B model, \model, achieves 84\% on V* bench, 74\% on TallyQA-Complex, and 84\% on InfographicsVQA, marking the highest accuracy achieved by any open-source model to date. These results highlight the importance of pixel-space reasoning and the effectiveness of our framework.

1 Introduction

Pixel Reasoner extends VLM reasoning beyond text by incorporating visual operations into iterative reasoning. It uses warm-start instruction tuning and curiosity-driven RL to cultivate this capability and reports strong performance across visual reasoning benchmarks.

  • Motivation and paradigm: Text-only intermediate reasoning can limit VLM performance on visually intensive tasks requiring fine-grained details such as tiny objects, spatial relationships, embedded text, and video actions.Existing VLMs commonly conduct chain-of-thought reasoning purely in textual format.
  • Motivation and paradigm: Pixel-space reasoning incorporates operations applied directly to visual inputs as integral steps in a model’s reasoning chain.The framework enables models to inspect, interrogate, and infer from visual evidence while progressively refining their understanding.
  • Training challenges and approach: The approach combines warm-start instruction tuning with curiosity-driven reinforcement learning to address limited visual-operation proficiency and the learning trap.Instruction tuning synthesizes 7,500 reasoning traces, while RL uses a curiosity-driven reward scheme to balance pixel-space and textual reasoning.
  • Results: Pixel Reasoner achieves best-known open-source performance across information-rich image and video benchmarks, including V*, TallyQA, MVBench, and InfographicsVQA.The reported evaluations also compare Pixel Reasoner with proprietary models such as Gemini-2.5-Pro and GPT-4o.

2 Problem Formulation

Pixel-space reasoning models alternate between textual thinking and visual operations while iteratively constructing a solution. The formulation focuses on image and video inputs with ZOOM-IN and SELECT-FRAME operations.

  • Iterative reasoning formulation: A VLM constructs a solution iteratively, generating each reasoning segment conditioned on the query and all preceding reasoning steps.The query combines visual inputs and a textual query, and reasoning terminates when an end token is generated.
  • Reasoning-step types: Textual thinking performs reasoning within the textual domain, such as calculating equations or using domain knowledge.These steps do not directly manipulate the visual inputs.
  • Reasoning-step types: Visual operations directly manipulate or extract information from visual inputs by invoking predefined functions and producing execution outcomes.The formulation distinguishes these operations from purely textual reasoning steps.
  • Supported inputs and operations: The framework considers images and videos, using ZOOM-IN to inspect specified image regions and SELECT-FRAME to analyze specific video frames.Detailed operation protocols are provided in the appendix.

3 Warm-Start Instruction Tuning

Warm-start instruction tuning addresses VLMs’ limited zero-shot ability to execute visual operations by training on synthesized pixel-space reasoning trajectories. The data combines controlled visual-operation use with self-correction and ordinary textual reasoning where fine-grained visual analysis is unnecessary.

  • Collect Seed Datasets: The pipeline curates high-complexity image, webpage, and video data to support fine-grained visual reasoning trajectories.It uses SA1B, FineWeb, and STARQA, whose contents span segmentation-rich natural scenes, diverse webpages, and situated video reasoning.
  • Localize Reference Visual Cues: Queries are selected or synthesized to require locating specific fine-grained visual cues within visual inputs.FineWeb and STARQA provide reference cues, while SA1B queries are generated from GPT-4o-identified target details.
  • Synthesize Expert Trajectories: Direct GPT-4o distillation can produce bypassing trajectories that ignore problematic visual operations and rely on textual reasoning.The authors therefore use template-based synthesis to control the trajectory structure.
  • Synthesize Expert Trajectories: The template sequences whole-input analysis, visual-operation execution, detailed-cue analysis, and final answering.This structure uses each query’s reference visual cue to guide the synthesized trajectory.
  • Synthesize Expert Trajectories: Error-induced self-correction trajectories deliberately use incorrect visual cues so the policy learns to react to unexpected inputs or execution errors.Examples include irrelevant video frames and overly large image regions.
  • Warm-Start Instruction Tuning: Warm-start training mixes single-pass pixel-space trajectories, error-induced self-correction trajectories, and textual reasoning trajectories for queries not requiring fine-grained visual analysis.Loss masks exclude visual-operation outputs and designated erroneous operations, preventing the policy from learning incorrect actions.

4 Curiosity-Driven Reinforcement Learning

The RL phase addresses the warm-start model’s imbalance between strong textual reasoning and nascent pixel-space reasoning with curiosity incentives and operation-efficiency constraints. Its reward increases exploration when pixel-space reasoning is rarely attempted, while limiting excessive visual operations.

  • Motivation: The warm-start model’s learning trap arises from weak visual-operation proficiency and queries that can be answered without visual processing.Early visual-operation failures attract more negative feedback, while bypassing remains possible on queries not requiring visual operations.
  • Curiosity-Driven Reward: The curiosity-driven reward intrinsically rewards attempts to use pixel-space operations, countering discouragement from early failures and negative feedback.It is designed to sustain exploration rather than defaulting immediately to established textual reasoning.
  • Constrained Optimization: The objective maximizes expected correctness while constraining pixel-space reasoning usage and the number of visual operations.The formulation treats pixel-space usage as an explicit cultivation target rather than optimizing correctness alone.
  • Constrained Optimization: The RaPR constraint requires the average rate of pixel-space reasoning across rollouts for query x to meet threshold H.This encourages consistent attempts on a substantial proportion of queries.
  • Constrained Optimization: An upper bound N limits visual operations in each response, encouraging exploration while preserving computational efficiency.The constraint prevents overly complex or protracted visual processing.
  • Reward Design: Lagrangian relaxation combines correctness, curiosity, and efficiency-penalty terms into a single modified reward function.The curiosity bonus is strongest when RaPR is low and diminishes as pixel-space reasoning becomes mastered, reducing reward hacking risk.

5 Experiments

Experiments evaluate Pixel-Reasoner across four multimodal benchmarks and analyze how warm-start instruction tuning and curiosity-driven RL cultivate pixel-space reasoning. The results show strong benchmark performance and reveal that both visual-operation proficiency and exploration incentives are important.

  • Evaluation Settings: Pixel-Reasoner is evaluated on TallyQA, V*, InfographicVQA, and MVBench, spanning fine-grained recognition and reasoning over static and dynamic visual inputs.V* emphasizes high-resolution visual details, while InfographicVQA requires parsing layouts, embedded text, and semantic links.
  • Main Results: 84.3 vs 79.2 on V-star Bench gives Pixel-Reasoner a 5.1 percentage point lead over Gemini-2.5-Pro.The paper reports this as evidence that the model exceeds leading proprietary performance on that benchmark.
  • Main Results: RL training raises the model from below-baseline warm-start performance to state-of-the-art levels by enabling trial-and-error exploration of pixel-space operations.The authors contrast RL with static supervised demonstrations, which do not provide the same active exploration.
  • Ablation Findings: 2.5 points on average separate the full model from RL w/o Curiosity and RL w/o Warm-Start baselines.Those ablations tend to default to text-space reasoning because of insufficient incentives or limited visual-operation proficiency.
  • Learning Trap: Zero-shot training causes RaPR to decline from approximately 20% to zero, while single-pass tuning increases operation attempts but leaves a persistently high failure rate.The dynamics associate low initial proficiency with diminishing exploration and missing self-correction with operational failures.
  • Learning Trap: Without curiosity, RaPR falls from around 0.55 to 0 in 240 gradient steps; with curiosity, it later increases while operation failures remain low and stable.The curiosity bonus sustains exploration for roughly 200 steps before the policy begins leveraging pixel-space reasoning.

6 Related Work

Related work covers post-training methods for VLMs, multimodal reinforcement learning, and tool-enabled visual reasoning. Existing approaches include instruction tuning, RL refinement, selective replay, and direct pixel-level operations.

  • Post-Training for Vision-Language Models: Post-training methods use instruction tuning and reinforcement learning to adapt VLMs to complex tasks beyond pre-training.Prior work reports benefits from scaling instruction-tuning data and increasing task diversity.
  • Post-Training for Vision-Language Models: Multimodal RL approaches commonly combine supervised fine-tuning on distilled data with RL to refine reasoning capabilities.VL-Rethinker explores more direct RL for slow-thinking and introduces selective sample replay to address vanishing advantages.
  • Vision-Language Models with Tools: Tool-oriented VLM research trains models to use external tools or distills tool-based reasoning, including depth perception, plotting, zooming, and flipping.These methods expand visual interaction beyond purely textual reasoning.

7 Conclusion

The conclusion presents pixel-space reasoning as a new way to incentivize visual interaction in existing VLMs. It identifies warm-start instruction tuning and curiosity-driven RL as essential components while noting that the current framework remains limited in operation coverage.

  • Conclusion: Pixel-space reasoning is introduced as a paradigm for incentivizing visual interaction in an existing vision-language model.The paper frames this as its first demonstration of incentivizing pixel-space reasoning from an existing VLM.
  • Conclusion: Warm-start instruction tuning and curiosity-driven RL are both essential to achieving state-of-the-art performance.The framework combines operation preparation with intrinsic incentives for exploration.
  • Conclusion: The framework is currently limited to two primary visual operations, restricting its coverage for broader tasks.The paper proposes extending the operation library with capabilities such as depth maps and image search.

A Limitations

The approach remains constrained by limited cross-task and cross-content data and by its focus on two visual operations for image and video inputs.

  • Limitations: Limited data spanning tasks and contents remains a bottleneck for cultivating pixel-space reasoning.The paper identifies broader task and content coverage as a future direction.
  • Limitations: The study focuses on two visual operations for handling primary image and video formats.The authors plan to add more operations and evaluate the paradigm on more diverse task collections.

B Derivations of Curiosity-Driven Reward

The section formulates pixel-space reasoning as a constrained reward-maximization problem and derives a clipped reward that addresses over-satisfaction and query-level incentive issues.

  • Constrained Optimization: The objective maximizes expected correctness reward while constraining the average pixel-space reasoning rate and per-response visual-operation count.The constraints require RaPR(x) ≥ H and nvo(y) ≤ N.
  • Constrained Optimization: Standard Lagrangian relaxation penalizes violations of the minimum reasoning-rate and maximum-operation constraints.The formulation maximizes the expected Lagrangian reward with respect to the policy and minimizes it with respect to the multipliers.
  • Problems with Standard Relaxation: Direct Lagrangian rewards can encourage too few visual operations and fail to properly reward individual responses because one constraint acts only at the expectation level.These are the stated over-satisfaction and response-credit-assignment problems.
  • Modified Reward: The modified reward adds a clipped pixel-reasoning incentive and an operation-count penalty to the original correctness reward.The pixel-reasoning bonus activates when RaPR(x) falls below H, while the operation term uses the bound N.
  • Modified Reward: Clipping makes penalties zero when constraints are satisfied, preserving the constrained objective while allowing α and β to remain fixed hyperparameters.The indicator 1PR(y) converts the query-level expectation constraint into a response-level incentive.

C Data and Training Details

The data pipeline targets visually complex, annotated image and video sources, then synthesizes trajectories that combine global analysis, localized visual operations, and self-correction.

  • Visual Operations: The method supports two primary visual operations: cropping image regions and selecting video frames.CropImage focuses on fine-grained spatial details, while SelectFrames focuses on relevant temporal moments.
  • Seed Datasets: The seed datasets are selected for high visual complexity and explicit annotations that can guide visual operations.Sources include SA1B, FineWeb, and STARQA.
  • Trajectory Generation: GPT-based synthesis combines whole-image or video analysis with localized reference cues to form single-pass trajectories.The pipeline concatenates global analysis, the localized cue, and partial local analysis.
  • Trajectory Generation: Self-correction trajectories deliberately insert incorrect visual cues before the correct operation to train recovery from errors.The variants include irrelevant crops, repeated recrops, and excessively large zoom regions.
  • Trajectory Composition: Table 2 reports the proportions of the paper’s self-correction trajectory types.The supplied caption identifies the table’s subject but does not provide the proportions themselves.

C.3 Training Details

Training combines supervised warm-starting with reinforcement learning and selective replay, while experiments track both reward dynamics and pixel-space reasoning usage across benchmarks.

  • RL Procedure: The RL implementation uses GRPO with selective sample replay because the authors observed vanishing advantages.The behavior policy synchronizes with the improvement policy every 512 queries, and each query produces 8 responses.
  • RL Procedure: The reward combines correctness, curiosity, and efficiency terms, but reward uniformity without selective replay rises steadily to 90%.The authors associate this rise with a drastic performance plunge and increasing response-all-incorrect queries.
  • Hyperparameters: The training configuration uses α = 0.5, β = 0.05, H = 0.3, and N = 1, with 8 trajectories sampled per training query.The threshold encourages pixel-space reasoning, while additional visual operations beyond the first incur a penalty of −0.05.
  • Initial Capability: Qwen2.5-VL-Instruct invokes visual operations in 20.2% of zero-shot training rollouts, with 23.2% accuracy using pixel-space reasoning versus 49.5% using textual reasoning.Among operation-using rollouts, 40.6% incur an error and 36.2% produce an incorrect answer.
  • Benchmark Statistics: Pixel-Reasoner’s pixel-space reasoning rates are 78.53% on V-Star, 57.78% on TallyQA-Complex, 58.95% on InfographicsVQA, and 66.95% on MVBench.These rates show adaptive triggering across the four evaluation benchmarks.

D.2 Case Analysis

The case analyses illustrate successful visual inspection alongside two failure modes: hallucinated operation success and continuation of textual reasoning after execution errors.

  • Successful Cases: Successful examples show targeted infographic-region analysis through crop refinement or direct answering.The figure caption describes identifying the relevant region and then refining the crop or answering the query.
  • Failure Modes: The hallucination failure mode occurs when the policy behaves as though visual operations executed successfully.A separate example records an execution error during frame selection.
  • Failure Modes: The no-reaction failure mode occurs when the policy ignores an execution error and proceeds with textual reasoning.The example again reports the error “max() arg is an empty sequence.”
  • Reasoning Protocol: The training prompt asks the model to decide whether visual operations are beneficial, use select_frames for videos or crop_image for images, and answer step by step.The final answer is constrained to appear inside boxed notation.
  • Image Trajectories: Image trajectories first analyze the whole image, then zoom into a queried region before answering from the crop.The examples apply this pattern to leadership text, SEC championship results, and a rental price sign.
  • Video Trajectories: Video trajectories first analyze the full sequence, then select frames to clarify the queried action or object.The supplied examples focus on why measuring spoons were removed and which object was placed on a counter.

2. Limitations

The paper reports limitations-related and reproducibility information through its review responses, including an absence of error bars because experiments were not repeated. It also states that limitations are discussed in the conclusion and appendix.

  • Limitations discussion: The authors state that limitations are discussed in the conclusion and covered in an appendix section.
  • Reproducibility: The paper states that code, data, and models will be made public.
  • Experimental details: Training and test details are reported in the experiment section and appendix, according to the authors’ reproducibility justification.
  • Experimental significance: No error bars are reported because the experiments were not run multiple times, although fixed seeds and consistent settings were used.This limits direct assessment of variability across runs.
  • Asset documentation: The authors state that new code and data assets include usage instructions, schema descriptions, and license information.
Loading 2505.15966v3…