PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models

Yu Zeng, Wenxuan Huang, Shiting Huang, Xikun Bao, Yukun Qi, Yiming Zhao, Qiuchen Wang, Lin Chen, Zehui Chen, Huaian Chen, Wanli Ouyang, Feng Zhao

arXiv:2510.01304v3cs.AIcs.CL

TL;DR

VLMs remain weak on simple jigsaw tasks requiring perception and reasoning, while scalable, high-quality vision-language RL data is scarce. AGILE makes jigsaw solving interactive and feedback-guided; it reports improved performance on jigsaws and broader vision tasks, including accuracy rising from 9.5% to 82.8% in the 2 × 2 setting.

  • Problem

    Current VLMs can perform near randomly on simple 2 × 2 jigsaws, while high-quality vision-language RL data is scarce and difficult to scale.

    Source: §1 · pp. 1–2

  • Method

    AGILE trains VLMs through interactive jigsaw solving, with generated actions and environmental feedback, using scalable code- and rule-based data generation.

    Source: §1 · p. 2

  • Results

    Accuracy rises from 9.5% to 82.8% on the 2 × 2 setting, and the approach generalizes across diverse vision benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Jigsaw tasks can serve as a proxy for enhancing VLM perception and reasoning while providing scalable multimodal RL training data.

    Source: §1 · pp. 2–3

    Multi-turn interactions add context length and computational overhead; in the 3 × 3 setting, context often exceeds the model window, restricting RL training to 2 × 2.

    Source: F LIMITATIONS · p. 24

Abstract

from arXiv · show

Although current large Vision-Language Models (VLMs) have advanced in multimodal understanding and reasoning, their fundamental perceptual and reasoning abilities remain limited. Specifically, even on simple jigsaw tasks, existing VLMs perform near randomly, revealing deficiencies in core perception and reasoning capabilities. While high-quality vision-language data can enhance these capabilities, its scarcity and limited scalability impose significant constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction Learning for Enhancing visual perception and reasoning in VLMs. AGILE formulates jigsaw solving as an interactive process, enabling the model to progressively engage with the environment. At each step, the model generates executable code to perform an action based on the current state, while the environment provides fine-grained visual feedback to guide task completion. Through this iterative cycle of observation and interaction, the model incrementally improves its perceptual and reasoning capabilities via exploration and feedback. Experimental results show that AGILE not only substantially boosts performance on jigsaw tasks of varying complexity (e.g., increasing accuracy from 9.5% to 82.8% under the 2 $\times$ 2 setting) but also demonstrates strong generalization across 9 general vision tasks, achieving an average improvement of 3.1%. These results indicate notable enhancements in both perceptual and reasoning abilities. This work opens a new avenue for advancing reasoning and generalization in multimodal models and provides an efficient, scalable solution to the scarcity of multimodal reinforcement learning data. The code and datasets is available at https://github.com/yuzeng0-0/AGILE .

1 INTRODUCTION

Current VLMs remain weak on simple jigsaw tasks that require perception and logical inference, while scalable, high-quality vision-language RL data is scarce. AGILE trains VLMs through interactive jigsaw solving and scalable data generation, improving jigsaw performance and generalizing to diverse vision benchmarks.

1 INTRODUCTION

  • Existing VLMs perform near randomly on simple 2 × 2 jigsaws requiring perception and logical inference, and pretraining and fine-tuning have not developed robust abilities.The paper frames the jigsaw task as a test of core perceptual and reasoning capabilities.Source: §1 · pp. 1–2
  • High-quality vision-language RL data remains scarce and difficult to scale: human supervision is costly or small-scale, while automated synthesis can have quality, capability, and API-cost constraints.The paper identifies these challenges as hindering VLM reasoning and generalization.Source: §1 · p. 2
  • Code- and rule-based generation provides controllable task difficulty and scalable, strictly supervised synthetic data with available ground truth.Difficulty can be adjusted through the initial placement of correct tiles and the overall jigsaw size; the data can be scaled to arbitrary sizes.Source: §1 · pp. 2–3
  • Accuracy rises from 9.5% to 82.8% on the 2 × 2 setting, with improvements across jigsaw complexities and diverse vision benchmarks.Reported generalization spans high-resolution image understanding, real-world scene analysis, fine-grained recognition, visual reasoning, and hallucination benchmarks.Source: §1 · pp. 2–3
  • AGILE formulates jigsaw solving as a stepwise interaction in which VLMs generate actions and receive feedback to refine perception and reasoning.The action space includes swapping tiles, observing the state, cropping regions, and zooming for fine-grained analysis.Source: §1 · pp. 2–3

2 RELATED WORK

Related work applies RL to VLM perception and reasoning, but scalable, high-quality RL data remains a bottleneck. Proxy-task research shows transfer potential, while prior jigsaw RL work still struggled on simple puzzles; this paper proposes interactive jigsaw solving with stepwise feedback.

2 RELATED WORK

  • RL has been applied to VLM perception and reasoning, including object counting, grounding, visual-tool use, and mathematical tasks.These works use approaches such as object matching and IoU rewards, or rule-based RL.Source: §2 · p. 3
  • The literature identifies a lack of scalable, high-quality RL datasets as a fundamental bottleneck to improving VLM perception and reasoning.Prior dataset construction includes converting visual information into textual representations for multimodal chain-of-thought data.Source: §2 · p. 3
  • Proxy-task puzzles have improved general reasoning in language models, but Jigsaw-R1 still performed poorly on 2 × 2 puzzles; AGILE instead uses interactive solving with explicit stepwise feedback.The proposed interaction is intended to help models capture visual relationships and refine perception and reasoning.Source: §2 · p. 3

3 METHOD

AGILE frames jigsaw solving as an interactive task and organizes the method around environment interaction, trajectory construction, and cold-start plus reinforcement-learning training.

3 METHOD

  • AGILE combines model–environment interaction, high-quality trajectory collection, and GRPO training in a single framework.Figure 2 depicts the interaction and GRPO process, trajectory-data collection, and jigsaw rollout.Source: §3 · p. 3, Figure 2 · p. 4

3.1 JIGSAW TASK AND ENVIRONMENT INTERACTION

  • The model iteratively swaps pieces, observes the resulting configuration, and uses Python actions—including crop and zoom—to inspect the puzzle until completion.The environment executes generated code and returns the resulting image for further reasoning.Source: §3.1 · pp. 4–5

3.2 DATA CONSTRUCTION

  • Gemini 2.5 Pro supplies expert trajectories for cold-start training because weak instruction-following and code generation make direct RL inefficient.The trajectories are filtered against ground truth, manually checked, balanced across 4–8 steps and action types, and curated into a 1.6K-example dataset.Source: §3.2 · p. 5

3.3 TRAINING PARADIGM: COLD-START AND REINFORCEMENT LEARNING

  • Cold-start training uses the curated trajectories to teach instruction-following and code generation, followed by GRPO using grouped rollouts and a relative-reward baseline.GRPO uses the average reward of sampled outputs as a baseline instead of a learned value function.Source: §3.3 · pp. 5–6
  • The reward combines accuracy, output format, and step efficiency; the step reward applies only to correctly completed puzzles and gives incorrect solutions the maximum step penalty.The accuracy and format rewards are binary; the reward coefficients α, β, and γ are 0.8, 0.2, and 1.0, respectively.Source: §3.3 · p. 6

4 EXPERIMENT

Experiments evaluate AGILE on jigsaw solving and nine general-vision benchmarks, then analyze data scale, QA-data comparisons, and qualitative behaviors. Results show large jigsaw gains, broader benchmark improvements, and complementary perceptual strategies.

4.1 IMPLEMENTATION DETAILS

  • Experiments use 1.6K cold-start trajectories and 15.6K RL-stage images with Qwen2.5-VL-7B, and evaluate general benchmarks in a strict single-turn setting.Training uses full-parameter tuning; experiments run on eight NVIDIA A100 GPUs with 80GB memory each.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • Acc rises from 9.5% to 82.8% and Score from 29.4% to 89.0% on 2 × 2 jigsaws; on 3 × 3, Acc rises from 0.4% to 20.8% and Score from 31.1% to 62.1%.The results follow supervised fine-tuning with 1.6K trajectories and reinforcement learning with 15.6K images; the 3 × 3 results surpass Gemini2.5-Pro and Qwen2.5-VL-72B.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • The jigsaw-trained model surpasses the base model Qwen2.5-VL-7B by 3.1% on average across all nine benchmarks, with gains of 4.2% on HRBench4K, 5.2% on HRBench8K, and 4.2% on VStarBench.Source: §4.2 · p. 7

4.3 ANALYSIS

  • Scaling training data raises jigsaw accuracy from 22.0% to 82.8%, HRBench4K by 2.0%, and RealWorldQA by 1.8%.The training data are programmatically synthesized, allowing the dataset to scale.Source: Figure 3 · p. 8, §4.3 · p. 9
  • With 20K samples in each setup, 10K jigsaw samples plus 10K QA samples outperform 20K QA samples alone on general-vision benchmarks.The authors describe programmatic jigsaw synthesis as less costly and labor-intensive than constructing QA data.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Case studies show the model infers spatial relations, uses crops and zooms to validate edge alignment, and reasons about semantic consistency across pieces.The examples are presented as perceptual and reasoning strategies observed during jigsaw solving.Source: Figure 5 · p. 9, §4.3 · p. 9
  • After AGILE training, attention is more concentrated on small objects, text, and structurally important regions than in the original model.The authors report this qualitative pattern for Qwen2.5-VL-7B and align it with gains on HRBench, VStarBench, and RealWorldQA.Source: Figure 6 · p. 10, §4.3 · p. 10

5 CONCLUSION

The conclusion presents AGILE as interactive jigsaw training for VLM perception and reasoning, with gains on jigsaw and broader visual tasks. It identifies jigsaw as a proxy for addressing high-quality RL-data scarcity.

5 CONCLUSION

  • AGILE frames jigsaw solving as interactive training and reports improved performance on jigsaw tasks and general visual tasks, including visual question answering and scene understanding.The conclusion connects jigsaw data scale and comparisons with general QA data to its proposed role as a proxy for scarce high-quality RL data.Source: §5 · p. 11

6 ETHICS STATEMENT

The ethics statement reports use of public research datasets under their respective licenses and no use of private or personally identifiable information.

6 ETHICS STATEMENT

  • The study uses publicly available research datasets under their respective licenses and reports that no private or personally identifiable information was used.Source: §6 · p. 11

7 REPRODICIBILITY STATEMENT

The reproducibility statement says implementation details are provided in appendices and commits to releasing code, datasets, and evaluation scripts upon publication.

7 REPRODICIBILITY STATEMENT

  • Training data and hyperparameter settings are provided in Appendices A and D, and the authors state that code, datasets, and evaluation scripts will be released upon publication.Source: §7 · p. 11

A DATA DISTRIBUTION

The appendix describes a jigsaw training corpus spanning several visual domains, a prompt that guides stepwise puzzle solving, and representative trajectories showing interaction with environment feedback.

A DATA DISTRIBUTION

  • The corpus combines high-resolution and visual-search images (39.7%, 6.2K samples), text-recognition and structured-scene images (33.3%, 5.2K samples), and dense natural and real-world scenes (26.9%, 4.2K samples).The three groups draw on datasets including VStar, TextVQA, COCO2017, and RealWorldQA.Source: A DATA DISTRIBUTION · p. 16

B PROMPTS

  • The prompt has the model analyze patch continuity and structure, optionally crop or zoom for detail, then generate code for a swap or observation and continue from the updated state.Only one image operation or state observation is allowed per turn.Source: B PROMPTS · pp. 17–18

C CASE STUDY

  • Representative trajectories show the model swapping pieces, observing progress, and cropping or zooming regions, with environment feedback guiding subsequent reasoning.Figures 8–11 illustrate these interactions.Source: C CASE STUDY · p. 19, Figure 8 · p. 19, Figure 9 · p. 20

D THE IMPLEMENTATION OF TRAINING

The implementation section identifies the training hyperparameter tables and the hardware used for experiments.

D THE IMPLEMENTATION OF TRAINING

E ANALYSIS OF WANDB CURVES IN JIGSAW OPTIMIZATION

The training curves are described as showing an initial exploration phase followed by improved jigsaw-solving ability with fewer interactions; the authors also identify context length as a constraint on larger-grid RL training.

E ANALYSIS OF WANDB CURVES IN JIGSAW OPTIMIZATION

  • As training progresses, interaction turns briefly increase as the model explores, then decline alongside response length as rewards and validation accuracy improve.The authors interpret the later reductions as solving puzzles correctly with fewer interactions.Source: E ANALYSIS OF WANDB CURVES IN JIGSAW… · p. 23, Figure 12 · p. 23

F LIMITATIONS

  • Context length often exceeds the model window in 3 × 3 puzzles, restricting RL training to the 2 × 2 setting.The authors suggest more efficient environment interaction or memory modules as possible routes to larger tasks.Source: F LIMITATIONS · p. 24

G WHICH TASKS CAN BENEFIT MORE FROM AGILE?

The section describes skill areas reported to benefit from AGILE, discusses invalid-code handling and reward sensitivity, and reports further benchmark improvements from curriculum training on larger jigsaws.

G WHICH TASKS CAN BENEFIT MORE FROM AGILE?

  • AGILE is reported to improve fine-grained recognition, spatial-relation reasoning, and visual–textual integration, including real-world spatial reasoning tasks.The section links these gains to attention to subtle visual cues, scene layouts, and textual cues grounded in visual context.Source: G WHICH TASKS CAN BENEFIT MORE FROM… · p. 24

H HANDLING INVALID CODE EXECUTIONS

  • Invalid code is executed and returns explicit errors rather than being discarded, increasing steps and lowering step rewards to implicitly penalize invalid actions.The authors say this stabilizes training without complex filtering mechanisms.Source: H HANDLING INVALID CODE EXECUTIONS · p. 24

I.1 SENSITIVITY TO REWARD COEFFICIENTS.

  • Changing α and β causes only minor performance fluctuations, while removing the step reward γ notably degrades performance, especially on VStarBench and MMVP.The authors attribute a critical role to the step penalty in encouraging efficient interactions and preventing long, unfocused rollouts.Source: §I.1 · p. 24, Table 6 · p. 25

I.2 RESULTS OF LARGER JIGSAW GRID SIZES AND CURRICULUM RL TRAINING

  • Training on 8K 3 × 3 puzzles after the initial 15.6K 2 × 2 RL stage further improves results across all 9 benchmarks.The authors present curriculum RL on increasingly difficult grid sizes as a path for continued scaling.Source: §I.2 · pp. 24–25, Table 7 · p. 25

I.3 EXPERT TRAJECTORIES ABLATIONS

Jigsaw-based RL outperforms high-resolution QA-based RL under equal training budgets, and its gains persist without expert trajectories. Moderate SFT expansion has only marginal effects, while interactive RL is identified as the primary source of improvement.

I.3 EXPERT TRAJECTORIES ABLATIONS

  • Without expert trajectories, AGILE improves 1.8% over the Qwen2.5-VL-7B baseline, while removing Crop/Zoom causes a clear performance decline.The experiments use the 15.6K 2 × 2 jigsaw dataset; the authors attribute gains to agentic visual interaction with verifiable RL rather than imitation of Gemini 2.5 Pro trajectories.Source: §I.3 · p. 25, Table 8 · p. 25

I.4 GENERAL QA DATASET DETAILS AND EXTENDED COMPARISON WITH JIGSAW TRAINING

  • The 20K general QA corpus comprises 15K DeepEyes and 5K MMEureka samples from high-resolution real-world scenes.The samples cover fine-grained attributes, spatial relationships, counting, and commonsense scene understanding.Source: §I.4 · p. 25
  • Under equal 20K training budgets, jigsaw-based RL consistently outperforms high-resolution QA-based RL.The comparison is described as fair because both regimes use high-resolution, real-world, complex perceptual inputs.Source: §I.4 · p. 25, Table 9 · p. 26

I.5 IMPACT OF SCALING THE COLD-START (SFT) DATASET

  • Scaling the SFT dataset to 1.5× and 2× its original size yields only marginal differences, with no signs of overfitting or meaningful degradation.The cold-start stage teaches tool-use patterns, whereas the interactive RL stage is identified as the primary source of generalization and performance improvement.Source: §I.5 · p. 26, Table 10 · p. 26

J USE OF LLMS

  • LLMs were used only for language polishing and readability; the authors conducted the technical work, experiments, and analyses.Source: J USE OF LLMS · p. 26

Related papers

Loading 2510.01304v3…