G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning

1 Introduction

• VLMs’ strong performance on direct multimodal tasks does not reliably transfer to interactive decision-making, while scalable RL training for games with verifiable rewards remains lacking.Even leading models can score below random choice on straightforward games such as Shisen-Sho and 2048.Source: §1 · p. 2
• G0 models trained through pure RL self-evolution develop perception and reasoning patterns, raising Shisen-Sho score from 1.9 to 12.8 and surpassing OpenAI-o1 and Qwen2.5VL-72B.The models nevertheless face perception-prior, reward-credit-assignment, and sparse-reward challenges amid game diversity.Source: §1 · p. 2
• VLM-Gym provides multiple games, unified observation and action spaces, adjustable compositional difficulty, and support for scalable parallel training.The games include 2048, Shisen-Sho, Shisen-Sho-Cifar10, and Swap.Source: §1 · p. 2
• G1 models outperform their teacher across all games and significantly surpass Claude-3.7-Sonnet-Thinking; analysis finds perception and reasoning bootstrap each other during RL.G1 adds a perception-enhanced cold start and teacher knowledge distillation before RL training.Source: §1 · p. 2

2 VLM-Gym: A Scalable Interactive Environment for VLMs

• VLM-Gym supports parallel execution across game states and games, same-observation action sampling with reward evaluation, and adjustable, combinable difficulty dimensions.These features support large-batch training, advanced RL algorithms, and studies of generalization.Source: §2 · pp. 2–3

2.1 Game Environment Description

• In 2048, reward is tied to tile merging; in Shisen-Sho, reward is tied to matching tiles.2048 is a 4×4 sliding puzzle, while Shisen-Sho requires matching identical tiles with a path of no more than three straight lines.Source: §2.1 · p. 3
• Shisen-Sho-Cifar10 replaces traditional symbols or patterns with CIFAR10 images as more visually complex tiles for exploring perception improvement.It uses the same reward function as Shisen-Sho.Source: §2.1 · p. 3
• Swap is a tile-matching game in which adjacent tiles are swapped to form matches of three or more, and reward is tied to matched tiles disappearing.After matches disappear, new tiles fall to fill the empty spaces.Source: §2.1 · p. 3

2.2 Game Prompt and Action Format

• The prompt asks the model to describe the board in a perception block, provide reasoning in a think block, and give its final action in an answer block.Actions use a direction choice in 2048 and coordinate pairs for tile selection or swapping in the other described games.Source: §2.2 · p. 4

3.1 RL Training Objective

• Training combines game, format, and perception rewards and uses GRPO to optimize groups of outputs generated from game screenshots and prompts.Perception and format rewards are auxiliary 0–1 signals; the final reward is GR + α · FR + β · PR.Source: §3.1 · p. 4
• Random-action baselines expose the policy to diverse game states, with predetermined step counts varying by game.The authors note that more sophisticated search or a stronger model could further enhance exploration.Source: §3.1 · p. 5

3.2 Knowledge Distillation with Ground-truth Perception

• G1 receives supervised fine-tuning on Claude-3.7-Sonnet-Thinking responses concatenated with VLM-Gym ground-truth perceptions before RL.Per game, SFT used 1,000 observations and 1,000 distilled responses; Qwen2.5-VL-7B was the base model.Source: §3.2 · p. 5, §3.3 · p. 5

3.4 Evaluations

• Evaluation benchmarks models in multi-turn play using original game scoring and average cumulative scores across independent runs.The comparison includes Qwen2.5-VL-72B, OpenAI-o1, GPT4o, and Claude-3.7-Sonnet-Thinking; the caption specifies 10 runs of 100 steps for the stated setting.Source: §3.4 · pp. 5–6, Table 1 · p. 6

4.2 G1: Reinforcement Learning with Perception-Enhanced Cold Start

• G1-7B achieves the highest performance across all games and surpasses every baseline, including Claude-3.7-Sonnet-Thinking.The G1 evaluation follows perception-enhanced cold-start SFT and RL.Source: §4 · p. 6, §4.2 · p. 8

4.1 G0: From Zero to Game Masters without Supervision

• During 400 RL steps, Shisen-Sho G0’s reward rose from -1 to 0.8, raising evaluation performance to 12.8, above the 7B base model’s 1.9 and 72B version’s 2.6.After 400 steps, it developed exact-coordinate localization in perception and row-by-row enumeration in reasoning.Source: §4.1 · pp. 6–7
• G0 improves more slowly on Shisen-Sho-Cifar10, where harder visual recognition creates a perception bottleneck, although it still exceeds its baseline.The reported evaluation scores are 8.0 for G0 and 0.4 for its baseline.Source: §4.1 · p. 7
• G0 gains little on 2048 because inaccurate perceptions can still earn reward, and on Swap because the base model rarely receives positive reward.The paper attributes these cases respectively to inaccurate reward credit and sparse reward.Source: §4.1 · p. 7
• Localization patterns increase only in Shisen-Sho variants, preceding reward gains; 2048 shows none throughout training.The pattern counts were calculated from model outputs using regular expressions.Source: §4.1 · p. 7

4.2 G1: Reinforcement Learning with Perception-Enhanced Cold Start

• G1 trains more efficiently across games; its perception accuracy stays high in visually simple games but rises alongside reward on difficult Shisen-Sho-Cifar10.The authors attribute sustained accuracy in simpler environments to perception-enhanced cold start and their low perceptual complexity.Source: §4.2 · p. 9

4.3 Discussions

• Perception reward accelerates perception accuracy without changing game reward, while perception and reasoning can co-evolve using only final accurate, verifiable reward.In Shisen-Sho, complete tile recognition is needed for full perception reward but not always for actions that earn game reward.Source: §4.3 · p. 10

4.4 Limitations and Future Work

• VLM-Gym currently covers four games with relatively straightforward rules, and training still faces sparse-reward challenges, particularly in Swap.The authors identify more complex mechanics, deeper strategy, and long-horizon tasks with infrequent feedback as future-work directions.Source: §4.4 · p. 10

• Game evaluation progressed from text-only and text-converted states [39], [9] to direct visual-text inputs [21] [36], while several RL game studies remained text-based [41],.Source: §5 · p. 10
• Visual-domain RL studies report reasoning improvements,, [16], while Vision-R1 [13] and R1-OneVision [40] build multimodal chain-of-thought datasets from visual information.Source: §5 · p. 10
• Most existing work examines perception and reasoning separately in RL, leaving their mutual improvement unclear; G0 and G1 provide initial evidence of bootstrapping during RL.Source: §5 · p. 11

• G1 achieves state-of-the-art performance, outperforming even Claude-3.7-Sonnet-Thinking.Source: §6 · p. 11
• VLM-Gym is a scalable multi-game RL suite, while G0 uses pure RL and G1 adds perception-enhanced cold start and knowledge distillation.Source: §6 · p. 11
• The analysis reports synergistic bootstrapping between perception and reasoning during RL training.Source: §6 · p. 11
• The authors present VLM-Gym and the RL training framework as resources for future research on interactive agents and multimodal RL.Source: §6 · p. 11

• Figure 9 outlines perception-enhanced cold-start data construction through knowledge distillation and programmable environments.Source: Figure 9 · p. 16

• Before RL, incorrect perception and reasoning outputs in the 2048 case still received positive rewards, encouraging the policy to adopt those behaviors.Source: Figure 10 · p. 17

C Details of Games

• Matching tiles merge when they collide; successful merges spawn a new tile, while a swipe without a merge is a failed move that adds a new tile.The spawned tile is usually a 2 and occasionally a 4.Source: C Details of Games · p. 18
• The model takes the 4×4 game board as input and outputs one integer for its chosen swipe: 0 Up, 1 Right, 2 Down, or 3 Left.Source: C Details of Games · p. 18
• The game prompt asks the model to reach the 2048 tile and achieve the highest score.The game is described as a sliding-tile puzzle in which numbered tiles are combined to create a 2048 tile.Source: C Details of Games · p. 18
• The required response format puts the board description in <perception>, the thinking process in <think>, and the final action in <answer>.Source: C Details of Games · p. 18

C.2 Shisen-Sho and Shisen-Sho-Cifar10

• In Shisen-Sho, a pair can be cleared only when tiles match in both shape and color and an unobstructed orthogonal path connects them with at most two turns.A move fails if the tiles differ or no such path exists, leaving the board unchanged.Source: §C.2 · pp. 18–19
• The model receives an 8×8 board and returns coordinates for a matching pair connected by an unobstructed orthogonal path of at most two turns.For the image variant, a match is based on CIFAR class; for the traditional game, it is based on shape and color.Source: §C.2 · p. 19
• Shisen-Sho-CIFAR10 uses CIFAR-10 images as tiles, and a valid pair must belong to the same image class and have a connecting path with at most two turns.The listed classes include airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.Source: §C.2 · pp. 19–20
• The prompt requires a board description in <perception>, reasoning in <think>, and the selected coordinate pair in <answer>.Source: §C.2 · p. 20

C.3 Swap

• A successful Swap move exchanges side-adjacent shapes to create a horizontal or vertical line of at least three identical shapes; otherwise the swap fails and the tiles revert.Source: §C.3 · p. 20
• The model receives an m×n board and outputs the coordinates of the first valid adjacent swap that creates a match of three or more.Source: §C.3 · p. 20
• The game description says matched tiles are removed, tiles above fall, and new tiles appear at the top; no valid moves trigger a shuffle, and play ends when moves run out.Source: §C.3 · p. 20
• The prompt requires a board description in <perception>, reasoning in <think>, and the final coordinate pair in <answer>.Source: §C.3 · p. 20