AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPO

1 Introduction

• The proposed framework trains a standard LLM for maze navigation by using SFT to teach movement-token prediction, then GRPO and designed rewards to refine visual reasoning.The authors also analyze how reward functions shape performance.Source: §1 · pp. 1–2
• MazeBench provides a controlled, diverse benchmark for assessing maze-solving accuracy and emergent reasoning across spatial challenges.The introduction describes it as a benchmark for visual maze navigation.Source: §1 · p. 2
• The authors report that their tokenized-maze framework improves navigation accuracy and elicits emergent chain-of-thought reasoning in movement sequences.The introduction positions these findings in comparison with reasoning models such as DeepSeek-R1.Source: §1 · p. 2

2.1 Chain-of-Thought Reasoning in Language Models

• Chain-of-thought prompting encourages intermediate reasoning steps and has improved performance on arithmetic, commonsense, and symbolic reasoning tasks.AlphaMaze aims to induce a similar step-by-step process for visual spatial reasoning in maze navigation.Source: §2.1 · p. 2

2.2 Supervised Fine-Tuning for Visual and Spatial Tasks

• SFT adapts pretrained LLMs to downstream tasks; AlphaMaze uses it to teach processing of tokenized maze inputs and movement-token prediction before reinforcement learning.The paper presents this phase as the foundation for more sophisticated reasoning.Source: §2.2 · p. 2

2.3 Reinforcement Learning and GRPO for Reasoning and Reward Shaping

• GRPO and other self-play methods are presented as alternatives to human-feedback-intensive RLHF; GRPO estimates advantages from group scores without a separate critic network.The paper adapts reward shaping to encourage accurate, valid, and properly formatted maze outputs.Source: §2.3 · p. 2

2.4 DeepSeek-R1 and Emergent Reasoning through RL

• DeepSeek-R1 is cited as evidence that pure GRPO can elicit chain-of-thought and self-corrective “aha moments”; AlphaMaze adapts its optimization strategy to maze navigation.The paper hypothesizes that similar RL techniques can induce step-by-step, self-corrective visual spatial reasoning.Source: §2.4 · pp. 2–3

2.5 Visual Reasoning and Maze Solving in AI

• Prior work spans traditional search, neural and RL maze solving, and visual reasoning; AlphaMaze instead adapts standard language models using tokenized mazes with SFT and GRPO.The paper also identifies neural-symbolic visual reasoning as a complementary approach.Source: §2.5 · p. 3

3.1 Tokenized Visual Maze Representation

• The maze representation encodes each cell with a coordinate and wall tokens, marking origin and target explicitly.Blank tokens preserve the grid structure, making spatial relationships available in symbolic token form.Source: §3.1 · p. 3

3.2 Baseline Models

• The baselines include two distilled Qwen models and a 1.5B direct-prediction model trained to output complete movement sequences.Direct prediction uses cross-entropy against the ground-truth path, without intermediate reasoning or reinforcement learning.Source: §3.2 · pp. 3–4

3.3 Supervised Fine-Tuning (SFT) for Step-by-Step Reasoning

• SFT trains the Qwen 1.5B model on fixed-size 5x5 mazes to predict each next movement token from the maze and preceding moves.The step-by-step objective is intended to encourage sequential reasoning.Source: §3.3 · p. 4

3.4 Group Relative Policy Optimization (GRPO) for Enhanced Reasoning

• GRPO rewards correct solution steps, valid movement tokens, and correctly formatted <think> tags, while prioritizing correctness.Correctness adds 0.2 per solution step; integrity adds 0.5 per valid token; the thinking reward adds 0.25. GRPO estimates advantages from relative group scores rather than a critic.Source: §3.4 · p. 4

3.5 Training Procedure and Pipeline

• The pipeline applies 10 SFT epochs to Qwen 1.5B, then refines the SFT model with GRPO using LoRA.The GRPO phase uses Unsloth with VLLM for inference and saves checkpoints every 200 steps.Source: §3.5 · p. 4

4.2 MazeBench

• MazeBench contains 100 unique mazes sampled from the 30,000-maze test set, with difficulty based primarily on the number of steps to the goal.Success rate is the percentage of mazes solved correctly; a path is incorrect if its extracted movement tokens fail to reach the target or enter an invalid state.Source: §4.2 · p. 5

4.3.1 Model Performance on MazeBench

• 93.0% accuracy after 1600 GRPO steps follows 86.0% for SFT-only and 0% for the direct-prediction baseline on MazeBench.The comparison is reported in Table 2.Source: Table 2 · p. 6, §4.3.1 · p. 6

4.3.2 Model Evolution During GRPO

• MazeBench scores rise over GRPO steps in the Figure 1 trendline, which is described as indicating improved maze-solving policies.The figure displays scores, a linear regression trendline, and its ±1 standard deviation bounds.Source: §4.3.2 · p. 6

4.4 Qualitative Results

• The baseline often produces invalid or incomplete paths, while SFT outputs are more coherent but can loop or turn incorrectly on longer mazes.SFT+GRPO outputs show emergent reasoning about walls and spatial relationships, with fewer invalid moves and greater robustness to long-context challenges; backtracking and planning beyond the next step remain difficult.Source: §4.4 · p. 6

5.1 Analysis of GRPO’s Impact on Visual Maze Reasoning

• 93% accuracy on MazeBench after 1600 GRPO steps, up from the SFT model’s 86%.The discussion describes GRPO’s gain as 7 percentage points.Source: §5.1 · p. 6
• GRPO training accompanies more pronounced chain-of-thought reasoning and self-correction during maze navigation.The reward function incentivizes correctness, valid movements, and structured output.Source: §5.1 · p. 6

5.2 Comparison with DeepSeek-R1 and RL for Reasoning

• DeepSeek-R1 Distill-Qwen variants score 0% accuracy on MazeBench, while the two-stage-trained distilled model reaches 93%.The base DeepSeek-R1 model shows emergent visual reasoning with an extremely long context window.Source: §5.2 · p. 6

5.3 Limitations

• The reported GRPO gain is small at 7% accuracy improvement, and larger gains from longer training or reward changes remain to be investigated.The authors suggest that reward design could better incentivize backtracking or proactive exploration.Source: §5.3 · p. 7
• Evaluation relies primarily on maze-solving accuracy, which gives a limited view of reasoning; the authors call for path-efficiency, robustness, and interpretability measures.They note that qualitative chain-of-thought observations need more rigorous analysis.Source: §5.3 · p. 7
• Experiments use synthetically generated mazes, which may not capture the variability of real-world visual spatial tasks.The authors call for testing generalization to more diverse, ecologically valid environments and tasks.Source: §5.3 · p. 7

• Pre-trained LLMs achieved 0% accuracy on MazeBench, while SFT taught movement prediction from tokenized maze inputs and reached 86% accuracy.The conclusion presents SFT as the foundation for the later GRPO stage.Source: §6 · p. 7
• The method reached 93% on MazeBench after 1600 training steps; qualitative analysis found GRPO fostered more sophisticated, self-corrective reasoning strategies, including emergent chain-of-thought patterns.Source: §6 · p. 7
• The authors see the two-stage approach as potentially applicable to visually grounded tasks requiring spatial understanding and sequential decision-making beyond mazes.They describe GRPO’s transfer to spatial domains as suggesting the transferability of these techniques.Source: §6 · p. 7
• Future work will test the approach across other reasoning domains, vary SFT and GRPO configurations, diversify training data, and refine reward functions.Source: §6 · p. 7

• The algorithm’s stated purpose is to augment maze-navigation problems with incorrect attempts followed by resets and correct solutions.Source: A Algorithm · p. 9
• The main process extracts each maze’s adjacency list, origin, target, and correct path, then routes examples with one or two origin walls to separate processing procedures.Processed examples from the two groups are combined into the final dataset.Source: A Algorithm · p. 9
• For one-wall origins, the procedure explores adjacent nodes outside the correct path and extends wrong paths until a dead end or step limit.The generated demonstrations include chain-of-thought, a wrong-direction message, a reset marker, and the original correct solution.Source: A Algorithm · p. 9
• For two-wall origins, the procedure searches for a wrong path from the origin, labels it as a dead end or wrong direction, then adds a reset and the correct solution.Source: A Algorithm · p. 10