Source-linked AI summary
Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models
Minghe Shen, Zhuo Zhi, Chonghan Liu, Shuo Xing, Zhengzhong Tu, Che Liu
TL;DR
Research on RLVR’s reasoning effects has largely focused on language-dominant tasks, leaving visual-spatial reasoning less explored. This paper introduces Ariadne, a controlled synthetic-maze framework, and reports that RLVR-trained VLMs succeed beyond the base model’s effective support and improve zero-shot results on real-world navigation benchmarks.
Problem
Whether RLVR expands VLM spatial reasoning or only improves sampling of existing behaviors remains open, with prior evidence concentrated in language-dominant settings.
Source: §1 · p. 1Method
Ariadne uses synthetic maze navigation with regulated difficulty to probe VLM reasoning boundaries, and trains models with RLVR.
Source: §1 · pp. 1–2Results
RLVR-trained models achieve non-trivial success where the base model has 0% accuracy despite extensive sampling, with zero-shot improvements on MapBench and ReasonMap.
Source: §1 · p. 2Takeaways & Limitations
The findings support spatial reasoning capability expansion, although the precise boundary measured in synthetic mazes does not directly transfer to real-world tasks.
Source: §5 · p. 9The induced policy eventually encounters its own complexity horizon in extreme out-of-distribution settings.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Recent studies posit that Reinforcement Learning with Verifiable Rewards (RLVR) primarily amplifies behaviors inherent to the pre-training distribution rather than inducing new capabilities, but these insights are predominantly limited to language-only domains, leaving the dynamics of visual-centric spatial reasoning under-explored. To examine the impact of RLVR on the capability boundaries of Vision-Language Models (VLMs), we introduce \textbf{Ariadne}, a controlled framework based on synthetic maze navigation where the reasoning difficulty is precisely regulated by path length and the number of turns. We demonstrate that applying RLVR extends the spatial reasoning boundary, achieving success on problems where the base policy VLM consistently attains $0\%$ accuracy despite increasing pass@k sampling budgets, indicating that the optimized policy successfully navigates search spaces that were effectively unreachable by the base distribution. Furthermore, despite being trained exclusively on synthetic mazes, we evaluate the model on two real-world navigation benchmarks (MapBench and ReasonMap) in a zero-shot setting. The observed improvements in these out-of-domain tasks suggest genuine spatial reasoning capability expansion rather than mere sampling efficiency.
1 Introduction
The paper asks whether RLVR expands VLM spatial reasoning or only improves sampling of behaviors already latent in the base policy. It introduces Ariadne and reports success beyond the base model’s effective support, including zero-shot gains on real-world navigation benchmarks.
1 Introduction
- The central question is whether RLVR induces novel reasoning paths or merely improves sampling of behaviors already present in the base policy.Prior language-dominant findings favor the efficiency explanation, while visual-centric spatial reasoning is described as a low-support setting.Source: §1 · p. 1
- The study trains VLMs with GRPO to test whether they can navigate search spaces lacking effective support in the base distribution.The introduction frames success in such settings as evidence against an efficiency-only account of RLVR.Source: §1 · p. 2
- RLVR-trained models achieve non-trivial success where the base model has 0% accuracy despite extensive sampling, and improve zero-shot performance on MapBench and ReasonMap.The authors present these results as evidence of spatial reasoning capability expansion rather than mere sampling efficiency.Source: §1 · p. 2
- Ariadne uses synthetic maze navigation to regulate difficulty and probe VLM reasoning boundaries with verifiable rewards.The framework characterizes boundary complexity through path length and number of turns.Source: §1 · pp. 1–2
2 Related Work
Related work describes RLVR successes in language-heavy tasks alongside a debate about whether those gains expand capabilities or improve sampling. It identifies spatial planning in VLMs as an open setting for testing that distinction.
RLVR in Vision-Language Models
- VLM applications of RLVR include visual question answering and chart understanding, but evaluations remain predominantly language-centric (Huang et al., 2025; Wang et al., 2025a; Wan et al., 2025).The cited passage contrasts these tasks with executable spatial planning.Source: §2 · p. 2
- Whether RLVR improves VLM spatial reasoning in long-horizon navigation remains open, where pre-trained models often fail to generate valid action sequences (Feng et al., 2025; Xing et al., 2025).The paper positions its investigation as a counterpoint to the efficiency hypothesis, testing a setting where prior coverage is effectively absent.Source: §2 · p. 2
The Efficiency vs. Capability Debate
- Language-only analyses argue that RLVR mainly selects correct trajectories already within pre-trained models’ latent support rather than synthesizing novel capabilities (Ye et al., 2025).Math benchmarks can reach RLVR-level performance with sufficient sampling, consistent with strong prior coverage (AI et al., 2025).Source: §2 · p. 2
3 Method
Ariadne is a controllable maze-navigation framework for measuring VLM reasoning boundaries, training with GRPO, and evaluating generalization to more complex mazes. Its method combines verifiable maze tasks, a difficulty-aware curriculum, and dense prefix-matching rewards.
3 Method
- Ariadne generates synthetic maze tasks with explicitly regulated difficulty to identify regimes where the base model consistently fails to produce valid solutions.The framework is designed to test whether RLVR extends VLM reasoning beyond the base policy’s effective reach.Source: §3 · p. 3
3.1.1 Verifiable Maze Environment
- The maze environment presents images requiring textual sequences of discrete up, down, left, or right actions that obey wall constraints.Start and goal cells and walls are visually marked; path length and turns vary to control difficulty.Source: §3.1.1 · p. 3, Figure 1 · p. 3
3.1.2 Curriculum for Boundary Expansion
- The curriculum samples path lengths s ∈ {1, . . . , 5} with an inverted Gaussian-like distribution, emphasizing short anchors and boundary-frontier tasks.Training uses L ≤5 and Turns ≤2, while evaluation uses unseen OOD instances with L ≤10 and Turns ≤4.Source: §3.1.2 · p. 3
3.2 Policy Optimization via GRPO
- GRPO updates the policy to maximize a surrogate objective, shifting probability toward higher-reward trajectories.The method samples groups of outputs and derives advantages by normalizing their rewards relative to the group.Source: §3.2 · pp. 3–4
3.3 Dense Verifiable Reward Design
- The dense reward gives partial credit for matching the ground-truth action prefix and scales reward by turn-based complexity.It rewards correct steps even when the goal is missed and is intended to prevent collapse into simple straight-line heuristics.Source: §3.3 · p. 4
4 Experiments
The experiments test whether RLVR extends VLM spatial reasoning beyond the base model’s effective support, using controlled maze difficulty and zero-shot real-world navigation benchmarks. Ariadne extends the synthetic-maze boundary and improves navigation results on MapBench and ReasonMap, though gains in synthetic mazes taper at extreme complexity.
4.1 Experimental Setup
- Ariadne is trained with GRPO on 4,700 AlphaMaze samples; an SFT variant uses the same data and initialization without reinforcement learning.Training samples groups of eight candidate responses, and evaluation uses independent rollouts at temperature 1.0.Source: §4.1 · p. 4
4.2 Benchmarks and Metrics
- Evaluation combines a held-out maze test stratified by training-distribution and OOD difficulty with zero-shot MapBench street-map and ReasonMap transit-network benchmarks.Maze success rate requires an exact match to the ground-truth path; MapBench reports shortest-path score and ReasonMap uses weighted Map Score.Source: §4.2 · pp. 4–5
4.3 RLVR Extends the Effective Reasoning Boundary
- In controlled mazes, Ariadne shifts the reasoning horizon from about 2 to 6 steps and recovers about 10% success on 2-turn mazes where the base model fails.For base-model tasks beyond 5 steps or 2 turns, coverage remains at 0% through pass@512; Ariadne generalizes to unseen 6–7-step and 3-turn cases, but approaches its limit at extreme complexity.Source: §4.3 · pp. 5–6, Figure 4 · p. 6
4.4 Zero-Shot Transfer to Real-World Tasks
- Ariadne broadly improves zero-shot MapBench route optimality over the base and SFT models, with the largest gains on sparse, irregular Trail and National Park terrain.The models perform similarly in Urban layouts.Source: Table 2 · p. 7, §4.4 · p. 7
- Across MapBench model scales, Ariadne improves route optimality, with particularly pronounced gains for smaller models and structurally complex Trail and National Park environments.The reported gains continue across nearly all environments as model scale increases.Source: Table 3 · p. 8, §4.4 · p. 8
- On ReasonMap Long Questions, average token count doubles from 61 to 121, weighted accuracy gains 1.47% absolute, and Map Score rises from 4.51 to 5.15.SFT does not show similar gains in this regime.Source: Table 4 · p. 8, §4.4 · p. 8
- Across difficulty tiers, Ariadne maintains higher ReasonMap Map Scores than the base model, and its advantage persists as map difficulty rises.Ariadne’s gains also persist across model scales, especially for long questions.Source: §4.4 · pp. 8–9, Figure 6 · p. 9
5 Conclusion
The authors conclude that RLVR extends VLM spatial reasoning beyond the base policy’s effective support, with zero-shot transfer to real-world navigation tasks. They also note that synthetic-maze reasoning boundaries do not directly transfer to real-world tasks.
5 Conclusion
- RLVR extends VLM spatial reasoning beyond the base policy’s effective support, with zero-shot transfer suggesting learned behaviors generalize beyond synthetic mazes.The authors describe these behaviors as novel spatial primitives, evidenced by transfer to MapBench and ReasonMap and effectiveness across model scales.Source: §5 · p. 9
- Ariadne-trained models show zero-shot transfer to MapBench and ReasonMap and consistent effectiveness across model scales.Source: §5 · p. 9
- Visual-spatial reasoning appears to require boundary expansion, while synthetic-maze boundaries do not directly transfer to real-world tasks involving complex visual semantics and logical depth.Source: §5 · p. 9
Limitations
The analysis primarily uses Ariadne mazes as a proxy for spatial intelligence, and the authors identify limits in scenario coverage, model scale, and extreme out-of-distribution reasoning.
Limitations
- The primary analysis uses Ariadne mazes as a proxy for spatial intelligence, so broader real-world scenarios with rigorous difficulty definitions are needed to validate boundary transferability.Source: Limitations · p. 10
- Computational constraints limited experiments primarily to smaller Qwen2.5-VL and Qwen3-VL models, leaving frontier-model capacity for permeable boundaries unresolved.Source: Limitations · p. 10
- The induced policy reaches a complexity horizon in extreme out-of-distribution settings, while opaque pretraining distributions prevent definitive mapping of existing models’ absolute reasoning boundaries.Source: Limitations · p. 10
A.1 LLM Usage Statement
LLMs assisted with language refinement and manuscript clarity, but not with the research’s scientific work. The authors state that they verified the scientific content and take responsibility for the paper.
A.1 LLM Usage Statement
- LLMs were used as general-purpose assistive tools for language refinement and improving manuscript clarity.Source: §A.1 · p. 13
- The authors state that research ideation, experimental design, and core scientific contributions did not rely on LLMs.Source: §A.1 · p. 13
- The authors say they generated and verified all scientific content, results, and conclusions, and take responsibility for the paper.Source: §A.1 · p. 13
A.2 System Prompt
The system prompt frames AlphaMaze as visual path-finding from a marked start to a marked target through open cells. It specifies allowable moves and a staged reasoning and output format.
A.2 System Prompt
- The AlphaMaze assistant must find a valid path from the green start cell to the red target cell in a maze image.Source: §A.2 · p. 13
- Movement is restricted to open cells and the four cardinal directions, with black walls treated as impassable.Source: §A.2 · p. 13
- The prompt asks the assistant to reason through positions and moves inside think tags, then output only the final movement sequence using the allowed tokens.Source: §A.2 · p. 13
A.3 Additional Experimental Results
Finer-grained pass@k evaluations preserve the reported scaling patterns: coverage rises smoothly with sampling budget, and the relative behavior of the base model and Ariadne remains consistent.
A.3 Additional Experimental Results
- The fine-grained evaluation uses low-to-mid budgets in Table 6 and sampling strides of 10 up to pass@100 in Figure 7.These denser budgets resolve intermediate scaling behavior more precisely than the sparsely spaced main-text evaluations.Source: §A.3 · p. 13
- The base model and Ariadne retain their relative behavior across intermediate sampling budgets, consistent with stable policy scaling patterns.The reported consistency is not an artifact of the sparsely spaced evaluation points in the main text.Source: §A.3 · p. 13
- Coverage transitions smoothly from low to high as the sampling budget k increases.This trend is reported under the finer-resolution evaluation.Source: §A.3 · p. 13
A.4 Examples from Spatial Reasoning Benchmarks
The section presents MapBench and ReasonMap as external tests of spatial reasoning transfer beyond synthetic mazes, with figures showing representative benchmark examples.
A.4 Examples from Spatial Reasoning Benchmarks
- The external benchmarks test whether learned spatial reasoning strategies transfer beyond synthetic mazes across visual layouts that differ in structure and appearance.They are presented as a realistic testbed for assessing transfer beyond the training distribution.Source: §A.4 · p. 13
- MapBench evaluates instruction-following navigation on street maps, while ReasonMap evaluates fine-grained route planning on transit schematics.Source: §A.4 · p. 13
- Table 6 reports base-model and Ariadne coverage by step count and sampling budget, while Figure 7 plots pass@k coverage across step and turn complexity.Figure 7 covers sampling budgets through k ≤100, with stride = 10.Source: Table 6 · p. 14, Figure 7 · p. 14
- Figures 8 and 9 show representative examples from MapBench street maps and ReasonMap transit schematics, respectively.Source: §A.4 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15