Source-linked AI summary
What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis
Xirui Li, Ming Li, Tianyi Zhou
TL;DR
The paper addresses uncertainty about which visual-reasoning capabilities RL improves beyond supervised initialization and introduces a layer-wise Frankenstein-style analysis to separate them. It finds that RL consistently refines middle and late transformer layers, improving vision-to-reasoning alignment and reasoning, with these refinements transferable and necessary for RL gains.
Problem
It remains unclear whether RL improves vision, vision-to-reasoning alignment, or pure reasoning, because benchmark accuracy cannot distinguish these sources.
Method
The paper decomposes VLMs by transformer layers using causal functional localization, parameter-based update characterization, model merging, and model freezing.
Results
RL consistently induces structured refinements in middle and late layers, reflected in increased reasoning-to-vision attention and improved vision-to-reasoning alignment and reasoning; these refinements transfer through merging and are necessary for RL gains.
Takeaways & Limitations
RL's reliable visual-reasoning contribution is a systematic refinement of mid-late transformer computation rather than uniform improvement of visual perception or standalone reasoning.
Takeaways & Limitations
The functional regions are coarse reference frames rather than precise architectural boundaries, and the analysis interprets later effects through this Early-Mid-Late partition.
Abstract
from arXiv · showhide
Reinforcement learning (RL) with verifiable rewards has become a standard post-training stage for boosting visual reasoning in vision-language models, yet it remains unclear what capabilities RL actually improves compared with supervised fine-tuning as cold-start initialization (IN). End-to-end benchmark gains conflate multiple factors, making it difficult to attribute improvements to specific skills. To bridge the gap, we propose a Frankenstein-style analysis framework including: (i) functional localization via causal probing; (ii) update characterization via parameter comparison; and (iii) transferability test via model merging. Instead, RL induces a consistent inference-time shift primarily in mid-to-late layers, and these mid-to-late refinements are both transferable (via merging) and necessary (via freezing) for RL gains. Overall, our results suggest that RL's reliable contribution in visual reasoning is not a uniform enhancement of visual perception, but a systematic refinement of mid-to-late transformer computation that improves vision-to-reasoning alignment and reasoning performance, highlighting the limitations of benchmark-only evaluation for understanding multimodal reasoning improvements.
1 Introduction
The paper asks what RL changes in visual reasoning beyond benchmark accuracy and analyzes these changes by localizing functions, comparing updates, and testing transferability and necessity across layers.
- End-to-end benchmark accuracy cannot distinguish improvements in vision, vision-to-reasoning alignment, and pure reasoning.
- The Frankenstein-style framework localizes vision and reasoning through causal probing, characterizes layer-wise updates by parameter comparison, and tests transferability through model merging.
- Freezing mid or late regions largely removes RL gains, whereas freezing early regions has a much weaker effect, indicating the importance of mid-late refinement.
- RL consistently shifts inference behavior toward greater attention from reasoning tokens to visual tokens, primarily in mid-late transformer layers.
- RL produces structured mid-late layer refinements that contribute primarily to vision-to-reasoning alignment and reasoning performance, while fine-grained abilities do not improve monotonically from Base through IN to RL.
2 Motivation
End-to-end benchmark gains obscure which visual-reasoning abilities RL improves. Fine-grained metrics and attention analysis instead separate vision, alignment, reasoning, and inference-time computation.
- Motivation: Extended inference can improve language-side reasoning, but does not reliably add visual evidence or resolve perceptual errors.It may even amplify reliance on language priors, worsening visually grounded errors.
- Motivation: End-to-end accuracy cannot distinguish stronger reasoning from altered attention to visual evidence or improved visual representations.The same aggregate gain may arise from qualitatively different internal changes.
- Motivation: The evaluation decomposes visual reasoning into Vision, Vision-to-Reasoning alignment, and language-side Reasoning using paired visual, black-image, textualized, and text-only inputs.These controls target whether visual evidence is causally used, preserved in reasoning, or unnecessary for the answer.
- Inconsistent Fine-grained Improvements: Average benchmark accuracy rises monotonically from Base to IN to RL, whereas neither vision nor standalone reasoning improves monotonically across stages.This discrepancy shows why benchmark-only evaluation cannot identify which functional components RL strengthens.
- Consistent Attention Patterns: RL consistently increases attention from reasoning tokens to visual tokens, concentrated in mid-late layers while early-layer patterns remain largely unchanged.The shift appears across training recipes and is absent in earlier layers.
3 Frankenstein-Style Analysis
The Frankenstein-style framework localizes vision and reasoning functions, characterizes IN/RL parameter updates, and tests whether RL-refined regions transfer through model merging. Across these analyses, RL effects concentrate in Mid-Late layers and primarily preserve gains in vision-to-reasoning alignment and reasoning.
- Framework: The framework combines causal functional localization, parameter-update comparison, and region-wise model merging to identify where RL changes VLMs.It uses a coarse Early/Mid/Late partition as a functional reference frame for subsequent analyses.
- Functional Localization: Vision functionality is associated mainly with Early and Mid layers, whereas reasoning computations are concentrated in Late layers.The partition divides transformer depth into three equal regions and abstracts away from precise layer indices.
- Update Characterization: Both IN and RL concentrate update energy in Mid layers, consistent with their role in bridging visual perception and high-level reasoning.Per-layer update energy is measured with the Frobenius norm ∥∆W(ℓ)∥F.
- Update Characterization: RL updates show steeper singular-value decay than IN updates in Mid-Late layers, indicating concentration in fewer dominant directions.The normalized log spectrum compares directional decay while removing scale differences across layers.
- Model Merging: Hybrid models retaining RL-refined Mid-Late layers consistently preserve vision-to-reasoning alignment and reasoning improvements across training recipes.Alternative region combinations lack consistent improvements and can degrade performance, supporting transferability of Mid-Late refinements.
4 Necessity Validation via Model Freezing
Region-wise freezing tests whether Mid-Late refinement is necessary for RL gains rather than merely associated with them. Freezing Late layers substantially degrades performance, while preserving Mid-Late updates retains stronger results.
- Intervention Design: The freezing intervention blocks updates in one Early, Mid, or Late region during RL while keeping other training settings identical.This design tests whether preventing regional refinement mitigates RL improvements.
- Results: Freezing Late layers causes substantial degradation across both fine-grained and benchmark metrics.The result is reported in Table 3 for RL training.
- Results: Leaving Mid-Late layers trainable while freezing Early layers yields better performance than the other models, including IN and RL.This pattern supports the importance of preserving Mid-Late refinement during RL.
- Conclusion: Mid-Late refinement is identified as a necessary component for achieving RL improvements, not merely a correlated change.The conclusion is based on the region-wise freezing intervention.
5 Related Work
Related work positions RL post-training within a broader reasoning literature while contrasting benchmark-focused analyses with parameter-space studies and work examining visual processing changes.
- RL Post-Training: Recent reasoning post-training commonly uses supervised finetuning initialization followed by RL with verifiable rewards.RL variants such as GRPO, DAPO, GSPO, GFPO, and GDPO refine training stability or efficiency.
- Parameter-Space Analysis: Most prior RL analyses treat models as black boxes by emphasizing end-to-end metrics such as pass@k or accuracy.Only a limited number of studies examine where updates localize and how their geometry differs from supervised finetuning.
- Visual Reasoning: Chain-of-Thought finetuning and R1-style RL pipelines have been used to enhance visual reasoning behaviors in VLMs.Other approaches include model composition, vision tokens, and explicit grounding.
- Visual Processing: Prior work also reports that RL can improve the vision encoder more effectively than SFT, indicating that RL updates may extend beyond linguistic or reasoning components.This motivates examining visual processing alongside reasoning-related changes.
6 Conclusion
The paper finds that RL’s consistent contribution is a structured refinement of middle and late transformer computation, improving vision-to-reasoning alignment and reasoning rather than uniformly improving visual perception.
- RL is consistently associated with structured refinements in middle and late transformer layers.
- These refinements are reflected behaviorally in improved vision-to-reasoning alignment and reasoning capabilities.
- The framework reveals RL-induced changes that aggregate benchmark accuracy alone cannot explain.
A Limitations
The study’s scope is bounded by its focus on the dominant IN+RL paradigm, visual-reasoning VLMs, and Qwen-series models, while broader applicability remains untested.
- The analysis focuses on IN+RL post-training rather than direct RL from scratch.The findings may not directly generalize to alternative RL training recipes.
- The functional decomposition is limited to VLMs aimed at improving visual reasoning.Its transfer to other task domains or modalities remains uncertain.
- All experiments use models from the Qwen series.Applicability to models without clearly separable functional regions remains to be explored.
- The controlled analysis requires complete, well-aligned checkpoints across training stages.These checkpoints enable stage-wise comparison and layer-level interventions.
C Fine-grained Metrics
The appendix decomposes visual reasoning into vision, vision-to-reasoning alignment, and reasoning, then uses paired inputs and layer interventions to measure their functional localization.
- Metric Definitions: Visual reasoning is decomposed into Vision, Vision-to-Reasoning Alignment, and Reasoning abilities.These metrics use paired input settings that selectively control visual evidence.
- Vision: Vision measures whether visual input enables correct predictions where language-only inference fails.It is evaluated with an unconditional paired metric on General VQA tasks.
- Vision-to-Reasoning: Vision-to-Reasoning measures whether perceptual evidence is correctly incorporated into downstream reasoning.The metric requires correct outcomes for both real-image and textualized-visual inputs.
- Reasoning: Reasoning measures language-side multi-step inference without visual evidence.It is evaluated on textual math tasks.
- Functional Localization: Vision token swapping replaces visual tokens at a chosen transformer layer while preserving textual tokens and decoding.Output changes are interpreted as evidence that the layer consumes relevant visual information.
- Paired Image Datasets: Paired image datasets differ in exactly one visual attribute, enabling attribution of output changes to that perturbed attribute.The functions include OCR, object counting, grounding, and recognition.
- Layer Interventions: Layer-wise change rates form sensitivity profiles, with higher rates indicating stronger reliance on visual information at that layer.Reasoning localization separately uses zero ablation while preserving residual connections and normalization layers.
D.5 Interpretation and Limitations
The localization results provide a reference frame for interpreting RL effects, but their causal probes identify intervention-based necessity rather than a complete explanatory model of computation.
- High sensitivity indicates necessity under intervention, not sufficiency or exclusive responsibility.
- Vision token swapping and zero ablation jointly establish functional regions for interpreting RL-based post-training effects.
E Model Merging
The paper merges IN and RL checkpoints by transformer regions to compare region-wise effects, using a coarse Early/Mid/Late partition and complementary freezing experiments.
- Layer region partition: The 28-layer transformer is divided into equal-depth Early, Mid, and Late regions for consistent region-wise comparisons.Early covers layers 0–9, Mid layers 10–18, and Late layers 19–27.
- Layer region partition: The region boundaries are an analytical abstraction rather than precise architectural divisions.They provide a coarse depth-based reference frame for comparing relative functional roles.
- Model merging procedure: Hybrid models selectively transfer layers from IN or RL checkpoints while leaving other components unchanged.The compared checkpoints come from the same training recipe.
- Model merging procedure: Merging copies complete parameter states, including self-attention, feed-forward, and layer-normalization parameters, without post-merge fine-tuning or calibration.No re-normalization is performed after merging.
- Region-wise freezing: Freezing experiments exclude selected transformer layers from optimization while updating all remaining parameters normally.The experiments use the standard GRPO-based RL post-training pipeline with only the freezing modification.
F.2 Hardware and training setup.
The region-wise freezing experiments use a fixed multi-GPU training setup with reduced batching and sequence lengths for stability, while keeping other settings identical across conditions.
- Experiments use 2× NVIDIA H200-SXM GPUs, with each RL run trained for 2000 steps.
- Table 6 summarizes the full batching and rollout configuration for all region-wise freezing experiments.
- Training uses a per-device batch size of 1 with PPO minibatching applied across GPUs.
- Each prompt produces multiple candidate responses to support reward-based optimization.