PaperlayerView exact revision on arXiv

Source-linked AI summary

RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning

Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, Huan Wang

arXiv:2510.02240v2cs.CVcs.AI

TL;DR

Fine-grained visual reasoning in MLLMs is challenged by sparse rewards and difficult structured inputs such as transit maps. The paper introduces ReasonMap-Plus and RewardMap, combining dense supervision, difficulty-aware rewards, and multi-stage RL; RewardMap improves results on targeted benchmarks and averages 3.47% gains across six broader benchmarks.

  • Problem

    Fine-grained visual reasoning over structured inputs remains difficult for MLLMs, while standard RL faces sparse rewards and unstable optimization.

  • Method

    The paper constructs ReasonMap-Plus and proposes RewardMap, which combines difficulty-aware detail rewards with an easy-to-hard multi-stage RL curriculum.

  • Results

    3.47% average improvement across six benchmarks accompanies consistent gains on ReasonMap and ReasonMap-Plus, with RewardMap outperforming baselines across their question types.

  • Takeaways & Limitations

    RewardMap provides a unified approach for improving visual understanding and reasoning across transit-map, spatial, fine-grained visual, and general tasks.

  • Takeaways & Limitations

    The evaluation uses publicly available transit maps and automatically generated question–answer pairs released for academic research.

Abstract

from arXiv · show

Fine-grained visual reasoning remains a core challenge for multimodal large language models (MLLMs). The recently introduced ReasonMap highlights this gap by showing that even advanced MLLMs struggle with spatial reasoning in structured and information-rich settings such as transit maps, a task of clear practical and scientific importance. However, standard reinforcement learning (RL) on such tasks is impeded by sparse rewards and unstable optimization. To address this, we first construct ReasonMap-Plus, an extended dataset that introduces dense reward signals through Visual Question Answering (VQA) tasks, enabling effective cold-start training of fine-grained visual understanding skills. Next, we propose RewardMap, a multi-stage RL framework designed to improve both visual understanding and reasoning capabilities of MLLMs. RewardMap incorporates two key designs. First, we introduce a difficulty-aware reward design that incorporates detail rewards, directly tackling the sparse rewards while providing richer supervision. Second, we propose a multi-stage RL scheme that bootstraps training from simple perception to complex reasoning tasks, offering a more effective cold-start strategy than conventional Supervised Fine-Tuning (SFT). Experiments on ReasonMap and ReasonMap-Plus demonstrate that each component of RewardMap contributes to consistent performance gains, while their combination yields the best results. Moreover, models trained with RewardMap achieve an average improvement of 3.47% across 6 benchmarks spanning spatial reasoning, fine-grained visual reasoning, and general tasks beyond transit maps, underscoring enhanced visual understanding and reasoning capabilities.

1 INTRODUCTION

Fine-grained visual reasoning on structured visual inputs remains difficult for MLLMs, while sparse RL rewards and SFT’s limitations hinder effective training. RewardMap addresses these challenges with dense supervision, difficulty-aware rewards, and multi-stage RL, improving performance across targeted and broader benchmarks.

  • 1 INTRODUCTION: ReasonMap targets practical and scientifically important spatial reasoning over high-resolution transit maps, exposing gaps in current MLLMs.The task combines visual understanding with spatial reasoning for applications such as navigation and transportation systems.
  • 1 INTRODUCTION: Standard RL struggles because complex visual reasoning provides sparse feedback only after long reasoning chains, destabilizing optimization and hindering exploration.SFT provides denser supervision but does not equip models for the long-chain decisions intrinsic to visual reasoning.
  • 1 INTRODUCTION: RewardMap combines an extended dataset, difficulty-aware detail rewards, and multi-stage RL to address sparse supervision in fine-grained visual reasoning.ReasonMap-Plus supplies dense supervision, while RewardMap progresses from perception to reasoning and directly employs RL for cold-start training.
  • 1 INTRODUCTION: 3.47% average improvement across six benchmarks shows that RewardMap-trained models improve beyond transit-map tasks in spatial, fine-grained visual, and general reasoning.The reported benchmarks span spatial reasoning, fine-grained visual reasoning, and general tasks.

2 RELATED WORK

Prior work develops multimodal reasoning, reinforcement-learning optimization, curriculum strategies, and specialized transit-map pipelines. These lines of research motivate RewardMap’s focus on visual grounding, RL-based reasoning, and progression from easier to harder tasks.

  • Visual Reasoning in MLLMs: Multimodal reasoning research extends chain-of-thought prompting to vision-language models, but early approaches rely heavily on textual rationales.The passage identifies textual-rationale reliance as a limitation of early multimodal chain-of-thought methods.
  • Reinforcement Learning for Reasoning: RL research has progressed from RLHF toward direct objectives such as DPO and GRPO, while curriculum RL studies easy-to-hard task progression.These approaches seek stronger reasoning than static SFT and more stable policy optimization.
  • Spatial Reasoning on Maps: ReasonMap-Plus broadens map reasoning data to 4,018 questions spanning five question types and maps from 30 cities across 13 countries.The dataset overview provides the scale and geographic diversity of the extended benchmark.
  • Spatial Reasoning on Maps: Transit-map reasoning traditionally uses OCR, image processing, graph construction, and pathfinding rather than a unified multimodal reasoning model.The pipeline extracts station names and topological elements before solving routes on the resulting graph.

3 REASONMAP-PLUS CONSTRUCTION

ReasonMap-Plus is constructed from existing high-resolution transit maps and annotated line-stop data, with automatically generated, manually reviewed question–answer pairs. Its questions cover multiple visual-understanding categories and use map difficulty to support an easy-to-hard curriculum.

  • 3 REASONMAP-PLUS CONSTRUCTION: ReasonMap-Plus reuses high-resolution maps and annotated line-stop information, automatically generates question–answer pairs, and manually reviews them for correctness and diversity.The construction pipeline includes data collection and preprocessing, question–answer generation, and quality control.
  • 3 REASONMAP-PLUS CONSTRUCTION: Question types extend planning with global counting, local counting, and True-or-False tasks that probe fine-grained visual understanding.Questions are built from predefined templates and answers are automatically derived from Metro Data.
  • 3 REASONMAP-PLUS CONSTRUCTION: Question difficulty is assigned from the corresponding map difficulty, whose three levels distinguish easy, middle, and hard maps.These questions target basic visual understanding rather than the complex reasoning of planning questions.

4 METHODOLOGY

RewardMap combines difficulty-aware reward shaping with multi-stage GRPO training to address sparse rewards in fine-grained visual understanding and reasoning tasks. Its curriculum progresses from perception-oriented VQA to increasingly complex spatial reasoning while using partial-credit detail rewards and stochastic within-stage sampling.

  • 4.1 OVERVIEW: The target tasks cast high-resolution image-question answering as VQA, covering fine-grained visual understanding in REASONMAP-PLUS and spatial reasoning in REASONMAP.Answers must follow the required format and correctly address the question, while REASONMAP route planning additionally evaluates spatial reasoning.
  • 4.1 OVERVIEW: Sparse rewards make GRPO advantages collapse toward zero or become highly skewed, producing low-signal or high-variance gradients that slow convergence.
  • 4.2 DIFFICULTY-AWARE REWARD DESIGN: RewardMap adds detail rewards and difficulty-aware weighting to format and correctness rewards, providing richer supervision for difficult visual reasoning tasks.The detail term awards partial credit for origin and destination stops, route names, transfer stations, and route-segment counts; the overall reward is scaled by difficulty.
  • 4.3 MULTI-STAGE REINFORCEMENT LEARNING: The multi-stage curriculum schedules binary judgment, counting, and planning tasks from visual understanding to visual reasoning before applying stochastic sample shuffling within each stage.The global ordering builds perceptual skills before abstract reasoning, while local randomness avoids overfitting to a fixed curriculum trajectory.
  • 4.1 OVERVIEW: REASONMAP-PLUS supplies dense-reward questions for cold-start training, and REWARDMAP uses GRPO to combine reward shaping with task scheduling for sparse-reward problems.

5 EXPERIMENTS

Experiments evaluate REWARDMAP on its target datasets, broader visual reasoning benchmarks, qualitative map-solving cases, and diagnostic ablations. Across these evaluations, REWARDMAP consistently improves performance, with its reward and multi-stage components jointly producing the strongest results.

  • 5.2 MAIN RESULTS: REWARDMAP consistently outperforms baselines on REASONMAP and REASONMAP-PLUS, approaching the closed-source leader on REASONMAP and surpassing it on REASONMAP-PLUS.The comparison covers different question templates in REASONMAP and question types in REASONMAP-PLUS.
  • 5.2 MAIN RESULTS: 13.51% is the largest reported gain across six broader benchmarks, observed on SpatialEval, while improvements also appear across multiple spatial, perception, reasoning, and alignment tasks.Reported examples include mazenav accuracy rising from 19.60% to 57.20% and HRBench single-image reasoning from 85.25% to 88.00%.
  • 5.3 QUALITATIVE RESULTS: REWARDMAP produces correct target routes more consistently than reference and baseline RL models, reducing visual confusion and route hallucination in qualitative map comparisons.The comparison includes cases where other models mistake routes or stops or repeat the same route.
  • 5.4 DIAGNOSTIC EXPERIMENTS: Combining reward design with multi-stage training achieves the best ablation results, while either component alone improves performance on both REASONMAP and REASONMAP-PLUS.The ablation supports the complementarity of the two components.
  • 5.4 DIAGNOSTIC EXPERIMENTS: Cold-start training with REASONMAP-PLUS improves both evaluation datasets under both SFT and RL, and finer-grained staging outperforms a coarse strategy.The coarse strategy still outperforms the baseline, but switching to it degrades performance relative to the proposed multi-stage design.
  • 5.4 DIAGNOSTIC EXPERIMENTS: REWARDMAP alleviates reward sparsity, remains effective across model scales and architectures, and shows limited sensitivity to α within a reasonable higher range.The reward-trajectory comparison, scale study, architecture study, and α ablation jointly support robustness of the training framework.

6 CONCLUSION

The paper addresses sparse-reward optimization in fine-grained visual reasoning by combining a difficulty-organized dataset with a multi-stage reinforcement-learning framework. Experiments report stronger target-dataset performance and broader visual-reasoning robustness when these components are integrated.

  • 6 CONCLUSION: REASONMAP-PLUS provides dense supervision along a difficulty continuum, while REWARDMAP combines curriculum-style task scheduling with difficulty-aware reward design.The framework is intended to facilitate cold-start training for fine-grained visual reasoning.
  • 6 CONCLUSION: Each component contributes to stable and effective training, and their integration yields the strongest improvements on target and broader visual reasoning benchmarks.The conclusion reports improved perceptual and reasoning capabilities of multimodal models.

ETHICS STATEMENT

The paper reports no human-subject or sensitive-personal-data concerns and releases its automatically constructed dataset for academic research. It also states that the reported results are reproducible with released code and configurations.

  • ETHICS STATEMENT: REASONMAP-PLUS uses publicly available transit maps and automatically generated question–answer pairs, without human subjects or sensitive personal data.The authors state that this creates no privacy or security concerns.
  • ETHICS STATEMENT: The datasets are released for academic research under Apache License 2.0 on HuggingFace, with code and configurations provided for reproducibility.

A DATASET CONSTRUCTION DETAILS

REASONMAP-PLUS extends the transit-map benchmark with automatically generated counting and True/False questions, process-level annotations, and difficulty metadata. Its construction derives answers and intermediate reasoning information from structured route–stop metadata.

  • A DATASET CONSTRUCTION DETAILS: REASONMAP-PLUS contains 4,018 questions from high-resolution transit maps covering 30 cities, divided into five categories and three difficulty levels.The dataset includes 1,259 easy, 1,342 middle, and 1,417 hard questions.
  • A DATASET CONSTRUCTION DETAILS: Questions are generated from templates for counting and True/False tasks, with answers automatically derived from Metro Data to preserve consistency with REASONMAP.The templates include intermediate-stop counting, line counting, total-line counting, and line-membership judgments.
  • A DATASET CONSTRUCTION DETAILS: Construction first records each map’s ordered line stops, interchange stations, and branching points in JSON metadata.A Singapore example lists route sequences such as the Circle Line and Downtown Line.
  • A DATASET CONSTRUCTION DETAILS: Task-specific scripts use the metadata to identify queried stations, compute answers, and record fine-grained intermediate results such as the stops between two stations.The same process supports True/False and shortest-path question types.
  • A DATASET CONSTRUCTION DETAILS: The appendix presents the complete computation pipeline for the Detail Reward used by the proposed training framework.

C EVALUATION DETAILS

REASONMAP-PLUS assigns reward weights solely by map difficulty, increasing from easy to hard.

  • REASONMAP-PLUS uses weights of 1.0, 1.5, and 2.0 for easy, medium, and hard maps, respectively.

C.2 DETAILS OF EVALUATION DATASETS

The evaluation spans spatial, fine-grained visual, high-resolution, chart, and broad vision-language benchmarks, alongside detailed planning-reward procedures.

  • The evaluation covers map, spatial, attribute-recognition, high-resolution-image, chart, and general vision-language tasks.The benchmarks include SEED-Bench-2-Plus Map, SpatialEval, V*Bench, HRBench, ChartQA, and MMStar.
  • SEED-Bench-2-Plus provides human-verified multiple-choice items across Charts, Maps, and Webs, with this evaluation using its Map slice.
  • The planning reward algorithm initializes the score at zero and returns immediately when route data is empty or incorrectly formatted.
  • SpatialEval measures spatial intelligence across relationships, position, counting, and navigation.
  • V*Bench evaluates fine-grained attribute recognition and spatial relationships on high-resolution images.
  • MMStar contains 1,500 human-curated, vision-indispensable samples spanning six core capabilities and 18 axes.

D.1 COMPARISON CASES

Comparison cases show RewardMap correcting transit-map counting errors made by the base model, while additional results report gains on structured visual benchmarks beyond transit maps.

  • RewardMap correctly identifies one intermediate stop and selects option C, whereas Qwen2.5-VL-7B-Instruct misidentifies three stops and selects option A.
  • The comparison question asks for the number of intermediate stops between A Station and B Station, excluding the endpoints.
  • 86.12% → 87.24% on ChartQA, 66.41% → 68.90% on Charxiv, and 82.22% → 82.71% on InfoVQA support extension beyond transit maps.The paper associates these gains with detail rewards and the multi-stage RL scheme.
Loading 2510.02240v2…