Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching

Hao Zhong; Muzhi Zhu; Shenyan Zeng; Anzhou Li; Cong Chen; Hua Geng; Duochao Shi; Wentao Ye; Tao Lin; Hao Chen; Chunhua Shen

Source-linked AI summary

Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching

Hao Zhong, Muzhi Zhu, Shenyan Zeng, Anzhou Li, Cong Chen, Hua Geng, Duochao Shi, Wentao Ye, Tao Lin, Hao Chen, Chunhua Shen

arXiv:2606.03577v1Jun 2, 2026cs.CV

TL;DR

MLLMs lack unified, scalable, and verifiable ways to evaluate and train cross-view spatial reasoning, especially for fine-grained wide-baseline correspondence. The paper introduces ReasonMatch-Bench, a scalable video-3D data pipeline, and DCRL, substantially improving matching performance while transferring to related spatial benchmarks and preserving general visual understanding.

Problem
MLLMs lack unified, scalable, and verifiable evaluation and training frameworks for cross-view spatial reasoning involving geometry, viewpoint changes, fine-grained perception, and occlusion.
Method
The paper introduces ReasonMatch-Bench, a video-3D data pipeline, and DCRL, which uses dual-level curricula and verifiable rewards to train wide-baseline matching.
Results
70.5% F1: DCRL substantially improves ReasonMatch-Bench, transfers to related spatial benchmarks, and maintains general visual understanding performance.
Takeaways & Limitations
Wide-baseline correspondence is a useful testbed for cross-view spatial reasoning in MLLMs, while DCRL provides positive transfer beyond the training task.
Takeaways & Limitations
The work focuses on pairwise matching, leaving multi-view reasoning with 3D scene understanding, temporal dynamics, and semantic knowledge for future research.

Abstract

from arXiv · show

Wide-baseline matching (WBM) requires integrating geometric understanding, viewpoint changes, fine-grained perception, and occlusion reasoning, making it a challenging testbed for spatial reasoning in multimodal large language models (MLLMs) deployed in physical environments. However, current MLLMs lack systematic evaluation and training frameworks for these capabilities. We introduce ReasonMatch-Bench, a benchmark stratified by viewpoint displacement and matching granularity across indoor, outdoor, and object-centric scenarios, and show that current MLLMs still struggle with fine-grained wide-baseline correspondence: on a difficult 90-sample subset, human annotators achieve 84.0 F1, while the best existing baseline reaches 37.2. To bridge this gap, we build a scalable data-generation pipeline that automatically extracts wide-baseline view pairs from large-scale video-3D corpora, including RGB-D videos and SfM reconstructions, yielding diverse and verifiable supervision. We further propose Dynamic Correspondence Reinforcement Learning (DCRL), which combines Image-Level Viewpoint Progression and Point-Level Correspondence Curriculum to improve WBM training through verifiable rewards without explicit CoT supervision. Extensive experiments show that DCRL substantially improves ReasonMatch-Bench and transfers to related spatial benchmarks, while maintaining general visual understanding performance with modest gains on several benchmarks.

1. Introduction

The introduction frames wide-baseline matching as a demanding, under-evaluated test of spatial reasoning in MLLMs and presents ReasonMatch-Bench, scalable verifiable data generation, and DCRL as a unified response. DCRL uses dual-level curricula and verifiable rewards to improve matching, transfer to related spatial benchmarks, and preserve general visual understanding.

Motivation: Physical-world MLLM deployment requires spatial reasoning across viewpoints, including geometry, viewpoint imagination, fine-grained perception, occlusion, topology, scale, and depth.The paper identifies unified, scalable, and verifiable training and evaluation for these capabilities as an open problem.
Motivation: Existing supervision is costly and brittle because manual annotation misses combined geometry, semantics, and context, while synthetic data lacks real-world diversity and scalable verification.The paper motivates leveraging large-scale video-3D data to test and improve spatial reasoning with minimal human effort.
Benchmark: ReasonMatch-Bench evaluates wide-baseline matching across indoor, outdoor, and object-centric scenarios, stratifying difficulty by viewpoint change magnitude and matching granularity.On a difficult 90-sample human-study subset, human annotators achieve 84.0 F1.
Data generation: The data-generation pipeline automatically harvests wide-baseline view pairs from RGB-D videos and Structure-from-Motion reconstructions, producing diverse, verifiable supervision.The supervision spans scenarios and matching granularities.
DCRL: Dynamic Correspondence Reinforcement Learning combines Image-Level Viewpoint Progression and Point-Level Correspondence Curriculum with matching-accuracy rewards, improving spatial reasoning without explicit reasoning supervision.The framework uses reinforcement learning with verifiable rewards to progressively train complex correspondence capabilities.
DCRL: DCRL improves ReasonMatch-Bench, transfers to related spatial benchmarks, and does not degrade general visual understanding.These outcomes are stated as core contributions of the framework.

2. Related Work

Prior work benchmarks spatial reasoning in MLLMs but often isolates capabilities, while correspondence methods remain limited in viewpoint range, task format, and training approach. This work instead uses wide-baseline matching and verifiable geometric constraints to elicit complex spatial reasoning through reinforcement learning.

Spatial Reasoning in MLLMs: Existing benchmarks, including OmniSpatial and VSI-Bench, assess spatial understanding, but individual samples typically probe isolated capabilities.
Spatial Reasoning in MLLMs: Multi-SpatialMLLM explores correspondence matching but is limited to small viewpoint changes, restricted formats such as multiple-choice, and supervised fine-tuning alone.
Spatial Reasoning in MLLMs: Wide-baseline matching provides a challenging task that naturally demands complex spatial reasoning.
Spatial Reasoning in MLLMs: Reinforcement learning leverages matching tasks' inherent verifiability through geometric constraints to enable MLLMs to autonomously explore and acquire complex spatial reasoning capabilities.

3. Method

The method formulates wide-baseline matching as symbolic, partial correspondence prediction between two viewpoint-shifted images, then builds verified training data from RGB-D and SfM sources. Dynamic Correspondence Reinforcement Learning combines verifiable rewards with image- and point-level curricula to progressively improve spatial reasoning under difficult viewpoint changes.

Problem Definition: WBM is formulated as predicting cross-view correspondences between points depicting the same 3D scene, with camera geometry enabling relative pose recovery from accurate matches.The formulation uses image projections under intrinsic and extrinsic camera parameters and targets correspondence pairs representing the same 3D point.
Text-driven correspondence reasoning: The MLLM outputs a textual mapping that assigns each query point a counterpart or ∅, treating matching as partial bipartite association under occlusion or limited overlap.This symbolic formulation allows geometric, semantic, and contextual cues to contribute beyond continuous feature similarity.
Data Generation: The data pipeline combines RGB-D datasets and RGB videos with SfM reconstructions, extracting verified correspondences through reprojection consistency or shared, geometrically verified 3D landmarks.The resulting dense correspondence sets contain thousands of matches per image pair before preprocessing.
Data Generation: Viewpoint difficulty is stratified using overlap ω, with viewpoint-change magnitude defined as ∆v = 1−ω, increasing with baseline distance and occlusion.For RGB-D pairs, ω measures successfully matched pixels; for SfM pairs, it reflects shared 3D landmarks.
Dynamic Correspondence Reinforcement Learning: DCRL uses holistic matching rewards and a two-dimensional curriculum that progressively increases viewpoint divergence while adapting correspondence cardinality and spatial distribution.The reward scores all query regions, including unmatched points, and its matching component controls dynamic task-difficulty adaptation.

4. Experiments

Experiments show that DCRL improves difficult wide-baseline correspondence, transfers to related spatial benchmarks, and preserves general visual understanding performance. Ablations indicate that verifiable-reward reinforcement learning and progressive curriculum design outperform supervised imitation and fixed-difficulty training.

ReasonMatch-Bench Results: DCRL achieves the strongest performance overall, with especially robust gains on difficult indoor and instance-level matching configurations.Outdoor scenes are generally easiest, whereas instance-level matching is most challenging and baseline performance degrades sharply on L3 tasks.
Human Comparison: 84.0 F1 is achieved by humans versus 52.0 for DCRL on 90 high-divergence samples, leaving a substantial correspondence gap.On object-centric uCO3D, the gap is 62.1 vs. 27.8.
Failure Analysis: Locally accurate descriptions can still fail to identify targets uniquely, causing ambiguous appearance matching instead of geometric correspondence reasoning.This failure arises when multiple scene regions share similar local properties and the model does not exploit holistic 3D relationships.
Transfer Evaluation: 9.6% and 8.38% are the largest reported gains on OmniSpatial’s Dynamic Reasoning and Complex Logic categories, respectively.DCRL improves over the base model across the reported spatial benchmarks, indicating transfer beyond the specific matching task.
Training Objective Comparison: DCRL outperforms SFT by +19.5 on ReasonMatch and +34.0 on SAT, while improving all reported spatial benchmarks.The comparison suggests that teacher-forced imitation transfers inconsistently, whereas verifiable-reward reinforcement learning develops more transferable spatial reasoning.
Curriculum Ablation: +5.2 points is achieved by the dynamic curriculum over uniform sampling, with stable convergence during DCRL training.Uniformly sampled RL already outperforms easy-only and hard-only subsets.

5. Conclusion

The paper introduces ReasonMatch-Bench, a scalable video-3D data pipeline, and DCRL for verifiable-reward training of wide-baseline spatial correspondence in MLLMs. Current models remain substantially below human performance on a difficult 90-sample subset.

Contributions: ReasonMatch-Bench evaluates wide-baseline spatial correspondence in MLLMs.The paper presents it alongside a scalable video-3D data pipeline and DCRL.
Contributions: The scalable video-3D pipeline and DCRL framework support training on wide-baseline correspondence with verifiable rewards.DCRL is described as a reinforcement-learning framework for this task.
Performance: 84.0 human F1 versus 52.0 for the best model on the difficult 90-sample subset demonstrates a substantial performance gap.This comparison spans open- and closed-source baselines.

Supplementary Material · 6. Appendix Overview

The supplementary material provides comprehensive supporting details, covering implementation, ablations, benchmark analysis, and extended discussion of limitations and future directions.

6. Appendix Overview: The supplementary material provides comprehensive details supporting the main paper.It is organized into implementation details, additional ablations, benchmark analysis, and extended discussion.
6. Appendix Overview: Section 7 specifies the data-generation pipeline, experimental setup, prompt template, and curriculum progression schedules.
6. Appendix Overview: Together, the appendix sections document how the study generates data, conducts experiments, evaluates models, and examines remaining challenges.
6. Appendix Overview: Section 8 analyzes curriculum design variants, overlap scheduling strategies, and RL versus SFT across training stages.
6. Appendix Overview: Section 9 reports benchmark statistics, per-category model performance, failure modes, and qualitative examples of capabilities and limitations.
6. Appendix Overview: Section 10 expands the discussion of limitations and concrete future research directions for advancing MLLM spatial intelligence.

7. Implementation Details

The implementation generates verifiable wide-baseline correspondences from RGB-D videos or SfM reconstructions, constructs spatially controlled matching queries, and trains with format and matching rewards. A hierarchical curriculum increases task and spatial difficulty while structured prompts standardize model outputs.

Correspondence generation: RGB-D generation back-projects valid-depth pixels into 3D and reprojects them into a consecutive image, filtering correspondences with visibility, boundedness, depth, and photometric criteria.Optional dynamic-object and invalid-depth masks can be applied to either image.
Correspondence generation: For datasets without ground-truth depth, the pipeline uses SfM reconstructions, directly consuming COLMAP models for DL3DV and reconstructing RealEstate10k from RGB video streams.SfM pair overlap is defined from shared visible 3D landmarks, and viewpoint-change magnitude increases monotonically with viewpoint disparity.
Query construction: Queries select spatially diverse core matches through stage-specific clustering and pruning, optionally add non-overlapping distractors, shuffle labels, and package a ground-truth mapping.The default policy applies DBSCAN in joint space and retains one representative per cluster.
Verifiable reward design: The verifiable reward combines format compliance with matching correctness, evaluating exact agreements across all query regions, including correctly predicted unmatched regions.High rewards require both structurally valid outputs and geometrically accurate mappings.
Curriculum design: The curriculum combines three cardinality settings with three progressively difficult training stages, while independent spatial refinement moves each setting from clustered sampling toward dispersed points.Adaptive demotion temporarily returns training to the previous stage when matching performance degrades.

8. Ablation Studies

The ablations compare DCRL’s curriculum-based reinforcement learning with supervised fine-tuning using the same cross-view matching data. DCRL preserves general vision-language capabilities while substantially improving spatial and geometric reasoning, especially fine-grained correspondence.

Training strategy: DCRL uses reinforcement learning with holistic geometric rewards, whereas SFT uses 300 teacher-forced supervised training steps on the same cross-view matching data.This isolates training strategy as the main ablation variable.
General vision-language understanding: +2.7 on MMStar and +1.1 on V* are achieved by DCRL, while SFT degrades by −3.4 points and −2.6 points, respectively.DCRL modestly improves general vision-language performance over both the base model and SFT baseline.
Spatial intelligence and geometric reasoning: 70.5% on ReasonMatch versus 51.0% for SFT gives DCRL a 19.5-point improvement on the cross-view matching benchmark.The advantage is especially relevant to fine-grained geometric correspondence.
Spatial intelligence and geometric reasoning: +34.0 points on SAT accompanies DCRL’s improvement from 70.0 to 75.3, while SFT drops from 70.0 to 41.3.This larger contrast further favors reinforcement learning over supervised fine-tuning for spatial reasoning.
Training dynamics: SFT may form rigid correspondence associations that generalize poorly, whereas RL explores diverse matching strategies under holistic geometric feedback.The proposed explanation is that exploration and dynamic curriculum help preserve prior capabilities while improving the target task.

Matching Prompt

The matching prompt asks models to identify correspondences between annotated regions in two images of the same physical scene, allowing unmatched regions and requiring tagged reasoning and final answers. The section also reports that DCRL’s dynamic curriculum improves spatial reasoning transfer and wide-baseline matching performance over alternative training strategies.

Task specification: The task presents two images of the same physical scene with circularly annotated regions and IDs, requiring correspondence identification even when some regions are unmatched.Responses must contain separate thinking-process and final-answer sections wrapped in special tags.
Transfer results: +43.0 on ReasonMatch, +5.3 on SAT, +5.3 on OmniSpatial, and +3.5 on MindCube are the gains achieved by DCRL over the base model.SFT instead improves ReasonMatch (+23.5) and MindCube (+5.1), is roughly flat on OmniSpatial (−1.0), and degrades on SAT (−28.7).
Curriculum ablation: 70.5% F1 is achieved by dynamic curriculum learning, outperforming uniform sampling at 65.3% by +5.2 points.The curriculum progressively increases viewpoint divergence, correspondence complexity, and spatial-distribution challenges.
Curriculum ablation: 59.9% for easy-only training and 62.3% for hard-only training both fall below uniform sampling, by 5.4 and 3.0 points respectively.Easy-only training limits exposure to small viewpoint changes and simple correspondences, whereas hard-only training begins with extreme divergence and complex correspondences.
Curriculum ablation: +10.6 over easy-only and +8.2 over hard-only quantify the curriculum’s advantage over fixed-difficulty training.The adaptive mechanism increases difficulty after sustained performance improvements across viewpoint span, correspondence complexity, and spatial distribution.

9. ReasonMatch-Bench Results

ReasonMatch-Bench exposes substantial weaknesses in current MLLMs’ fine-grained wide-baseline correspondence, with performance falling sharply on harder and object-centric cases. DCRL substantially improves an 8B model, but a large gap to human performance remains, while error analysis identifies global spatial alignment and reasoning-to-answer consistency as key bottlenecks.

Overall performance: 57.9 overall F1 is achieved by GPT-5-mini, while GPT-5-Chat and Qwen3-VL-235B reach 51.5 and 49.2, respectively.Other strong closed-source systems remain in the 33–43 F1 range.
Overall performance: 70.5% F1 makes the DCRL-trained 8B model the best performer, versus 27.5% for its baseline and 49.2% for Qwen3-VL-235B.The result suggests targeted benchmark training can outperform simply scaling model size in this setting.
Difficulty analysis: 68.2 to 47.0 and 75.8 to 51.4 are GPT-5-mini’s Indoor and Outdoor F1 drops from L1 to L3, while Object L3 is consistently hardest.Scores generally decline from L1 to L3 within each scenario, indicating that the difficulty annotation captures genuine changes in task hardness.
Error analysis: F1 and F2 show the strongest model differences, identifying globally consistent spatial alignment from local cues as the central bottleneck.F4 additionally exposes cases where models reason correctly but produce an incorrect structured answer.
Human comparison: 84.0% overall human F1 exceeds the best model’s 52.0% by 32 points, while humans reach 62.1% versus 27.8% on object-centric uCO3D scenes.Humans exceed 93% F1 on structured indoor and outdoor environments, but severe self-occlusion and nondistinctive surfaces remain difficult.

10. Limitations and Future Work

The approach improves spatial matching but still falls short of human performance, especially in object-centric scenes, and future work should extend pairwise matching to holistic multi-view reasoning.

Current limitations: 52.0% F1 for the best model versus 84.0% for untrained humans reveals a persistent gap in fine-grained geometric reasoning.The gap is most pronounced in object-centric scenes, where performance is 27.8% versus 62.1%.
Future work: Future research should move beyond pairwise cross-view matching toward multi-view reasoning that integrates correspondence, 3D scene understanding, temporal dynamics, and semantic knowledge.The goal is to synthesize information across viewpoints into coherent 3D mental models resembling human spatial cognition.