PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Cross-View Point Correspondence in Vision-Language Models

Yipu Wang, Yuheng Ji, Yuyang Liu, Enshen Zhou, Ziqiang Yang, Yuxuan Tian, Ziheng Qin, Yue Liu, Huajie Tan, Cheng Chi, Zhiyuan Ma, Daniel Dajun Zeng, Xiaolong Zheng

arXiv:2512.04686v2cs.CV

TL;DR

Vision-language models lack robust cross-view point correspondence, despite its importance for precise affordance interaction. The paper introduces a hierarchical benchmark and an affordance-focused dataset, then trains CroPond on that data. CroPond achieves 76.8% accuracy on CrossPoint-Bench, while evaluation highlights a substantial gap between model and human point prediction.

  • Problem

    Existing VLMs lack robust cross-view correspondence, and prior evaluations do not systematically test continuous point localization on interaction-relevant affordances or robustness to scale and occlusion.

    Source: §1 · p. 2

  • Method

    The paper defines CVPC, builds a hierarchical benchmark and an affordance-focused dataset, and trains CroPond using supervised fine-tuning with multiple data sources.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    CroPond achieves 76.8% accuracy on CrossPoint-Bench, while Gemini-2.5-Pro achieves 16.41% on point-level prediction compared with the human average of 93.63%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CroPond is presented as a strong baseline and a starting point for future research on cross-view correspondence.

    Source: §1 · p. 3

    CroPond is currently trained with a simple supervised fine-tuning scheme; exploring reinforcement learning and integrating CVPC with multi-agent task planning remain future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Cross-view correspondence is a fundamental capability for spatial understanding and embodied AI. However, it is still far from being realized in Vision-Language Models (VLMs), especially in achieving precise point-level correspondence, which is crucial for precise affordance interaction. So we propose the Cross-View Point Correspondence (CVPC) task and CrossPoint-Bench, a comprehensive benchmark with hierarchical design, inspired by the human cognitive process of "perceive", "reason", and "correspond". Our evaluation shows the state-of-the-art models (e.g., Gemini-2.5-Pro) still fall far behind humans, with a gap of over 54.65% in overall accuracy, exposing a challenge in transitioning from coarse-grained judgement to fine-grained coordinate prediction. To address this problem, we construct CrossPoint-378K, a dataset with 378K question-answering pairs across 900 scenes, focused on actionable affordance regions that better reflect real-world manipulation and interaction scenarios. Furthermore, we propose CroPond that trained on the CrossPoint-378K dataset. Our CroPond achieves state-of-the-art performance on CrossPoint-Bench, surpassing Gemini-2.5-Pro by 39.7% accuracy, which offers a foundation for advancing future work on cross-view correspondence. The benchmark, dataset, and model are publicly available at https://github.com/WangYipu2002/CrossPoint.

1. Introduction

The paper formalizes cross-view point correspondence and introduces a hierarchical benchmark to assess precise affordance-point prediction. It also presents an affordance-focused dataset and CroPond as a baseline for the task.

1. Introduction

  • 16.41% accuracy for Gemini-2.5-Pro on point-level prediction contrasts with 93.63% for humans, while models degrade from discrete region selection to continuous coordinates.The benchmark also finds weaker performance on fine-grained semantic parts than on general objects.Source: §1 · p. 2
  • CrossPoint-Bench evaluates grounding, visibility reasoning, correspondence judgement, and correspondence pointing across affordance regions and tests robustness to scale variation and occlusion.Its hierarchy follows the cross-view reasoning chain from single-view localization to correspondence.Source: §1 · p. 2
  • CrossPoint-378K contains approximately 378k question-answering pairs across 900 real-world indoor scenes, and CroPond achieves 76.8% accuracy on CrossPoint-Bench.The dataset focuses on affordance regions to target geometric consistency and fine-grained affordance.Source: §1 · p. 2
  • The contributions include defining CVPC, constructing an affordance-focused dataset, and developing CroPond as a starting point for future cross-view correspondence research.The dataset spans 162 interaction types and 8k object instances.Source: §1 · pp. 2–3

2. Related Work

Prior work advances single-view spatial understanding and several strands of multi-view reasoning, but the paper identifies precise multi-view point correspondence as an unresolved gap.

2.1. Spatial Understanding with VLMs

  • Single-view spatial understanding has progressed from region-level boxes or masks to point-level grounding that can support executable pointing or manipulation.Examples include RoboPoint for affordance points and RoboRefer for multi-step spatial reasoning.Source: §2.1 · p. 3

2.2. Multi-view Understanding for VLMs

  • Multi-view research spans geometry and consistency, 3D-aware semantic grounding, and multi-view cognition and action, yet precise-coordinate correspondence remains unaddressed systematically.The paper positions CVPC as a task to bridge this gap.Source: §2.2 · p. 3

3. Cross-View Point Correspondence

CVPC models cross-view point matching as a staged task: ground an instructed target, assess its visibility in another view, and then predict its corresponding point.

3.1. Problem Definition

  • Instruction-conditioned grounding localizes the point specified by a natural-language instruction in the first image view.This stage is described as the foundation of CVPC.Source: §3.1 · p. 3
  • Visibility reasoning determines whether the localized target is visible in a second view, requiring reasoning about occlusion and depth relations.Source: §3.1 · p. 3
  • When the target is visible, correspondence predicts its 2D location in the second view; the two view-specific points project the same 3D point.The example traces this process from locating a cup handle to mapping it across views.Source: §3.1 · p. 3

CrossPoint 378K

CrossPoint-Bench evaluates affordance-focused cross-view tasks, while CrossPoint-378K supplies diverse training data generated through an automated pipeline. CroPond is trained using supervised fine-tuning with multiple data sources.

CrossPoint 378K

  • CrossPoint-378K contains approximately 378k visual question-answer pairs across 900 indoor scenes, 162 interaction types, and 8k object instances.Source: §3.3 · p. 5

3.2. CrossPoint-Bench

  • CrossPoint-Bench uses 1k question-answer pairs from real indoor scenes and targets operable affordance regions at object and semantic-part granularities.Its four task types are fine-grained grounding, visibility reasoning, correspondence judgement, and correspondence pointing.Source: §3.2 · p. 4

3.3. CrossPoint-378K

  • The dataset emphasizes actionable affordance regions and spans single-view grounding and spatial understanding as well as cross-view visibility, depth, transformation, and correspondence reasoning.Its automated pipeline samples and filters images, segments affordance regions, pairs points across views, and generates question-answer samples.Source: §3.3 · pp. 4–5

3.4. Training Details

  • CroPond uses Qwen2.5-VL as its base model and is supervised-fine-tuned on CrossPoint-378K alongside single-view, multi-view, and instruction-tuned data.The joint training strategy is reported to improve cross-view point correspondence while retaining general multimodal knowledge.Source: §3.4 · p. 5
  • CroPond-7B excels across the four CrossPoint-Bench cases shown in the results figure.Source: Figure 4 · p. 5

4. Experiment

On CrossPoint-Bench, models struggle to move from coarse correspondence judgments to precise point prediction. CroPond improves benchmark performance and also shows results on long-range, spatial, and general benchmarks.

4. Experiment

  • Multiple-choice tasks use average accuracy, pointing tasks use in-mask hit rate, and comparisons include proprietary, open-source, and specialist models.In-mask hit rate tests whether a predicted 2D coordinate falls within the ground-truth mask.Source: §4 · p. 5

4.3. Results on CroPond

  • CroPond-7B reaches 76.8% overall accuracy and improves over CroPond-3B across all evaluation subsets.It also outperforms Qwen3-VL-235B-A22B-Thinking on Visibility Reasoning, Correspondence-Judgement, and Correspondence-Pointing.Source: Table 2 · p. 6, §4.3 · p. 7

4.1. Results on CrossPoint-Bench

  • Humans reach 91.75% overall accuracy versus 52.70% for the best model, while general VLMs average 12.78% on Correspondence-Pointing.General VLMs average 49.8% on Correspondence-Judgement versus 12.78% on Correspondence-Pointing; specialists average 22.56% overall, and general VLMs score 46.48% on general objects versus 36.47% on semantic parts.Source: Table 2 · p. 6, §4.1 · p. 6

4.2. Error Analysis

  • Spatial reconstruction failure dominates across models, while frame-transfer errors occur mainly in open-source models.The analysis also identifies semantic–point decoupling: models may infer a target’s meaning but fail to ground it precisely at the corresponding pixel.Source: Figure 5 · p. 6, §4.2 · p. 7

4.4. Results on CrossPoint-LR-Bench

  • CroPond-7B achieves 55.5% accuracy on the 90-sample, three-stage CrossPoint-LR-Bench, outperforming other models despite performance drops in multi-stage tasks.Trials require all three stages to be completed correctly.Source: Figure 6 · p. 7, §4.4 · p. 7

4.5. Results on Spatial and General Benchmarks

  • CroPond leads on spatial benchmarks, with CroPond-7B improving over Gemini-2.5-Pro by approximately 4.27% on the averaged score.CroPond-3B performs comparably to or slightly better than other models on general vision-language tasks.Source: §4.5 · pp. 7–8, Figure 6 · p. 7, Table 3 · p. 8

5. Real-world Applications

The paper evaluates CroPond in a multi-agent desk-organization scenario involving three toys.

5. Real-world Applications

  • The practical evaluation uses a multi-agent desk-organization task in which three toys must be organized.The passage introduces a master agent but provides no further task results.Source: §5 · p. 8

6. Conclusion

The paper defines CVPC, introduces CrossPoint-Bench, and trains CroPond on CrossPoint-378K, reporting strong benchmark and cross-benchmark performance. It identifies simple supervised fine-tuning as CroPond’s current training scheme.

6. Conclusion

  • The paper defines CVPC, constructs CrossPoint-378K across 900 indoor scenes, 162 interaction types, and 8k object instances, and reports CroPond’s strong benchmark performance and generalization.CrossPoint-Bench is described as a comprehensive hierarchical benchmark for CVPC.Source: §6 · pp. 8–9
  • CroPond is currently trained with simple supervised fine-tuning; reinforcement learning and integration with multi-agent task planning are future directions.The stated planning goal is collaborative robotics.Source: §6 · p. 9

Appendix

  • The supplement provides implementation details and additional resources for the dataset, benchmark, model, and evaluation.It also includes exploratory error analyses and visualizations.Source: Appendix · p. 14

A.1. Data Source Description

  • CrossPoint-Bench and CrossPoint-378K use ScanNet and ScanNet++, while CrossPoint-LR-Bench uses ETH3D and ARKitScenes.The passage describes these source datasets as indoor-scene resources, with ETH3D also covering outdoor scenes.Source: §A.1 · p. 14

A.2. Data Generation Pipeline

The data pipeline samples and filters images, generates affordance masks and cross-view pairs, and creates task-aligned question–answer samples. Separate procedures construct CrossPoint-Bench and CrossPoint-LR-Bench.

A.2. Data Generation Pipeline

  • Video frames are sampled at an average ratio of 1/50 and filtered for blur, low information, or interfering visual marks, yielding 23k curated images.Qwen2.5-VL-7B performs the filtering.Source: §A.2 · p. 14, Figure 10 · p. 15
  • Annotators manually verified all bounding boxes Brain2.0 extracted for semantic parts; these boxes served as auxiliary cues to SAM2.1, which produced 58k fine-grained mask annotations.Examples include masks for general objects and semantic parts.Source: §A.2 · p. 15, Figure 11 · p. 15
  • Geometric projection from ScanNet RGB–D frames and COLMAP reconstructions for ScanNet++ produces 111k one-to-one cross-view correspondence pairs.The pairs are generated by mapping sampled instance-mask points into other views.Source: §A.2 · p. 15
  • Templates populated with categories, actions, coordinates, and camera-derived judgments generate 378k question–answer samples aligned with task semantics.Cross-view pairs form the base units for multi-view tasks, while individual images underpin single-view tasks.Source: §A.2 · p. 15

A.3. CrossPoint-Bench Details

  • CrossPoint-Bench uses 100 held-out ScanNet and ScanNet++ scenes to create 1,000 manually verified, unambiguous samples.Its construction follows the CrossPoint-378K data-generation pipeline.Source: §A.3 · p. 15
  • CrossPoint-LR-Bench is manually constructed from ETH3D and ARKitScenes with 90 samples for extended spatial correspondence.The long-range benchmark is designed to assess reasoning under extended correspondence conditions.Source: §A.3 · p. 15

B.1. Architecture

CroPond combines a Qwen-2.5-VL architecture with multi-source training and supervised fine-tuning. CrossPoint-378K improves CrossPoint-Bench performance, while prompting alone yields limited gains and spatial-task performance is strongest on tasks aligned with CVPC.

B.1. Architecture

  • CroPond uses a Qwen-2.5-VL vision encoder, language model, and multimodal projector, with adjacent patch features grouped to shorten sequences while preserving spatial structure.The vision transformer uses patch size 14, 2D-RoPE, and dynamic image resolutions.Source: §B.1 · p. 15

B.2. Training Data

  • Training combines CrossPoint-378K supervision with RefSpatial, SAT, multi-view data, and LLaVA-1.5 instruction-tuned samples.The additional data targets spatial reasoning, multi-view understanding, instruction following, and general vision-language understanding.Source: §B.2 · p. 16

B.3. SFT Training Details

  • Supervised fine-tuning maximizes the likelihood of answers given image-text pairs containing an RGB image, prompt, and answer.The training setup updates all model parameters.Source: §B.3 · p. 16

C.1. Experiments Compute Resources

  • The 3B and 7B models required approximately 39 and 49 hours of training, respectively, on a server with eight NVIDIA H100 GPUs.The server also had two 48-core Intel Xeon Platinum 8468 CPUs.Source: §C.1 · p. 17

C.2. Effectiveness of CrossPoint-378K

  • 72.90% overall score after fine-tuning Qwen2.5-VL-3B on CrossPoint-378K, up from 25.40%.The reported gains span all metrics, with notable improvements in Correspondence-Judge and Correspondence-Pointing.Source: §C.2 · p. 17, Table 5 · p. 17

C.3. Effectiveness of Vanilla CoT

  • CoT prompting shows no consistent trend across models, and its overall gains on CrossPoint-Bench remain limited.Some subtasks show minor improvements, but the reported results emphasize the need for training data that enforces cross-view consistency.Source: §C.3 · p. 17, Table 9 · p. 17

C.4. Evaluation of CrossPoint-LR-Bench

  • CrossPoint-LR-Bench requires a model to answer all three question rounds correctly for a sample to count as correct.The rounds correspond to the three stages defined in the CVPC task.Source: §C.4 · pp. 17–18

C.5. Evaluation on Spatial VLM Benchmarks

  • CroPond generalizes to spatial benchmarks, performing strongly on tasks close to CVPC but not notably on SPAR-Bench Low-level tasks.The authors note that Low-level tasks involve precise metric estimation, which is not CroPond’s focus.Source: §C.5 · p. 18

C.6. Evaluation on General VLM Benchmarks

CroPond-3B retains comparable general vision-language benchmark performance while improving some tasks. The accompanying error analysis groups failures into three patterns and tests error-aware prompt suffixes.

C.6. Evaluation on General VLM Benchmarks

  • CroPond-3B achieves comparable performance across general vision-language tasks and surpasses its base model on examples including OK-VQA and MMBench dev.The authors report no compromise to general vision-language abilities alongside spatial supervision and multi-source visual instruction data.Source: §C.6 · pp. 18–19, Table 12 · p. 19

D. Exploratory Experiments on Error Patterns

  • The exploratory analysis categorizes CrossPoint-Bench errors as Frame Transfer Failure, Spatial Reconstruction Failure, or Semantic–Point Decoupling.The authors test prompt suffixes tailored to each error pattern.Source: D. Exploratory Experiments on Error… · p. 19

D.1. Experimental Setup

Error-aware suffixes modestly improve correct-case rates, with stronger effects on Semantic–Point Decoupling than on other error types. The analysis treats prompt response as diagnostic, while noting that persistent errors point to deeper representation limits.

D.1. Experimental Setup

  • The setup applies error-type-specific natural-language suffixes to mispredicted examples while keeping images and answer formats unchanged.Examples are manually labeled as one of three error types or No Error, then re-evaluated and categorized using the same annotation scheme.Source: §D.1 · p. 19

D.2. Quantitative Results

  • Correct-case rates rise from 34% to 43% for closed-source models, 42% to 61% for open-source models above 7B, and 28% to 37% for models at or below 7B.Frame Transfer Failure is nearly unchanged, while Spatial Reconstruction Failure declines only slightly across model groups.Source: §D.2 · p. 19, Table 13 · p. 19

D.3. Insights from Error-Aware Prompting

  • Prompting affects error patterns unevenly: Semantic–Point Decoupling is prompt-sensitive, Spatial Reconstruction Failure is only moderately reduced, and Frame Transfer Failure is essentially unchanged.The authors interpret the last pattern as difficult to address through prompt-level changes alone.Source: §D.3 · p. 20
  • Open-source models above 7B benefit most, while models at or below 7B retain more Spatial Reconstruction and Frame Transfer failures.The authors connect the larger models’ gains primarily to reductions in Semantic–Point Decoupling.Source: §D.3 · p. 20

E.1. Visualization of Prompts

  • The supplementary material provides the Gemini-2.5-Pro recognition prompt and CrossPoint-LR-Bench evaluation prompt.Source: §E.1 · p. 20

E.2. Visualization of CrossPoint-Bench

  • The supplementary examples compare CroPond with other models and show detailed multi-step reasoning for CrossPoint-Bench.Source: §E.2 · p. 20

E.3. Visualization of CrossPoint-378K

  • The CrossPoint-378K example figures span six categories across single-view and cross-view settings.Source: §E.3 · p. 20

F. Future Work

The paper presents CrossPoint-Bench and CroPond as a baseline for CVPC in realistic indoor scenes, and releases the benchmark, dataset, and model. Future directions include richer training and connecting cross-view perception to planning.

F. Future Work

  • CrossPoint-Bench and CroPond provide a comprehensive CVPC benchmark and a strong baseline designed for realistic indoor scenes.The authors release CrossPoint-Bench, CrossPoint-378K, and CroPond to support community-driven extensions.Source: F. Future Work · p. 20
  • Future training work may explore richer optimization strategies, including geometry-aware reinforcement-learning rewards, beyond CroPond’s current supervised fine-tuning.The proposed rewards may enable self-correction of inaccurate correspondences and long-horizon geometric consistency.Source: F. Future Work · p. 20
  • CroPond’s current multi-agent applications are limited to perception; a proposed direction is coupling CVPC with task-planning modules for coordinated interactions.Source: F. Future Work · p. 20
Loading 2512.04686v2…