Source-linked AI summary
PointArena: Probing Multimodal Grounding Through Language-Guided Pointing
Long Cheng, Jiafei Duan, Yi Ru Wang, Haoquan Fang, Boyang Li, Yushan Huang, Elvis Wang, Ainaz Eftekhar, Jason Lee, Wentao Yuan, Rose Hendrix, Noah A. Smith, Fei Xia, Dieter Fox, Ranjay Krishna
TL;DR
Pointing benchmarks often focus on object localization, while language-guided pointing requires spatial outputs that identify image regions. PointArena combines a curated benchmark, human-preference arena, and robotic manipulation evaluation; Molmo-72B leads Point-Bench, with proprietary models performing comparably.
Problem
Pointing evaluation needs spatial outputs that identify image regions, beyond conventional text-only visual question answering.
Source: §1 · p. 2Method
PointArena combines Point-Bench for controlled localization, Point-Battle for blinded human-preference comparisons, and Point-Act for real-world robotic evaluation.
Source: §1 · pp. 2–3Results
Molmo-72B achieves the highest Point-Bench performance, proprietary models such as Gemini-2.5-Pro perform comparably, and explicit pointing supervision consistently improves performance.
Source: §1 · p. 3Takeaways & Limitations
PointArena is designed to support rigorous real-world evaluation of grounded spatial reasoning as the field evolves.
Source: §5 · p. 10Coarse, imprecise mask boundaries can degrade segmentation quality and add evaluation noise, while static Point-Bench faces increasing risk of training-data contamination.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts, with applications spanning robotics, assistive technologies, and interactive AI systems. While recent multimodal models have started to support pointing capabilities, existing benchmarks typically focus only on referential object localization tasks. We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. PointArena comprises three components: (1) Point-Bench, a curated dataset containing approximately 1,000 pointing tasks across five reasoning categories; (2) Point-Battle, an interactive, web-based arena facilitating blind, pairwise model comparisons, which has already gathered over 4,500 anonymized votes; and (3) Point-Act, a real-world robotic manipulation system allowing users to directly evaluate multimodal model pointing capabilities in practical settings. We conducted extensive evaluations of both state-of-the-art open-source and proprietary multimodal models. Results indicate that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance. Additionally, we find that supervised training specifically targeting pointing tasks significantly enhances model performance. Across our multi-stage evaluation pipeline, we also observe strong correlations, underscoring the critical role of precise pointing capabilities in enabling multimodal models to effectively bridge abstract reasoning with concrete, real-world actions. Project page: https://pointarena.github.io/
1 Introduction
Pointing provides a direct way to ground language in visual space, with potential uses ranging from robotics to assistive technology. PointArena evaluates this ability through a curated benchmark, human-preference comparisons, and real-world robotic manipulation.
1 Introduction
- Pointing offers a low-bandwidth spatial interface for grounding language in vision across applications including robotics, assistive technology, and interactive systems.The section connects precise spatial grounding with practical tasks such as manipulation and answering spatial queries.Source: §1 · pp. 1–2
- Existing pointing evaluations often prioritize pixel-level accuracy over conceptual reasoning, while object-localization datasets capture only part of the grounding needed for interactive or embodied applications.The introduction identifies limited diversity and scalability as further shortcomings of existing setups.Source: §1 · p. 2
- PointArena evaluates language-guided pointing with spatial outputs that identify image regions, including both unambiguous and ambiguous scenarios.The authors describe pointing as a high-precision signal that avoids reliance on object contours or dense annotations and is compatible with human evaluation.Source: §1 · p. 2
- PointArena combines Point-Bench, Point-Battle, and Point-Act to assess pointing through curated tasks, blind user comparisons, and real-world robotic manipulation.Point-Battle had collected over 4,500 votes from more than 100 participants.Source: §1 · pp. 2–3
- Molmo-72B achieves the highest Point-Bench performance, proprietary models such as Gemini-2.5-Pro perform comparably, and explicit pointing supervision consistently improves performance.The introduction also reports a strong correlation between static benchmark accuracy and human preference, while adding language reasoning does not improve pointing grounding.Source: §1 · p. 3
2 Related work
Related work spans visual grounding datasets, arena-style model comparisons, and multimodal systems that point or sketch. The authors position PointArena as combining ground-truth evaluation with human preference assessment while explicitly evaluating pointing behavior.
Grounding benchmarks
- Visual grounding datasets cover 2D and 3D localization, phrase-region grounding, and dialog-based object grounding, but generally do not explicitly evaluate pointing behavior.Examples include RefCOCOg for long-form referring expressions [2016b], ScanRefer for language-geometry grounding [2020], and GuessWhat?! for multi-turn dialog [2017].Source: §2 · p. 3
Arena style evaluation
- Arena-style evaluations scale pairwise user comparisons and align with preferences, but lack ground-truth supervision and can be vulnerable to adversarial influence.PointArena addresses this gap by pairing Point-Bench ground-truth evaluation with Point-Battle human evaluations.Source: §2 · p. 3
Models that point or sketch
- Prior multimodal systems support spatial grounding and robotic affordance prediction, while opportunities remain to improve localization precision and understand what drives pointing capability.Molmo predicts normalized coordinates directly from modality-specific embeddings; Robo-Point uses instruction-tuned models trained on synthetic data for robotic affordances.Source: §2 · pp. 3–4
3 PointArena
PointArena evaluates language-conditioned pointing from image-space localization through human preference and physical manipulation. Its stages combine a mask-based benchmark, live pairwise comparisons, and a robotic task setting.
3 PointArena
- PointArena combines controlled localization measurement, blinded human-preference comparisons, and real-world robotic execution to evaluate how models ground language in visual and physical space.The framework addresses the need for standardized evaluation across specialized pointing models and general-purpose multimodal models.Source: §3 · p. 4
3.1 Task formulation
- An MLLM receives an RGB image and language instruction, predicts image-coordinate points, and is evaluated against target masks; success requires the predicted point count to match and every target region to be covered.With ground-truth masks, evaluation can be automated without test-time human annotators.Source: §3.1 · p. 4
3.2 Point-Bench
- Point-Bench contains 982 image-query pairs with pixel-level masks across Spatial, Affordance, Counting, Steerable, and Reasoning tasks.Queries are category-guided and retained when no more than one of three anonymized models is judged correct; annotators create and refine masks, which are then manually verified.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
3.3 Point-Battle
- Point-Battle anonymously compares randomly selected model pairs on user-provided instructions and images, collecting over 4,500 votes from approximately 100 participants.Unlike the static benchmark, it is continuously updated to capture human preferences and track progress.Source: §3.3 · pp. 5–6
3.4 Point-Act
- Point-Act translates model-predicted points from natural-language instructions into pick-or-place commands executed by an xArm 6 Lite robot using depth sensing.The setup highlights that even small localization errors can cause execution failures, whereas accurate predictions enable consistent real-world success.Source: §3.4 · p. 6
4 Experiments
The experiments evaluate open-source and proprietary multimodal models through Point-Bench, Point-Battle, and Point-Act, using standardized pointing protocols. Results compare benchmark accuracy, human preferences, and robotic task performance, and examine supervision and prompting choices.
4.1 Evaluation Setup
- Evaluations used zero-shot prompting and standardized coordinate outputs; Point-Bench tested 16 models on the same 982 image-instruction pairs three times.Point-Battle used anonymous head-to-head voting and Elo ratings, while Point-Act recruited 10 participants to test three agents across three trials and complete SUS surveys.Source: §4.1 · p. 7
4.3 Results between three evaluation frameworks.
- Point-Battle scores correlated with Point-Bench results at R2 = 0.85, while Point-Act success rates correlated with Point-Bench scores at R2 = 0.92.The Point-Act comparison tested Molmo-7B-D, GPT-4o, and a human reference; Point-Battle is presented as a continuously updated arena alongside the static benchmark.Source: Figure 5 · p. 8, §4.3 · p. 9
4.2 Main Results
- Molmo-72B achieved the highest Point-Bench performance, while its 0.43-percentage-point lead over Gemini-2.5-Pro was statistically insignificant (p ≈0.29).Open-source models trained explicitly on pointing data often matched or outperformed proprietary models; increasing model size brought marginal gains, with Qwen2.5-VL-7B within 3% of Qwen2.5-VL-72B and Molmo-7B-O less than 1% from Molmo-72B.Source: Figure 3 · p. 6, §4.2 · pp. 8–9
- Within the Qwen family, adding PixMo data raised Qwen2.5-VL-7B performance to 52.3%, versus 17.4% for Qwen2-VL-7B without that data.LLaVA variants without explicit pointing supervision achieved 4.8–17.4% on average.Source: §4.2 · p. 8
- GPT-o3 improved by 21.1 percentage points over GPT-4-Turbo, and Gemini-2.5-Flash by 45.9 points over Gemini-1.5-Flash; the authors suggest recent proprietary models may use PixMo or similar data.Proprietary training data is opaque, so the suggested link to open-source pointing datasets is not established.Source: §4.2 · p. 8
- In Point-Battle, Molmo-7B-D beat Gemini-2.5-Flash by 196 Elo points and won 79% of 115 head-to-head comparisons; Molmo-7B-D also remained superior to Qwen2.5-VL-7B in preference-aligned pointing performance.Both models exceeded the 1000-point baseline, indicating a statistically significant advantage over random guessing.Source: §4.2 · p. 8, Figure 6 · p. 9
- On Point-Act, Molmo-7B-D outperformed GPT-4o by 65% and scored 60.3 points higher in SUS, approaching the human oracle baseline.The robot system evaluates language-guided pointing through real-world manipulation.Source: §4.2 · p. 8, Figure 7 · p. 9
4.4 What Other Factors Drive Pointing Performance?
- Adding Chain-of-Thought reduced pointing accuracy by 2.9% for GPT-4o and 16% for Gemini-2.5-Flash; raw user queries caused further drops of 2.6% and 3.7%, respectively.The authors conclude that clear, targeted prompts with well-defined coordinate systems are important, while additional language reasoning does not enhance pointing performance.Source: §4.4 · p. 10, Table 1 · p. 10
5 Limitations, Discussion, and Conclusions
PointArena combines static benchmarking with human preference comparisons, while the discussion identifies annotation, contamination, and comparison-efficiency constraints and proposes refinements.
Discussion
- The discussion proposes free-form contouring for more precise masks, user-generated Point-Battle content to keep Point-Bench current, and adaptive model-pair sampling for more informative comparisons.User-generated annotations are noisier than manually curated ones, while adaptive sampling is planned to target models with similar performance.Source: §5 · p. 10
Limitations
- Current limitations include coarse SAM-refined masks that add noise, static-benchmark contamination risk, and random Point-Battle pairings that can be uninformative when model performance gaps are large.The mask problem is especially pronounced for fine-grained or irregular shapes.Source: §5 · p. 10
Conclusion
- PointArena offers an extensible framework combining static evaluations and live user-driven comparisons for real-world evaluation of grounded spatial reasoning.Proposed improvements target scalability, annotation fidelity, and comparative signal.Source: §5 · p. 10