Source-linked AI summary
MolmoPoint: Better Pointing for VLMs with Grounding Tokens
Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, Taira Anderson, Winson Han, Ranjay Krishna
TL;DR
VLMs commonly point by generating coordinates, requiring a complicated coordinate system and many output tokens. MolmoPoint instead selects visual tokens through coarse-to-fine grounding tokens and improves image, GUI, video, and tracking results while increasing sample efficiency. The method also supports ordered point generation and stopping through relative-position encoding and a no-more-points class.
Problem
Most VLMs generate text coordinates for pointing, requiring a complicated coordinate system and a high token count.
Method
MolmoPoint uses grounding tokens that cross-attend to image or video tokens, selecting a patch, subpatch, and location in sequence.
Results
MolmoPoint improves pointing across natural images, GUIs, video, and tracking, including 62.5 J&F versus 56.7 for Molmo2 on video tracking.
Takeaways & Limitations
Grounding tokens improve pointing performance, sample efficiency, and training speed across multiple domains.
Takeaways & Limitations
Without a no-more-points class, the model can be forced to select points and generate excessive numbers of them.
Abstract
from arXiv · showhide
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.
1 Introduction
MolmoPoint replaces text-coordinate pointing with grounding tokens that select visual content coarse-to-fine, while adding mechanisms for ordered, nonredundant point generation. Across image, GUI, video, and tracking tasks, the resulting models improve pointing performance and efficiency.
- Motivation and approach: Grounding tokens directly select visual tokens containing the target concept instead of generating text coordinates.The approach applies to both image and video inputs.
- Motivation and approach: Each point is generated coarse-to-fine through <PATCH>, <SUBPATCH>, and <LOCATION> tokens.The tokens select an image or video patch, a finer subpatch, and then a location within that subpatch.
- Motivation and approach: Relative-position encoding and a no-more-points class support consistent point ordering and prevent excessive point generation.The no-more-points class lets the model stop instead of being forced to select another visual token.
- Motivation and approach: Grounding tokens avoid learning a coordinate system, reduce output tokens, and improve learning speed, resolution generalization, and inference latency.The paper also reports stronger pointing performance and signs of transfer beyond grounding.
- Models and data: The authors train general-purpose, GUI-specialized, and video-specialized models, and introduce MolmoPoint-GUISyn and MolmoPoint-Track datasets.GUISyn targets high-resolution GUI grounding, while Track provides broader human-annotated and synthetic tracking coverage.
- Results: MolmoPoint-8B improves natural-image pointing by 2 points on PointBench and 4 points on PixMo-Points over previous methods.The model sets a new state of the art on both benchmarks.
- Results: MolmoPoint improves video pointing and tracking, reaching 62.5 J&F versus 56.7 for Molmo2 and showing gains from new data and model design.MolmoPoint-Vid-8B further improves video pointing metrics, while MolmoPoint-8B shows several-point counting gains and better human preference scores.
2 Related Work
Prior work grounds VLM outputs through coordinates, synthetic or human GUI data, specialized video models, and decoder-based grounding tokens. MolmoPoint instead uses lightweight visual-token selection with spatial refinement and extends the approach across images, videos, and GUIs.
- Generating Coordinates: Coordinate generation remains a common VLM grounding approach, supported by large pointing datasets such as PixMo-Points.MolmoPoint presents grounding tokens as a stronger and more efficient way to learn pointing.
- GUI Grounding: GUI grounding research improves performance through data generation, reinforcement learning, or agentic zooming and cropping strategies.Zooming and cropping can increase compute costs.
- GUI Grounding Datasets: MolmoPoint-GUISyn emphasizes high-resolution images, diverse interface conditions, and dense annotations averaging 54 points per image.The dataset spans operating systems, websites, software, apps, resolutions, and aspect ratios.
- Video Grounding: Video grounding is still generally handled by specialized models, with relatively few VLMs supporting it.MolmoPoint extends token referencing to video grounding.
- Grounding Tokens: Existing grounding-token methods often rely on pretrained decoders for segmentation or depth estimation.MolmoPoint instead decodes grounding tokens through lightweight projectors over hidden states.
- Grounding Tokens: PaDT also cross-attends generated tokens to visual tokens but uses a separate decoder for boxes or other grounding outputs.MolmoPoint uses visual-token spatial locations and additional refinement tokens, including for videos and GUIs.
- Grounding Tokens: GUI-Actor cross-attends a special token to visual patches but lacks refinement stages and targets GUIs and single points.MolmoPoint differs by supporting high-precision refinement and broader application domains.
3 Method
MolmoPoint grounds points by selecting visual tokens directly, then refines each selection from a coarse image patch to a subpatch and precise location. Sequential ordering, relative-position encoding, and a no-more-points class make multi-point generation more consistent and bounded.
- Core approach: Grounding tokens select visual tokens containing the target, replacing text-coordinate generation with direct visual-token selection.The method uses lightweight selection over visual representations rather than requiring the model to learn a coordinate system.
- Patch selection: The <PATCH> token scores image tokens using a query from its hidden state and keys generated from image-token hidden states.Training adds a selection loss, while inference chooses the highest-scoring image token and feeds its selected embedding back to the model.
- Location refinement: The <SUBPATCH> token selects one of the ViT patches pooled into the selected image token, exposing finer-grained features for refinement.This addresses the coarse spatial resolution of pooled image tokens.
- Location refinement: The <LOCATION> token selects one of 9 positions in a 3x3 grid within the chosen subpatch, yielding about 4.7-pixel precision with 14x14 ViT patches.The resulting resolution remains about 4 pixels regardless of input size, unlike text-coordinate pointing.
- Sequential pointing: Rotary embeddings encode candidate patches relative to the previously selected patch, helping enforce ordered points and support video-frame tracking.The relative-position encoding uses the prior selected image-token position, or 0 when no prior patch exists.
- Stopping criterion: A no-more-points class lets <PATCH> stop without selecting a visual token, preventing excessive point generation.When selected, the model is prevented from generating a <SUBPATCH> token and stops pointing.
4 Training and Inference
The authors train three MolmoPoint variants and construct specialized GUI and tracking data. Training and inference use grounding-token supervision, constrained point generation, and cached visual-token keys.
- Point supervision: Training maps each point to an image-token, ViT-patch, and location triple, replacing string coordinates with grounding tokens and a no-more-points marker.
- Inference: Inference caches image-token and ViT-patch keys, adding memory overhead roughly comparable to cached keys and values for 1–2 LLM layers.
- Inference: Constrained decoding orders selected image tokens, prevents repeated patch selections, and terminates point lists after the no-more-points class.
- GUI data: MolmoPoint-GUISyn generates HTML screenshots, extracts UI-element bounding boxes, and uses language models to annotate interaction intents.
- Tracking data: MolmoPoint-Track combines human-annotated arbitrary-object tracks with synthetic tracks covering diverse motion and occlusion patterns.
5 Results
MolmoPoint improves pointing across natural images, GUIs, video, and tracking, while ablations and qualitative comparisons show benefits from grounding tokens and associated design choices. The method also learns pointing more efficiently, though point ordering creates a video trade-off.
- Image Pointing: MolmoPoint-8B surpasses Molmo2 by almost 2 points on PointBench and by 4 points on PixMo-Points using the same data and training procedure.
- GUI Pointing: MolmoPoint-GUI-8B achieves state-of-the-art among fully open models on all GUI tasks after fine-tuning on instruction-image data.
- GUI Pointing: The GUI model exceeds a text-coordinate baseline by 2 to 9 points on ScreenSpotPro, with the authors hypothesizing stronger effects at high resolution.
- Video Pointing: MolmoPoint-8B wins 59.1% of non-tied human video-pointing comparisons against Molmo2, while MolmoPoint-Vid-8B provides more consistent gains.
- Tracking: MolmoPoint-8B improves Molmo2-Track by +5.7 J &F, +3.1 F1, and +2.5 HOTA across video domains, except on semantically reasoning-heavy ReasonVOS.
- Ablations: Removing grounding tokens drops tracking quality by 4.6 F1 and 4.0 HOTA overall, while also removing MolmoPoint-Track causes further losses.
- Ablations: Removing the no-more-points token more than doubles overcounting, while randomized point order significantly reduces video performance but improves PointBench.
- Sample efficiency: With 8192 pointing examples, MolmoPoint gains 20 points over the captioning baseline, indicating more efficient learning than text coordinates.
6 Conclusion
The conclusion reports that grounding tokens improve pointing across multiple domains. It also highlights sample-efficiency and training-speed gains, especially for low-resource settings, while identifying extensions to other modalities as future work.
- Grounding tokens significantly improve pointing across multiple domains.
- Improved sample efficiency and training speed suggest particular value in low-resource settings.
- Future work could extend grounding-token pointing to text and audio modalities.
Appendix
The appendix organizes supporting material into training details, video-pointing human evaluation, multi-task results, specialized GUI data, and tracking-data details.
- Section 7 covers training details.
- Section 8 covers the VideoPoint human evaluation, while Section 9 presents multi-task results.
- Sections 10 and 11 provide details on MolmoPoint-GUISyn and MolmoPoint-Track.
7 Training Details
Training largely follows Molmo2 but changes packing, optimization, batch configurations, and specialized-model training to improve efficiency and exploit faster grounding-token learning.
- Training pipeline: The training pipeline largely follows Molmo2, with changes for hardware, efficiency, faster learning, and tracking robustness.
- Pre-training: Pre-training permits up to 16 images per input sequence and reduces training steps from 32k to 23k while keeping examples seen unchanged.
- SFT: SFT uses adjusted mixture rates, 22k steps, and batch size 160, reducing examples seen by about 8.3%.
- Long-context SFT: Long-context SFT uses batch size 160 with up to 384 frames and 16384 tokens per training example.
- Specialized models: Specialized GUI and video models receive distinct training schedules on MolmoPoint-GUISyn and Molmo2 video-pointing data.
- Infrastructure: Training uses B200 GPUs with PyTorch, FSDP 2, and AMP, without sequence parallelism because long-context models fit with FSDP.
8 VideoPoint Human Evaluation
The VideoPoint human evaluation compares MolmoPoint-8B with a text-coordinate baseline on a manually assembled set of 470 challenging video-query examples using pairwise preferences.
- Setup: The evaluation compares video-pointing outputs from MolmoPoint-8B and a text-coordinate baseline.
- Annotation: Two annotators collected pairwise preferences through a dedicated interface following standard practice.
- Dataset: The test set contains 470 examples, combining 271 challenging Molmo2-VideoCountEval pairs with 199 newly written queries.
9 Multi-Task Results
MolmoPoint shows mixed transfer beyond pointing: it improves several high-resolution and OCR-heavy image tasks but produces mixed or similar video understanding results and lower video captioning performance.
- Video results: Video QA results are mixed across long- and short-video benchmarks, with no clear overall pattern and similar performance to Molmo2.MolmoPoint-8B gains on MLVU, Video-MME-SUB, and TempCompass but drops on VideoEvalPro and Tomato.
- Image results: Image results improve on several high-resolution and OCR tasks but decline slightly on counting benchmarks.Gains include InfoQa (+2.9), Ai2D (+0.8), and ChartQA (+0.8), while CountBench (-0.8) and PixMoCount (-0.9) decrease.
- Image results: The authors suggest grounding tokens improve transfer from pointing data to high-resolution OCR-heavy tasks.They connect this pattern to similar improvements observed with GUI pointing.
- Captioning: 54.47 for MolmoPoint-8B versus 53.62 for Molmo2 on image captioning contrasts with a significant drop in video captioning.The comparison uses the F1 captioning metric from [22].
10 MolmoPoint-GUISyn Details
MolmoPoint-GUISyn generates diverse synthetic GUI screenshots, extracts element geometry from their HTML, and converts each element into natural-language grounding instructions.
- Query and screenshot coverage: MolmoPoint-GUISyn spans desktop, web, and mobile screenshots across platforms, resolutions, aspect ratios, domains, and task stages.Queries are randomly combined from these fields to cover varied digital-world scenarios.
- Screenshot generation: An LLM converts natural-language screenshot queries into HTML code that is rendered into screenshots.The pipeline begins with queries such as “a screenshot of AutoCAD” paired with randomly selected personas.
- Element annotation: Custom JavaScript extracts each visible element’s bounding box, label, source-code lines, center, and dimensions.The pipeline uses underlying HTML access to recover geometric annotations automatically.
- Element annotation: The annotation LLM assigns each element a natural-language name and five user intents for interacting with it.These annotations provide pointing instructions grounded in the generated GUI elements.
- Dataset characteristics: The dataset costs about $0.2 per example and averages 54 pointing annotations per example.Figure 8 shows qualitative examples spanning desktop, web, and mobile screenshots.
11 MolmoPoint-Track Details
MolmoPoint-Track combines human-annotated and synthetic video tracks to support language-conditioned multi-object tracking across varied motion, camera, and visibility conditions.
- Data sources: MolmoPoint-Track combines TrackAny human annotations with TrackSyn synthetic tracks containing complex occlusion and motion dynamics.The two sources complement one another by covering broad real-video categories and controllable synthetic scenarios.
- Human annotation: The human-in-the-loop pipeline simplifies annotation by having annotators track one designated object while viewing other points for context.This reduces the need to identify and track multiple objects jointly and yields 13K videos with 17K text queries.
- Synthetic tracking: TrackSyn renders videos with static or moving cameras, configurable rendering settings, object trajectories, and per-instance binary masks.Camera-aware visibility constraints control whether objects remain visible or move off-screen over selected time spans.
- Synthetic tracking: Filtered 3D assets are sampled into scenes with 1–3 object categories, randomized lighting and camera parameters, and waypoint-based motion.GPT-based caption filtering retains independently trackable objects and assigns noun-only semantic categories.
- Query generation: GPT generates concise group-level referring queries, which are matched to object identities and stored with frame-wise tracking annotations.The resulting examples are language-conditioned multi-object tracking data.
- Dataset scale: The final training set contains 76k unique queries across 25k videos, averaging 3.3 unique objects per video.Point tracks are obtained from the center of the largest connected component in each segmentation mask.