PaperlayerView exact revision on arXiv

Source-linked AI summary

Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, Ranjay Krishna

arXiv:2601.10611v4cs.CVcs.AI

TL;DR

Video-language models lack fully open training foundations and robust grounding, including pointing and tracking in videos. Molmo2 introduces fully open models, data, and code built around nine new datasets and an efficient training recipe, achieving strong open-model performance and leading video-grounding results while still using closed text-only LLMs for data generation.

  • Problem

    Current video-language models are often proprietary, and existing open data may rely on proprietary VLM distillation; video grounding is also limited even in proprietary systems.

  • Method

    Molmo2 combines nine new video and multi-image datasets, human-and-LLM data pipelines, and staged training with token weighting, packing, message trees, and bidirectional visual-token attention.

  • Results

    Molmo2 achieves strong or state-of-the-art performance across short-video understanding, counting, captioning, video grounding, image, and multi-image benchmarks.

  • Takeaways & Limitations

    Molmo2 provides an open foundation for video-language research with grounding capabilities spanning single-image, multi-image, and video tasks.

  • Takeaways & Limitations

    The data-generation pipeline uses closed text-only LLMs, reducing transparency, although it avoids closed VLMs.

Abstract

from arXiv · show

Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).

1 Introduction

Molmo2 targets the limited openness and grounding of current video-language models with fully open models, datasets, weights, and training code. Its data and training innovations support strong general video, image, multi-image, and video-grounding performance.

  • Current video-language models are often proprietary, while video grounding remains limited even in some proprietary systems.
  • Molmo2 supports grounding across single images, multiple images, and videos while releasing model weights, training data, and code.The data is constructed without distilling from proprietary models.
  • The nine-dataset suite targets open-vocabulary video pointing and tracking, dense captioning, long-form question answering, long-video QA, and multi-image understanding.The datasets include 520k pointing and tracking instances, 104k videos with dense captions, and around 1.3M long-video QA instances.
  • Molmo2 combines human-and-LLM data pipelines with three training stages spanning image pre-training, joint multimodal fine-tuning, and long-context training.The collection process includes human narration, transcription, enrichment, and synthetic question generation from captions and transcripts.
  • Training innovations include token weighting, sequence packing, message-tree scheduling, and bidirectional attention between visual tokens.
  • Molmo2 performs strongly across short-video understanding, counting, captioning, grounding, image, multi-image, and human-preference evaluations.It establishes new state-of-the-art results in video tracking and pointing and remains competitive on traditional image and multi-image benchmarks.

2 Data

Molmo2’s data collection combines human annotation, synthetic generation, and curation of open academic sources to cover detailed video understanding, grounding, and multi-image tasks. The resulting datasets emphasize diverse, fine-grained, and long-duration inputs.

  • The data collection comprises five human-annotated datasets, four synthetic datasets, and two curated datasets from existing open-source data.
  • Molmo2-Cap provides 104k video-level and 431k clip-level dense captions designed for high detail and broad diversity.Its pipeline combines spoken human descriptions, transcription, rewriting, frame-level captions, and caption merging.
  • Molmo2-AskModelAnything contains 140k human-authored video QA pairs covering fine-grained questions about text, actions, and temporal relations.Counting questions are removed because the model is intended to point for those queries.
  • Synthetic caption-based pipelines generate large-scale video QA from segmented scenes, video metadata, and subtitles.
  • Molmo2-VideoPoint contains over 650k pointing queries on 280k videos across objects, events, references, spatial relations, comparisons, and visual artifacts.
  • Additional datasets provide complex-query object tracking, converted pointing and tracking supervision, human multi-image QA, and synthetic multi-image pointing, counting, and document examples.The multi-image synthetic collection includes over 470k pointing and counting examples and 188k text-rich-image QA examples.

I T T

Molmo2 uses a vision-language architecture and an attention mask for packed multimodal sequences. The mask separates examples and QA branches while allowing frame tokens forward attention.

  • The packed-sequence mask blocks cross-attention between different examples and between distinct QA pairs within one example.
  • Frame tokens retain forward attention within the packed sequence.
  • Point annotations encode temporal and spatial coordinates alongside object identifiers for pointing, tracking, and counting.
  • Molmo2 connects a vision encoder and a language model to process video inputs.

3 Training

Molmo2’s training pipeline combines a vision-language model architecture with staged multimodal training, long-context adaptation, and efficiency-oriented sequence design. Token weighting addresses imbalance between short and long outputs, while packing and message trees improve utilization.

  • The model combines a pretrained LLM and vision transformer through a connector that produces visual tokens for the LLM.ViT patch features are pooled and projected before being interleaved with text inputs.
  • Video inputs are sampled at 2 fps with up to 128 frames during standard training and 384 frames during long-context training.
  • Visual tokens are interleaved with timestamps or image indices, and visual tokens can attend bidirectionally across frames or images.
  • Training proceeds through image-only pre-training, joint video/image supervised fine-tuning, and a short long-context fine-tuning stage.
  • The training mix includes dense captions, transcripts, language data, pointing data, Molmo2 datasets, and other open-source video and image datasets.
  • Long-context training increases sequence length to 36,864 tokens and frame capacity to 384, but adds substantial overhead.
  • Token weighting reduces the loss contribution of video captions and pointing examples to balance long and short output tasks.The fixed weights are 0.1 for video captions and 0.2 for pointing; other tasks use a 4√n heuristic.
  • Message-tree encoding and packing prevent branch cross-attention while fitting 3.8 examples into a 16,348-token sequence, yielding 15x training efficiency.

4 Evaluation

Molmo2 is evaluated across video understanding, captioning, counting, grounding, image, and multi-image tasks, with ablations probing data, modeling, sampling, and context choices. It performs strongly among open-data models, leads on several grounding and short-video capabilities, but long-context post-training improves long-video QA while reducing caption quality.

  • Overall results: Molmo2 ranks equal to or better than other open-weight models in human preference and far ahead of previous fully-open models.
  • Grounding results: Molmo2 scores highest on BURST-VC and Molmo2-VP, and second highest on Molmo2-VC close accuracy, slightly behind Gemini 2.5 Pro.
  • Grounding results: Molmo2 outperforms all baselines, including specialized segmentation models, across tracking benchmarks, particularly on ReasonVOS and Molmo2-Track.
  • Image results: Molmo2 generally outperforms previous open-data models, leads on VQA v2.0 and RealWorldQA, and is strongest on challenging PixMo-Count, but trails on OCR-heavy and multimodal reasoning benchmarks.
  • Ablations and specialized models: Long-context post-training significantly improves long-video QA, while video caption performance drops and short-video and image QA remain largely unchanged.

5 Related works

Video-language research has advanced through multimodal encoders, efficient video representations, synthetic instruction tuning, and increasingly open-vocabulary tracking and pointing. Molmo2 addresses gaps in openness, data provenance, and video grounding by building data without proprietary-model distillation.

  • Multimodal LLMs: Video LLMs commonly encode frames individually with image encoders, but visual-token growth makes longer or higher-frame-rate videos computationally expensive.Efficient video-encoding methods have therefore become increasingly important.
  • Open research: The strongest video LLMs remain proprietary, while many open models withhold training recipes or data, or use data generated by proprietary VLMs.This motivates a fully open video-LLM training pipeline without previously trained multimodal models generating the data.
  • Video-language instruction tuning datasets: Video instruction-tuning commonly segments videos, generates clip captions, and uses powerful language models to synthesize video-level captions and question-answer pairs.Molmo2-CapQA follows a related pipeline but uses a captioner trained on the fully open Molmo2-Cap dataset.
  • Video tracking: Existing tracking datasets often target closed object sets, while newer language-guided methods add referring expressions but lack prompts covering multiple objects and diverse actions.Molmo2-VideoTrack adds non-trivial text queries focused on multiple objects.
  • Video pointing: Video-pointing datasets extend human image-annotation pipelines by selecting relevant frames and generating complex, diverse queries from captions.This adapts point grounding to video inputs.

6 Conclusion

Molmo2 is presented as an open-science VLM supporting grounding across single images, multi-image inputs, and videos. The appendix documents its data formats, visual processing, implementation, and training details to support transparent reuse.

  • Conclusion: Molmo2 supports single-image, multi-image, and video grounding while releasing its training data, model weights, and training code.The work positions these releases as part of closing the gap between proprietary VLMs and the broader research community.
  • Model details: Video frames are sampled at a target rate and capped at a maximum frame count, with tracking videos and point tracks trimmed together to preserve alignment.The last frame is included, and uniform sampling is used when the frame cap would otherwise be exceeded.
  • Model details: Visual inputs are inserted before text, with frame timestamps or image labels interleaved to distinguish video and multi-image content.Point outputs encode image or frame identifiers together with object indices and normalized coordinates.
  • Implementation: The implementation uses FSDP 2, scaled dot-product attention, torch.compile, static shapes, and mixed-precision computation to improve throughput.Layer normalization and rotary positional embeddings remain in full precision.
  • Implementation: Per-device losses are normalized by the average number of loss tokens across devices, avoiding effective up-weighting of examples with short responses.This correction addresses a subtle gradient bias in distributed training.

B Training details

Molmo2 training combines packed multimodal examples, staged pre-training and fine-tuning, and task-specific data mixtures. The recipe balances throughput and heterogeneous task learning, while its practical bottleneck remains video frame loading and extraction.

  • Packing: Molmo2 training uses a 48-example preprocessing pool and dynamic programming to select packed sequences under token and crop constraints.The selected examples are emitted as one packed sequence; long-context training uses larger image and token limits.
  • Packing: Packing efficiency shows diminishing returns beyond M = 48, while overly small crop weights can fill the pool with examples that cannot be packed efficiently.Despite DataLoader overhead, packing does not slow training in practice.
  • SFT: Fine-tuning mixes diverse datasets across images, videos, and multi-image inputs, using human-written long-form questions directly and templates for grounding and other tasks.The full mixture is visualized in Figure 4 and listed in Table 13.
  • Specialized models: Specialized models receive shorter fine-tuning runs on task-specific subsets for QA, captioning, pointing, or tracking.The pointing and tracking variants retain a three-stage pipeline before their final task-specific fine-tuning.

C Evaluation Details

Evaluation covers captioning, human preference, counting, pointing, and tracking with task-specific protocols and metrics. Molmo2 shows especially strong grounding and tracking performance, while comparisons require caution because baseline evaluation details are sometimes unavailable.

  • Captioning: Captioning is evaluated on 693 diverse videos using an F1 score for caption accuracy and detail.The evaluation videos are drawn from creative-commons Vimeo and are disjoint from the mostly YouTube-based training data.
  • Human evaluation: Human preference evaluation uses 1,000 bootstrap rounds to estimate more stable Elo ratings and confidence intervals, supplemented by task-specific ratings and pairwise win rates.Molmo2-8B beats Qwen3-VL-8B in 53% of pairwise comparisons, while Molmo2-4B beats Qwen3-VL-4B in 51%.
  • Human evaluation: Molmo2 performs better than Qwen3-VL on open-ended QA but underperforms Qwen3-VL and GLM-4.1V on captioning.These task-specific findings qualify the broader human-preference comparison.
  • Counting and pointing: Video pointing uses 2 fps inputs capped at 384 frames and reports recall, precision, F1, and valid accuracy.Molmo2-4B and Molmo2-8B achieve 39.9 and 38.4 F1, compared with 20.0 for Gemini 3 Pro and 1.5 for Qwen3-VL-8B.
  • Tracking: Molmo2 substantially outperforms API-based and open-source VLMs, as well as specialized segmentation models, across academic tracking benchmarks.The results indicate that these baselines are poorly suited to object tracking and semantic grounding.
  • Tracking: On Molmo2-Track, Molmo2 leads other VLMs and specialized open video models, with point-based F1 and identity-aware HOTA measuring precise object grounding and tracking.The benchmark includes diverse domains, complex object movements, and occlusions.

D.2 NLP Benchmarks

Molmo2 is evaluated on selective NLP benchmarks spanning general knowledge, math, reasoning, and coding. It is comparable on MMLU and GSM8K but shows coding drops on MBPP+ relative to base Qwen3 models.

  • Molmo2 is evaluated on MMLU, GSM8K, ARC-C, and MBPP+ for general knowledge, math, reasoning, and coding.The comparison uses base Qwen3 language models and follows OLMo 3’s evaluation protocol.
  • Molmo2 achieves comparable numbers on MMLU and GSM8K but suffers drops on the MBPP+ coding benchmark.

E Test time scaling with 128-frame model

The section examines increasing frame access while controlling visual-token cost for long-video understanding. Query-guided SlowFast selection reaches performance close to 224-frame inference with substantially fewer visual tokens, while some alternatives trade off short-video performance.

  • Increasing max frames: 224 frames gives Molmo2’s best average performance on six long-video benchmarks.The model is evaluated with maximum frame counts from 160 through 512.
  • Keeping Vision tokens fixed: The default 128-frame setting feeds approximately 10.6k maximum vision tokens, motivating test-time strategies that keep token counts near this budget.
  • Keeping Vision tokens fixed: 5 × 5 pooling improves long-video understanding by accessing more frames, but both tested pooling settings regress on short-video understanding.
  • SlowFast encoding: SlowFast encoding assigns default 3 × 3 pooling to slow frames and 9 × 9 pooling to fast frames while keeping the maximum token count approximately fixed.Periodicity controls which frames enter the slow pathway and supports maximum frame counts up to 368.
  • SlowFast encoding: Query-guided slow-pathway selection boosts long-video understanding, nearly matches 224-frame inference, and uses approximately 43% fewer visual tokens with minor short-video regression.

F.3 Data annoation

Molmo2’s annotation work combines human collection, synthetic generation, and academic dataset conversion to build diverse pointing, tracking, and video-question-answering resources. Tracking data emphasizes multi-object queries and integrates segmentation and bounding-box sources.

  • Captioning and QA: The captioning pipeline splits videos into information-density-based clips, collects spoken descriptions, and combines clip-level and frame-level captions into long captions.
  • Molmo2-VideoPoint: Molmo2-VideoPoint collects visual questions requiring evidence from one or several frames, with workers recording screenshots and object points.
  • Molmo2-VideoTrack: Molmo2-VideoTrack expands tracking coverage using existing human-annotated tracks because off-the-shelf trackers produced incomplete or unstable trajectories.
  • Molmo2-VideoTrack: The tracking pool combines VOS videos with bounding-box datasets spanning occlusion-heavy scenarios and diverse domains such as pedestrians, animals, and UAV footage.
  • Tracking data processing: Bounding-box tracks are converted into segmentation tasks, with tracks filtered when predicted masks have average IoU below 0.5.
  • Academic data: Academic-VideoTrack aggregates academic segmentation and bounding-box tracking datasets, producing complementary coverage for general and domain-specific tracking.
  • Captioning and QA: Molmo2-AskModelAnything gathers non-trivial visual-reasoning questions while discouraging vague, subjective, unverifiable, or simple-counting prompts.

H Limitations

The authors identify limitations in model openness, video-grounding consistency, long-video support, point tracking, and long-caption generation. These boundaries reflect data, sampling, representation, and joint-task issues.

  • Model openness: Molmo2 still relies on the closed-data SigLIP 2 image encoder because no competitive open-data encoder was available.
  • Data generation: Closed text-only LLMs are used for data generation, reducing transparency even though the pipeline avoids closed VLMs.
  • Video grounding outputs: Video pointing and tracking can produce repeated or degenerate points, especially for high-frequency objects and long videos.The authors hypothesize that joint-training interference may contribute because specialized models show the issue less often.
  • Video grounding: Video grounding remains less consistent than image grounding, with no tested model exceeding 40% on the authors’ counting or pointing metrics.Image grounding models often achieve 70–90% on PointBench metrics.
  • Long video grounding: Long-video grounding is limited to videos around three minutes because annotations are collected at 2 fps and can misalign when sampling below 2 fps.
  • Point tracking: Generated tracking points may shift within the target object because the data pipeline does not always enforce consistent point placement across frames.
  • Captioning: Very long video captions can repeat text after thousands of tokens under greedy decoding, potentially reflecting limited captioning data and unusually long outputs.

I Qualitative results

Molmo2-8B is illustrated on qualitative captioning, counting, tracking, pointing, and question-answering examples, alongside explicit failure cases. The failures include false positives and missed penguins.

  • The examples show model queries, responses, selected video frames, and pink annotations for returned points.
  • Molmo2-8B qualitative examples cover captioning, counting, tracking, pointing, and question answering.
  • Failure cases include false positives in the first two examples and missed penguins in the bottom example.
Loading 2601.10611v4…