Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

• The authors release Molmo2 weights, data, and code in 4B, 8B, and 7B variants based on Qwen3 and OLMo language models.The release is intended to support transparent, fully open research.Source: §1 · pp. 2–3
• The work introduces nine datasets covering video pointing and tracking, dense captioning, long-form and long-video QA, and multi-image pointing and document understanding.The suite includes 520k grounding instances, 104k densely captioned videos, 212k long-form QA instances, and around 1.3M long-video QA instances.Source: §1 · pp. 2–3, Table 1 · p. 4
• Molmo2 uses human-and-LLM collaboration and model-generated annotations to build long-form multimodal data from scratch without relying on proprietary-model distillation.Dense captions begin with spoken human narrations and are enriched with frame-level visual details, while medium-length video QA uses captions, transcripts, and the authors’ captioning model.Source: §1 · pp. 2–3
• Molmo2 brings grounding to video by pointing to event locations in space and time and continuously tracking objects across videos.The models extend image pointing to temporal video tasks and support single-image, multi-image, and video inputs.Source: §1 · pp. 2–3
• All variants follow three training stages, enhanced by token weighting, sequence packing, message-tree scheduling, and bi-directional attention between visual tokens.The stages comprise image-captioning and image-pointing pre-training, joint multimodal supervised fine-tuning, and short long-context training.Source: §1 · p. 3
• Molmo2 establishes new state-of-the-art video-grounding results while matching or surpassing existing models on short-video understanding, counting, captioning, and human preference.It also maintains strong performance on traditional image and multi-image benchmarks and challenges some proprietary systems.Source: §1 · p. 3

Dataset suite

• Nine newly created datasets comprise five human-annotated and four synthetic datasets, supplemented by two curated open-source datasets.The datasets are designed and collected through distinct pipelines summarized in this section.Source: §2 · p. 4

Video captioning

• 924 words per video makes Molmo2-Cap the densest video-caption dataset to date, with 104k video-level and 431k clip-level human captions.Its two-stage pipeline combines clip descriptions, whole-video summaries, frame-level captions, transcription, and LLM rewriting to capture dynamic and low-level visual details.Source: §2 · p. 4

Video QA

• 140k human-authored Molmo2-AskModelAnything QA pairs cover fine-grained questions across 31 video categories while excluding counting questions for pointing-based supervision.Annotators refine initial LLM answers through dialogue, followed by filtering for non-English, mismatched, or counting questions.Source: §2 · pp. 4–5

Synthetic video QA

• 1M synthetic video QA pairs and 300k subtitle-grounded QA pairs use scene-level captions, metadata, and transcripts to support detailed visual and audiovisual reasoning.The datasets cover 200k and 100k videos respectively, with five and three QA pairs per video.Source: §2 · p. 5

Video grounding

• Over 650k VideoPoint queries on 280k videos and 15k VideoTrack queries on 3.6k clips provide point-based supervision for diverse spatial-temporal localization and complex object tracking.VideoPoint spans eight query categories and averages six points per video, while VideoTrack averages 2.28 objects per query and validates natural-language queries against selected tracks.Source: §2 · p. 5

Multi-image data

• Multi-image datasets add 72k human QA pairs across 45k image sets, over 470k pointing and counting examples, and 188k synthetic text-rich-image QA examples.Image sets contain 2–5 semantically related images, while canonicalized labels and stochastic sampling preserve cross-image consistency and lexical diversity.Source: §2 · p. 5

• Packed-sequence attention masks block cross-attention between examples and between distinct QA pairs while allowing forward attention over frame tokens.Source: Figure 3 · p. 6
• The point representation sequences counts, x-y coordinates, and temporal information before an end-of-points marker and the language model.Source: I T T · p. 6
• Molmo2 connects a vision encoder to a language model to process video inputs.Source: Figure 2 · p. 6

3.1 Architecture

• The architecture combines a pre-trained LLM and ViT [36] through a connector [29] [89] that pools patch features into visual tokens interleaved with text, timestamps, or image indices.Image inputs use one downscaled crop plus up to K overlapping crops, with K = 8 during training and K = 24 during inference; videos use S = 2 fps and up to F = 128 frames, or F = 384 for long-context training.Source: §3.1 · p. 6

3.2 Training

• Molmo2 uses three stages: lightweight image-only pre-training, joint video/image supervised fine-tuning, and short long-context SFT; the final stage uses sequence length 36,864, F = 384, and 2k steps.Training uses Molmo2 data, PixMo image data, and various open-source datasets; long-context SFT uses the same SFT mixture and context parallelism across groups of 8 GPUs.Source: §3.2 · p. 7
• Token weighting assigns video captions weight 0.1 and pointing weight 0.2, while other tasks use 4√n to balance long and short outputs.The weighting addresses the risk that 4,000+ token captions dominate loss tokens and degrade short-answer or multiple-choice performance.Source: §3.2 · p. 7
• Packing merges multiple short examples into a single long sequence to reduce padding waste, while message trees encode videos and images with multiple annotations.An on-the-fly packing algorithm builds efficient packed sequences from in-memory examples and integrates with standard PyTorch data loaders.Source: §3.2 · p. 7

4.1 Overall results

• Molmo2 is state-of-the-art among non-proprietary models on short-video benchmarks, captioning, and counting, while matching or exceeding other open-weight models in human preference.It outperforms previous fully open models but trails the best open-weight models on long videos, likely because open long-video training data and compute are limited.Source: Table 2 · p. 8, §4.1 · p. 9

4.2 Grounding results

• Molmo2 scores highest on BURST-VC and Molmo2-VP and second highest on Molmo2-VC close accuracy, while significantly outperforming Gemini Pro 3.0 on video pointing.Counting uses close accuracy, and pointing evaluates F1, recall, and prediction against ground-truth masks.Source: Table 3 · p. 9, §4.2 · pp. 9, 11
• Molmo2 outperforms all tracking baselines across the academic and Molmo2-Track benchmarks, particularly on ReasonVOS and Molmo2-Track, which require reasoning and occlusion handling.Gemini 2.5 Pro is the strongest API model but still struggles to generate accurate object tracks.Source: Table 4 · p. 10, Table 5 · p. 10, §4.2 · p. 11

4.3 Image results

• Molmo2 robustly outperforms previous open-data image models, achieves state-of-the-art performance on VQA v2.0 and RealWorldQA, and is strongest on challenging PixMo-Count counting.It trails the best open-weight models on OCR-heavy and reasoning benchmarks, remains competitive on multi-image tasks, and GLM-4.1V-9B leads multi-image performance.Source: Table 6 · p. 11, §4.3 · p. 12
• Molmo2 surpasses every model on the Point-Bench leaderboard and the dedicated pointing model Poivre [171].The gain over Molmo is attributed to the improved vision encoder, pointing pre-training, and token-weighting.Source: §4.3 · p. 12, Table 7 · p. 12

4.4 Ablations and specialized models

• Joint video-data training improves captioning and QA, while bi-directional attention, token-weighting, and time tokens improve performance; larger video pooling degrades captioning.Removing timestamps diminishes both metrics, and frame-level captions outperform transcript-only training, whereas mixing caption types adds no improvement.Source: §4.4 · pp. 13–14
• Pointing data transfers positively to tracking, bounding-box and Molmo2 tracking data improve results, temporal grounding helps, and single-point tracking slightly degrades performance.The two pointing sources are complementary, pointing before counting is better than direct count prediction, and upsampling high-frequency points improves counting and pointing.Source: §4.4 · p. 14
• Long-context post-training significantly improves long-video QA, while video-captioning performance drops and short-video QA and image QA remain largely unchanged.This comparison uses Molmo2-4B and averages results across short- and long-video, captioning, and image benchmarks.Source: §4.4 · p. 14, Table 11 · p. 14

Multimodal LLMs

• Frame-wise CLIP-style image encoders dominate video LLMs, but visual-token counts grow with video length and sampling rate, motivating efficient video encodings.Video encoders using more frames lag behind image encoders such as SigLIP 2 [139], while common multimodal systems align image embeddings to LLM inputs through connector modules [29] [90].Source: §5 · pp. 14–15
• Closed-source video LLMs lead performance, while most open-weight models withhold training recipes or data; fully open pipelines avoiding proprietary VLM-generated data remain needed.A few fully open projects release data and details but use data generated by proprietary VLMs such as GPT4 and LLaMA3 [18] [4].Source: §5 · p. 15

Video-language instruction tuning datasets

• Synthetic video instruction data typically depends on closed-source VLM clip captions, introducing inaccessible training-data and bias effects.Molmo2-CapQA instead uses a captioner trained on fully open Molmo2-Cap to generate scene captions and 1M QA pairs, while Molmo2-AskModelAnything extends caption-assisted annotation to videos.Source: §5 · p. 15

Video tracking

• Video tracking evolved from closed-set bounding boxes toward track-any-point, segmentation, and language-guided open-vocabulary tasks, but specialized models lack broader capabilities.Molmo2-VideoTrack adds annotator-written, non-trivial queries emphasizing multiple objects and constructs masks from diverse open-source segmentation tracks and bounding-box tracks.Source: §5 · p. 15

Video pointing

• Image point-grounding datasets use detectors, referring-expression corpora, or human annotation; Molmo2 extends human annotation to video with frame selection and caption-based complex queries.The proposed query-generation process aims to ensure queries are complex and diverse.Source: §5 · p. 15

6 Conclusion

• Molmo2 supports open science by closing the gap between proprietary VLMs and the rest of the community.Source: §6 · p. 15

Appendix

• The appendix provides additional sections covering training, evaluation, results, test-time scaling, data, limitations, and qualitative results.Source: Appendix · p. 27

A Model details

• Images use overlapping crops resized to 378 without black padding, and all models use the SigLIP 2 So400m/14 384px ViT [139].The connector MLP’s intermediate dimension matches the LLM, so its size varies with the LLM.Source: A Model details · pp. 27–28
• Video frames are sampled at S fps and include the last frame, while tracking clips are capped at F frames to keep points aligned with annotations.Timestamp-based extraction handles variable frame rates, and uniform sampling is used when the frame limit would otherwise be exceeded.Source: A Model details · p. 27
• Pointing uses compact HTML-like strings with frame or image identifiers, sequential object indices, and normalized x-y coordinates from 0 to 1000.Object indices support counting and tracking, while the compact format reduces the number of tokens required to represent points.Source: A Model details · pp. 27–28
• Implementation uses PyTorch FSDP 2 [185], SDPA rather than FlashAttention [28] [27], torch.compile, static shapes, and mixed precision with bfloat16.Layer normalization and RoPE remain in full precision; loss normalization across devices avoids up-weighting examples with fewer loss tokens [51].Source: A Model details · p. 28
• Fine-tuning mixes datasets within batches, truncates fewer than 0.1% of overlong examples, and remains stable without loss spikes or NaNs.Source: A Model details · p. 28

Packing

• Packing selects examples with dynamic programming from a pool of M = 48, maximizing T + I ∗wi under token and crop constraints, with larger limits for long-context training.The solver uses T ≤16384 and I ≤128 normally, or 36864 tokens and 384 images for long-context training, with wi = 30 and 32-token quantization.Source: B Training details · p. 28

Training time

• Video-frame loading dominates data-loading cost, although packing does not slow training; during SFT, ViT computation dominates because each visual token processes 9 ViT patches.Increasing LLM size therefore has a reduced effect on training time.Source: B Training details · pp. 28, 31, 32

Pre-training and SFT

• Pre-training uses response-only dropout, length conditioning, captions and transcripts, connector-focused warmup, and a cosine learning rate decaying to 10% of peak.SFT uses regular residual dropout of 0.1, and the full SFT mixture is documented in Table 13 and Figure 4.Source: Table 13 · p. 30, B Training details · pp. 31–32, Figure 4 · p. 31

Prompting

• Prompting combines human-written long-form questions with templates for captioning, multiple-choice, and grounding, while style tags constrain terse academic answer formats.The human-written sources are PixMo-AskModelAnything, PixMo-CapQA, and Molmo2-AskModelAnything.Source: B Training details · p. 31

Specialized models

• Specialized models use shorter, capability-specific SFT pipelines, including 6k steps for QA, 5k for captioning, and staged pointing or tracking fine-tuning.Pointing uses 22k image-caption pre-training, 26k mixed SFT, and 6k video-pointing steps; tracking uses the same stages but 10k final steps.Source: B Training details · p. 32

C Evaluation Details

• Video captioning F1 is computed over 693 diverse, training-disjoint Vimeo videos by matching GPT-4.1-enumerated atomic statements between model captions and up to five human captions.The final metric is the harmonic mean of averaged precision and recall, with models prompted for long, detailed captions.Source: C Evaluation Details · p. 32
• 53% and 51% are Molmo2-8B’s and Molmo2-4B’s deterministic win rates against corresponding Qwen3-VL models, while Molmo2 ranks first among open models on open-ended QA but trails Qwen3-VL and GLM-4.1V on captioning.Human preference uses 1000-round bootstrap Elo estimates with confidence intervals; qualitative feedback attributes QA strength to adaptable explanations and captioning weakness to repetitive or nonsensical endings.Source: C Evaluation Details · pp. 32–34, Figure 5 · p. 33, Figure 6 · p. 34
• Molmo2-8B performs best on Action/Event and Object counting, matches Qwen3-VL on 0–10 objects, and exceeds 10% accuracy on 25–60 objects where Qwen3-VL obtains 0%.It trails slightly behind GPT-5 and Qwen3-VL-8B on Animal counting and remains just behind Gemini 2.5 Pro in the high-count range.Source: C Evaluation Details · p. 34, Table 16 · p. 35, Table 17 · p. 36
• 39.9 and 38.4 F1 are achieved by Molmo2-4B and Molmo2-8B on video pointing, versus 20.0 and 13.0 F1 for Gemini 3 and Gemini 2.5 Pro.Qwen3-VL-8B achieves only 1.5 F1, indicating that it rarely produces correct points.Source: C Evaluation Details · pp. 34–35
• Baseline counting and pointing evaluations use full videos or matched frame rates and model-specific preprocessing, with carefully designed prompts when baselines cannot follow Molmo2’s point format.Pointing predictions are evaluated from timestamps and normalized coordinates, while counting uses exact and close accuracy.Source: C Evaluation Details · pp. 34–35
• Tracking evaluation reports J&F at the original frame rate, point metrics at 1 fps, and HOTA for identity-aware tracking; Molmo2 directly predicts ID-linked point tracks, whereas baselines use boxes and SAM 2 when needed.HOTA adapts point-to-mask matching to measure detection and association accuracy while penalizing identity switches.Source: C Evaluation Details · p. 37
• Molmo2 substantially outperforms API-based, open-source, and specialized tracking models across complex video benchmarks, while specialized models can match or exceed it on simple single-object Ref-Davis tracking.On Molmo2-Track, point-based F1 and identity-aware HOTA expose failures hidden by high J&F from coarse masks in cluttered scenes.Source: C Evaluation Details · pp. 37–38

D.1 Additional model ablations

• Removing image-pointing pre-training causes a small decrease across benchmarks, including non-pointing tasks, despite compensatory SFT sampling and longer training.The authors hypothesize that pointing pre-training simplifies SFT by teaching the basic pointing format and task before fine-tuning.Source: §D.1 · p. 38, Table 18 · p. 38

D.2 NLP Benchmarks

• Molmo2 achieves comparable MMLU and GSM8K results, slightly outperforms its base model on ARC Challenge, but drops on MBPP+ coding [8].These benchmarks cover general knowledge QA, math, reasoning, and coding tasks.Source: Table 19 · p. 38, §D.2 · p. 38
• Evaluations follow OLMo 3’s protocol, except OLMo3-7B-Instruct’s MMLU and MBPP+ scores, taken directly from its model card.Results are compared with the base language models Qwen3 in Table 19.Source: §D.2 · p. 38

E Test time scaling with 128-frame model

• 224 frames gives Molmo2 its best long-video benchmark performance, while short-video averages remain 69.8 at 128 frames versus 69.7 for other settings.The long-video result averages six benchmark validation sets after SFT and before long-context training.Source: Figure 7 · p. 39, E Test time scaling with 128-frame model · p. 39
• SlowFast maintains approximately 10.6k vision tokens by assigning default 3 × 3 pooling to slow frames and 9 × 9 pooling to fast frames across up to 368 frames.Periodicities p ∈ {1, 2, 3, 4} correspond to maximum frame counts {128, 224, 300, 368}; query or neighborhood-based scoring selects slow-pathway frames when sampling is sparse.Source: E Test time scaling with 128-frame model · pp. 39–40
• Query-guided SlowFast selection boosts long-video understanding, approaches the 224-frame setting, and uses approximately 43% fewer visual tokens.Periodic SlowFast regresses, frame-difference selection improves but lags the default, and query selection causes only minor short-video regression.Source: Table 20 · p. 39, E Test time scaling with 128-frame model · p. 40
• Training without SlowFast followed by query-based test-time selection marginally beats training with SlowFast 10% of the time, indicating generalization to unseen 9 × 9 pooling.The training setup keeps maximum frames at 128 and randomizes slow-frame selection during a combined single stage.Source: Table 20 · p. 39, E Test time scaling with 128-frame model · p. 40

F.1 Dataset statistics

• Molmo2-VideoPoint covers seven pointing categories, with more Action/Event, Object, and Referring expression queries, but distributions skew toward low point counts and few annotated frames.Medium- and high-count examples are upsampled during training, and additional high-count examples are planned.Source: §F.1 · p. 40, Figure 8 · p. 41, Figure 10 · p. 41
• Counting and pointing validation sets use double annotations, high-confidence agreement, and category/count sampling, emphasizing object queries for counting and difficult referring categories for pointing.The designs are intentionally orthogonal to provide comprehensive evaluation of counting and pointing capabilities.Source: Figure 11 · p. 41, Figure 12 · p. 41, §F.1 · p. 42
• Molmo2-VideoTrack and Molmo2-Track contain 8k clips and 29k diverse text queries, emphasizing multi-object tracking with an average of 3.31 described objects per query.The datasets include segmentation masks, text queries, metadata, and videos up to two minutes, typically 10–30 seconds long.Source: §F.1 · p. 42

F.2 Data collection

• Data collection filters a pool exceeding 10M videos by normalized visual-temporal informativeness, then greedily selects approximately 100k videos maximizing semantic and visual diversity.For synthetic datasets, 500k Creative Commons videos are scene-captioned, repetition-filtered, and used for CapQA, SubtitleQA, VideoPoint, and AskModelAnything annotations.Source: §F.2 · p. 43

F.3 Data annoation

• Molmo2-Cap splits videos into information-density-based 10–30-second clips, obtains edited worker transcriptions, and then collects comprehensive descriptions of each full video.Videos are split into 4–5 clips on average, with shorter durations assigned to more informative segments.Source: §F.3 · p. 43
• Human annotators answer Molmo2-VideoPoint questions by screenshotting relevant moments, marking object points with timestamps and coordinates, and providing formatted answers or unanswerable/uncertain flags.The procedure repeats across all questions associated with each video.Source: §F.3 · pp. 43–44
• Generative-video anomaly pointing uses ViBe [124], Broken Video Detection Dataset [85], and VidProM [148] prompts to produce a 10k-video training set spanning defects from around 25 T2V models.The retained categories include Vanishing Subject, Physical Incongruity, and Temporal Dysmorphia, with pixel-accurate defect masks incorporated where available.Source: §F.3 · p. 44
• Molmo2-VideoTrack combines segmentation and bounding-box sources, converting boxes into SAM 2 mask tracklets, filtering by IoU, and sampling stable points for diverse object trajectories.The resulting corpus covers general and domain-specific tracking scenarios, while human-written queries are validated and approximately 70% are retained.Source: §F.3 · p. 44
• Academic-VideoTrack reuses the box-to-mask tracking pipeline and repurposes text-mentioned objects without tracks for single-point tracking, expanding training coverage to challenging non-exhaustive scenarios.This allows missing objects in non-exhaustive labels to contribute through a point-conditioned track-generation task.Source: §F.3 · p. 45
• Molmo2-AskModelAnything collects non-trivial visual-reasoning questions, then has workers revise backend-generated answers into validated final Q&A pairs.Workers are discouraged from submitting vague, subjective, unverifiable, low-level, or simple-counting questions.Source: §F.3 · pp. 45–46

G Data examples

• The dataset examples section presents randomly selected qualitative samples with bold prompts and target outputs shown across sampled video frames.Referenced examples cover Molmo2-Cap, AskModelAnything, CapQA, SubtitleQA, VideoPoint, VideoTrack, and the multi-image datasets.Source: G Data examples · p. 46

Closed image ViT

• Molmo2 still relies on the closed-data SigLIP 2 image encoder because no competitive open-data alternative currently exists.The authors call for open-source alternatives.Source: H Limitations · pp. 46–47

Use of closed LLMs

• Closed text-only LLMs generate data, reducing pipeline transparency, although avoiding closed VLMs prevents circular dependence in building a fully open system [89].Future open LLMs may enable fully open reproduction of the dataset.Source: H Limitations · p. 47

Video grounding repeating points

• Video grounding sometimes degenerates into repeated points across frames or long lines on one frame, especially for high-frequency objects and long videos.The issue may reflect insufficient training coverage or interference among jointly trained tracking and pointing tasks.Source: H Limitations · pp. 47–48

Video grounding

• None of the tested models exceeds 40% on video counting or pointing, versus 70–90% often achieved by image models on PointBench grounding metrics.Video grounding is harder because it requires more visual content, multi-object pointing, re-identification, and often lower-resolution processing; video-unadapted encoders may also contribute.Source: H Limitations · p. 48

Long video grounding

• Grounding has limited support for videos longer than 3 minutes because training covers only that duration and lower-than-2-fps sampling misaligns with annotations recorded at 2 fps.Customized frame sampling could preserve annotation alignment for longer videos.Source: H Limitations · p. 48

Point tracking

• Generated tracking points sometimes shift location on the target object because the data pipeline does not always enforce consistent within-object placement across frames.Generating points from bounding boxes or segmentation masks could mitigate this issue.Source: H Limitations · p. 48

Captioning

• Very long video captions can repeat text under greedy decoding, typically after thousands of tokens, due to known LLM behavior and limited captioning data.The issue is not observed in other tasks.Source: H Limitations · p. 48

Qualitative results

• Molmo2-8B provides qualitative examples spanning captioning, counting, and tracking.Successful examples are shown in Figure 34, with queries, model responses, and selected input-video frames annotated with returned points.Source: I Qualitative results · p. 48, Figure 34 · p. 56
• Molmo2-8B also demonstrates pointing and visual question-answering in qualitative examples.Figure 35 includes pointing and QA examples, including answers about marination duration and the UCLA baseball championship.Source: Figure 35 · p. 57, I Qualitative results · p. 57
• Failure cases show false positives in two examples and missed penguins in another.These cases are presented in Figure 36.Source: I Qualitative results · p. 48, Figure 36 · p. 58