Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

• Proprietary VLMs withhold weights, data, and code, while stronger open models often use proprietary or VLM-generated training data, motivating Molmo’s open weights and data without VLM-generated synthetic data.Source: §1 · p. 1
• PixMo includes three annotated datasets—dense captions, instruction following, and pointing—and four VLM-free synthetic datasets targeting skills such as clock reading and document understanding.Source: Figure 1 · p. 2, §1 · p. 2
• PixMo’s caption dataset covers 712k images with 200+-word descriptions elicited through 60–90 seconds of speech, with audio receipts documenting annotators’ recordings.The speech modality was found to produce more detailed descriptions in less time than asking annotators to write long captions; the collection approach was designed to work with untrained annotators.Source: §1 · p. 2
• For instruction following, users interactively edit responses with a language-only LLM, yielding 162k annotations across 73k images.Source: §1 · p. 2
• PixMo’s 2D pointing data comprises over 2.3 million grounding annotations, enabling faster collection than boxes or masks and supporting pixel-grounded answers and counting by pointing.Source: §1 · p. 2
• Molmo uses a pre-trained LLM and vision encoder with a simplified two-stage pipeline, overlapping multi-crops, efficient multi-annotation training, and optimizer and connector design choices.Source: §1 · p. 2
• Molmo-72B has the highest academic benchmark score and ranks second in human preference behind GPT-4o [90], outperforming many proprietary systems, including Gemini 1.5 Pro/Flash [103] and Claude 3.5 Sonnet.The evaluation covers 11 academic benchmarks and a human evaluation based on user preference.Source: §1 · pp. 2–3
• The team will also release a fully open Molmo model using a MetaCLIP [118] vision encoder and OLMo LLM, with all training data publicly available.Source: §1 · p. 3

Architecture

• The architecture combines image pre-processing, a ViT encoder [31], a connector that projects patch features into the LLM space, and a decoder-only LLM [109], following a standard design [69].Source: §2 · p. 3

Cropping

• Overlapping tiled crops provide neighboring context for border patches while excluding overlap features so the passed features tile the high-resolution image; a resized full image supplies an overview.More crops can offset the slight reduction in tiled resolution caused by overlap; the paper reports that overlap improves results in Section 6.Source: Figure 3 · p. 3, §2 · p. 3

Model variants

• The model family primarily uses OpenAI ViT-L/14 336px CLIP, with similar results achievable using SigLIP or MetaCLIP, and tests OLMo, OLMoE, and Qwen2 language models up to 72B.Training data and recipe are consistent across model choices except for learning rates; Qwen2 72B is identified as the best-performing model.Source: §2 · p. 3

Vision-language connector

• The connector concatenates third-to-last and tenth-from-last ViT features, attention-pools each 2×2 patch window using its mean as query, then maps pooled features to the LLM space via an MLP.The paper reports that two ViT layers slightly outperform one, and attention pooling outperforms simple feature concatenation.Source: §2 · p. 3

Training implementation

• Vision tokens follow full-image then row-major crop order with boundary markers; pre-training applies dropout only to text, and annotations for each image share a sequence with cross-annotation attention masked.The shared-sequence setup reduces processed images by two-thirds and training time by over half, with a 25% increase in sequence length for the data mix.Source: §2 · pp. 3–4

Data overview

• PixMo contains seven datasets: three with human annotations and four created through synthetic data-generation pipelines; Figure 4 shows fine-tuning data by sampling rate.The figure distinguishes collected human-annotated data, generated synthetic data, and pre-existing academic datasets.Source: §3 · p. 4, Figure 4 · p. 5

PixMo-Cap

• 712k distinct images yielded 1.3M PixMo-Cap transcripts and captions, averaging 196 words versus 11 for COCO captions [17] and 37 for localized narratives [93].Annotators described web images across ∼70 topics by speaking for at least 60 seconds; later, one annotator per image used a 90-second minimum, improving efficiency without sacrificing quality. A language-only LLM summarized multiple transcripts or enhanced single transcripts.Source: §3 · p. 4

PixMo-AskModelAnything

• 162k PixMo-AskModelAnything question-answer pairs cover 73k images.Annotators wrote image questions and reviewed answers generated by a language-only LLM using OCR data and a PixMo-Cap-trained model’s dense captions, requesting revisions when needed.Source: §3 · p. 4

PixMo-Points

• 2.3M question-points pairs from 223k images support text-based pointing, counting, and pointing as visual explanation.Annotators pointed exhaustively to described items and collected “not present” cases; a sequence of points can form a chain-of-thought for counting. The adapted question-answer pipeline also yielded 79k point-explanation annotations on 14k images.Source: §3 · p. 4

PixMo-CapQA

• 214k PixMo-CapQA question-answer pairs cover diverse topics and styles across 165k images.A language-only LLM generated questions and answers using only each image’s ground-truth caption.Source: §3 · p. 4

PixMo-Docs

• 255k text- and figure-heavy images were processed for PixMo-Docs, producing 2.3M question-answer pairs.An extensively tuned prompting framework generated code for images including charts, documents, tables, and diagrams; the LLM then generated questions and answers using privileged access to the code, not the images.Source: §3 · p. 4

PixMo-Clocks

• 826k PixMo-Clocks examples pair synthetic clocks with time-telling questions and answers.The images used ∼50 watch bodies and ∼160k realistic, diverse watch faces set to random times.Source: §3 · p. 4

PixMo-Count

• 36k PixMo-Count training images have object-center points and counting QA pairs, while validation and test sets each contain 540 images.A non-VLM detector selected the most-detected class after strict confidence thresholding; 120 samples per count from 2 to 10 were manually verified. The resulting diverse images form a more challenging counting QA set than CountBenchQA [10], which has reported limitations [10].Source: §3 · p. 4

Pre-training

• Pre-training uses all model parameters on PixMo-Cap to generate image captions or audio transcripts, with length hints included 90% of the time.The model trains for four epochs with AdamW and a cosine learning-rate schedule; connector, ViT, and LM learning rates are 2e-4, 6e-6, and 2e-5, respectively.Source: §4 · p. 5
• The training pipeline skips a separate connector-only stage when using PixMo-Cap, instead giving connector parameters a higher learning rate and shorter warmup.This reduces training time and complexity and avoids the noisy web-scale data typically used for connector tuning.Source: §4 · p. 5

Fine-tuning

• Fine-tuning combines PixMo datasets with open-source benchmark datasets, sampling proportional to square-root size and manually down-weighting several large synthetic datasets.Pointing data is significantly up-weighted because pointing tasks learn more slowly than QA tasks.Source: §4 · p. 5
• Task-specific style tags teach the model to use benchmark-specific answer styles only when requested, while most PixMo datasets use prompts or templates instead.PixMo-Cap uses about 30 caption prompts, and pointing data uses about 100 question templates; pointing-as-an-explanation retains a style tag because that mode can be less reliable.Source: §4 · p. 5
• Pointing outputs are plain-text coordinates normalized from 0 to 100, and sequential pointing supports a chain-of-thought approach to counting.For multiple items, points are ordered top-down and left-to-right, with each point numbered; counting questions also return a total count.Source: §4 · p. 5

5. Evaluation

• Molmo’s best-in-class Qwen2 72B model achieves the highest academic benchmark score, ranks second in Elo behind GPT-4o, and outperforms Gemini 1.5 Pro and Flash and Claude 3.5 Sonnet.The academic benchmark table covers 10 common datasets and PixMo-Count.Source: Table 1 · p. 6, §5 · p. 7
• Benchmark comparisons require caution: performance can vary by up to 10% with evaluation details, and missing prompts or processing steps hinder reproducibility.The evaluation gathers or computes results across 10 common datasets and PixMo-Count, prioritizing author-reported values when available.Source: §5 · pp. 5–6
• Human evaluation uses 15k image-text prompt pairs, about 870 annotators, and over 325k pairwise ratings to calculate Elo with the Bradley-Terry model.The study gathers about 450 ratings per model pair and follows Chatbot Arena methodology [21].Source: §5 · p. 6
• MolmoE-1B nearly matches GPT-4V academically and in Elo, while Molmo 7B models fall between GPT-4V and GPT-4o; Qwen2-VL’s academic strength does not carry over to human rankings.Broadly, academic and human evaluations agree, with Qwen2-VL as the noted exception.Source: §5 · p. 7
• On an independent Chatbot Arena Elo evaluation, Molmo-72B outperforms all open models but ranks below several proprietary models, including GPT-4o and Claude 3.5 Sonnet.The paper suggests question mix may explain the difference, noting that Molmo is particularly strong on counting and image-description questions.Source: §5 · p. 7
• Molmo matches or outperforms all models on zero-shot RealWorldQA and achieves state-of-the-art VQA v2.0 results; it leads open models and some proprietary ones on OCR, all models on counting, but lags on reasoning.The counting advantage is attributed to the new pointing data and chain-of-thought point-and-count abilities; the reasoning gap is attributed to a lack of advanced-reasoning training data.Source: §5 · p. 7
• Additional tests find Molmo excels at clock reading, has competitive AndroidControl accuracy, slightly trails its component LLM on NLP, and performs strongly on a new SAM-based pointing benchmark.Molmo-72B reaches 88.7% low-level and 69.0% high-level AndroidControl accuracy, compared with reported values of 83.2% and 70.8% [62].Source: §5 · p. 7

6. Ablations

• Overlapping multi-crop inputs, text-only dropout, length-conditioned captioning, and attention pooling improve reported ablation outcomes; MetaCLIP performs roughly like other noisy-web encoders.Increasing resolution generally helps except for pointing and captioning, and higher-resolution tuning yields a slight gain; captioning gains do not always track 11-avg.Source: §6 · p. 8
• Scaling PixMo-Cap improves captioning and benchmark metrics; human captions match GPT-4o captions on PixMo-Cap images, while noisy web-scale data and ShareGPT4v/o underperform.PixMo supervised fine-tuning improves benchmarks beyond academic datasets, mainly on document-heavy and counting tasks.Source: §6 · p. 8
• Point-then-count outperforms count-only or count-then-points; ordered points, correct locations and counts, and plain-text coordinates also perform better in the tested counting ablations.Pointing alone, counted by regular expression, is only slightly worse than point-then-count; correct point locations and counts beat random locations and/or point counts.Source: §6 · p. 8
• Human evaluation finds PixMo-Cap and PixMo-AskModelAnything important for user-preferred responses; academic datasets help but perform very poorly alone, while GPT-4o captions also perform well.The authors attribute GPT-4o captions’ performance to recent GPT advances and the diversity of their image collection.Source: §6 · pp. 8–9

Appendix

• The appendix covers model, training, evaluation, result, ablation, data, dataset-example, and related-work details.Source: Appendix · p. 9

A. Model Details

• The model-details appendix presents additional image-encoding, hyper-parameter, and implementation choices.Source: A. Model Details · p. 9

A.1. Image Encoding

• Images are fitted to a crop grid with preserved aspect ratio, black padding, and a maximum crop count; each crop is encoded independently, with padding-aware embeddings and special tokens.Overlapping crops use a fixed margin of 4 patches or 56 pixels; image patches are encoded before any text input.Source: §A.1 · p. 9

A.2. Hyper-Parameters

• All runs use a cosine learning-rate schedule ending at 10% of peak; the 72B model uses a reduced learning rate and fewer training steps.The connector MLP matches the LLM’s intermediate dimension, while connector pooling and ViT architecture are shared across models.Source: §A.2 · pp. 9–10

A.3. Implementation

• PyTorch FSDP based on the OLMo codebase is used; FlashAttention is omitted because it cannot support the required complex masks, while SDPA achieves close to the same speed.The complex masks are required for multi-annotated images.Source: §A.3 · p. 10
• AMP runs most operations in bfloat16, but model weights and gradient reduction remain in full precision because half precision degrades training loss.Source: §A.3 · p. 10
• Layer normalization and Rotary Position Embedding computations are explicitly performed in full precision.Source: §A.3 · p. 10
• Normalizing each device’s loss by the cross-device average number of loss tokens avoids up-weighting short-response examples; without this fix, captioning performance drops 0.5–1 points.The paper notes this issue has been discussed elsewhere [40] and affected many codebases.Source: §A.3 · p. 10

B. Training Details

• Fine-tuning mixes tasks within each batch; pre-training and fine-tuning both cap sequences at 2304 tokens, truncating longer examples such as some DVQA data.Source: §A.3 · pp. 10–11, B. Training Details · p. 11

A.3. Implementation

• Training is stable without loss spikes or NaNs, likely in part because the models are pretrained.Source: §A.3 · p. 11

B.1. Pre-Training Task Details

• Pre-training pairs each image with its caption and one randomly selected audio transcript per epoch, training on both through multi-annotation.When an image has multiple transcripts, one is selected randomly each epoch.Source: §B.1 · p. 11
• Pre-training prompts distinguish “long caption:” from “transcript:” and include a noisy character-based length hint 90% of the time, omitting it 10%.The hint is the caption or transcript character count plus Gaussian noise with standard deviation 25, divided by 15 and rounded down; it is intended as guidance, not a hard constraint.Source: §B.1 · p. 11
• Within 10 characters of the expected length on average, captions follow length hints, which trade precision against recall; short hints can reduce recall while boosting accuracy.A hint of 65 performs similarly or slightly better than no hint in ablations. Adding COCO Captions, Localized Narratives, or Visual Genome-derived captions did not improve scores, so pre-training uses PixMo-Cap alone.Source: §B.1 · p. 11, Figure 7 · p. 11

B.2. Fine-Tuning Task Details

• Fine-tuning uses training splits, label-only targets for academic multiple-choice questions, the most common VQA answer, and HTML-like pointing with coordinates scaled to 0–100.When multiple VQA answers are tied as most common, one is randomly selected each epoch; academic multiple-choice options are listed with capital-letter labels.Source: §B.2 · p. 12
• Point annotations number coordinates and use inline and alt text to identify targets; PixMo-Points excludes counts above 40, while counting questions without point data receive randomized no-point instructions.For pointing and counting, inline and alt text both name the target; PixMo-AskModelAnything counting prompts are heuristically detected and prefixed with one of 20 instructions such as “Answer without points.”Source: §B.2 · p. 12
• Fine-tuning reweights ChartQA’s synthetic examples, uses A-OKVQA in multiple-choice and eligible direct-answer forms, and treats TabWMP as short-answer.AI2D uses transparent and opaque region boxes, with transparent boxes for main results; prior studies used opaque [5] and transparent boxes [10] [106]. A custom AI2D validation set contains 384 images and roughly 2,000 question-answer pairs.Source: §B.2 · p. 12
• AndroidControl training covers four input-output configurations, supplies only instructions and screenshots as inputs, and represents target actions as text plus coordinates scaled to 0–100.The configurations include low-level instruction to action, high-level goal to action, combined low- and high-level inputs to action, and high-level goal to action with chain-of-thought reasoning; accessibility trees, action history, and available-action prompts are omitted.Source: §B.2 · p. 12

B.3. Training Time

• Training times and GPU counts are reported for models trained on H100 GPUs with InfiniBand connectivity.Table 8 is identified as the training-times table; the supplied passage does not give its values.Source: §B.3 · p. 12, Table 8 · p. 12

Captioning metric (cap F1)

• cap F1 scores caption quality on 15,006 disjoint evaluation images by taking F1 of image-averaged precision and recall from GPT-4o statement matching and consistency judgments.GPT-4o matches generated-caption atomic statements to statements from up to six ground-truth audio transcripts per image for recall, and judges statement consistency against raw transcripts for precision.Source: C. Evaluation Details · p. 13

Human evaluation

• Human evaluation covered ten image-question categories, with image-question pair counts reported for each.Counts: Output formatting 1,525; Fine-grained detailed QA 1,510; General 1,504; Documents 1,499; Captioning 1,493; Counting 1,490; Homework 1,489; Charts 1,473; Named entities 1,448; Creativity 1,420.Source: C. Evaluation Details · p. 13

Captioning metric (cap F1)

• Although cap F1 is imperfect because GPT-4o can err and transcripts omit true image statements, improvements tracked subjective caption quality and guided model and pre-training data design.The paper points to Section D and Figure 9 for further discussion of captioning-quality-driven design decisions.Source: C. Evaluation Details · p. 13

Human evaluation

• Human comparisons blinded model identities, gathered ∼450 responses per model pair in the main study and used 500 verified questions for ablations before Bradley-Terry Elo ranking.The main study sampled randomly across categories; the ablation study used exactly 50 manually verified questions per category, and removed “I don’t know” responses before ranking.Source: C. Evaluation Details · p. 13

AndroidControl

• AndroidControl evaluation used only the task instruction and current screenshot as input, reporting step-wise accuracy on the in-domain data test set.The task instruction could be high-level or low-level.Source: C. Evaluation Details · p. 13

D. Result Details

• Molmo-72B beat fully open and open-weight models but trailed some proprietary models in Chatbot Arena; it ranked first on VQA v2.0 and A-OKVQA, and third on DocQA and InfoQA.Source: D. Result Details · pp. 13, 16, Table 9 · p. 14
• Molmo stood out on clock reading while other VLMs struggled, though Molmo-72B underperformed Molmo-7B-D and MolmoE-1B.PixMo-Clocks is entirely synthetic and highly out-of-distribution from the real-world benchmark; the paper also reports generalization to more complex clock questions and captioning.Source: D. Result Details · pp. 13–14, Table 10 · p. 14
• Molmo showed superior pointing, but performance declined when training and test crop counts differed.Evaluation used 493 manually verified image-question pairs; for targets, predicted points were matched to ground-truth points and checked against segmentation masks.Source: D. Result Details · p. 14, Table 11 · p. 15
• 88.5→87.7 on CountBenchQA and 85.2→73.9 on PixMo-Count when inference crops rose from 12 to 36; high-resolution fine-tuning restored counting without sacrificing 11-avg.The 12-crop model received 3,000 additional fine-tuning steps with 36 crops; training directly with 36 crops raised 11-avg from 76.9 to 77.2.Source: Table 12 · p. 15, D. Result Details · p. 15
• Against Claude-3.5 Sonnet, Molmo-7B-D won 14.3% of matches and Claude won 26.1%; 45.5% were good-response ties and 14.1% bad-response ties.Against Molmo fine-tuned only on academic data, Molmo-7B-D won 41.1%, with 28.4% good-response ties, 22.1% bad-response ties, and 8.4% losses.Source: Figure 8 · p. 15, D. Result Details · p. 15
• Multimodal fine-tuning appeared to reduce Qwen2’s text-only knowledge, while adding Tulu 39 data improved text-task results; 10% down-sampling performed better on most text-only tasks and improved 11-benchmark average performance.Source: D. Result Details · p. 15
• ρ = 0.82 between cap F1 and 11-avg across 22 qualifying ablations, suggesting dense-caption optimization may proxy for downstream performance, without establishing causality.The analysis included experiments affecting pre-training that used PixMo-Cap and excluded runs with different pre-training data.Source: Figure 9 · p. 16, D. Result Details · p. 16

E.1. Discussion of Main Paper Ablations

• Web-scale noisy image-text encoders perform similarly on both metrics; MetaCLIP enables a fully open Molmo with OLMo, while DINOv2 performs only slightly worse.DINOv2 wins 45% of non-tie user-study matches against the standard Molmo-7B-D configuration.Source: §E.1 · p. 16
• More image crops generally improve results, but captioning and counting degrade when test and training crop counts differ; document-heavy tasks can benefit from extra inference crops.A small amount of high-resolution fine-tuning can address the crop-count mismatch, after which the same crop count can be used for all tasks.Source: §E.1 · pp. 16–17
• Both metrics improve as PixMo-Cap grows from zero to all 712k images, and removing it yields a 35% user-study win-rate against the standard model.The passage attributes the preference drop to a severe negative impact from removing PixMo-Cap.Source: §E.1 · p. 17
• LLM dropout generally improves pre-training and fine-tuning, text-token-only dropout improves captioning, and length conditioning affects captioning while improving downstream tasks.Text-token-only dropout applies to caption tokens, not vision or prompt tokens; the authors hypothesize it encourages reliance on vision and may reduce hallucinations.Source: §E.1 · p. 17
• LAION 2B connector-only pre-training yields no metric improvement, ShareGPT4V/o [15] captions perform worse than PixMo-Cap, and GPT-4o captions on PixMo-Cap perform strongly.The authors suggest PixMo-Cap’s more diverse image distribution and GPT-4o captioning improvements may explain the latter result.Source: §E.1 · p. 17
• Using both raw audio transcripts and LLM-cleaned transcripts performs slightly better than using either transcript source alone.Source: §E.1 · p. 17
• Academic-only fine-tuning scores 72.2% versus 76.8% for the full mixture; document and counting data explain much of the gap, while special point tokens underperform plain-text coordinates.Other PixMo fine-tuning datasets have small or slightly negative effects on the 11 benchmarks but add skills and improve user experience.Source: §E.1 · p. 17

E.2. Additional Ablations

• Additional model ablations examine vision-encoder layers, learning-rate warmup, and gradient normalization.Source: §E.2 · p. 17

• PixMo-Points contains 229k unique images and 1.98M referring expressions, averaging 8.7 expressions per image, 5.5 points per expression, and 47.7 points per image.Source: F. Data Details · p. 17
• Using both the 3rd-to-last and 10th-to-last ViT layers yielded cap F1 11-avg 54.1, versus 53.7 using only the 3rd-to-last layer and 52.5 using only the 10th-to-last.Source: Table 15 · p. 18, F. Data Details · p. 18
• PixMo-Points is described as larger and more diverse than gRef-COCO, RefCOCO, RefCOCOg, and RefCOCO+, and focuses on points rather than segmentation masks for more efficient collection.gRef-COCO has 20k images and 278k expressions; RefCOCO, RefCOCOg, and RefCOCO+ have about 86k, 142k, and 141k unique referring expressions, respectively.Source: F. Data Details · p. 18
• PixMo-Cap annotators were prompted to describe the image at first glance, objects and counts, text, positions, subtle details, background, and style and color.Source: F. Data Details · p. 18
• PixMo-Docs generates text- and figure-heavy images with programs produced by a text-only LLM, then uses those programs as context for instruction-tuning data; its framework supports seven rendering tools.Claude-3.5 Sonnet generates code and GPT-4o-mini generates instruction-tuning data, prioritizing cost efficiency.Source: F. Data Details · p. 18
• PixMo-Docs uses varied text queries to select generation tools and personas [35] to control synthetic data content and style.For example, a barbecue enthusiast persona shaped a restaurant-menu example into a Southern fusion menu combining traditional BBQ with international flavors.Source: F. Data Details · p. 18

G. Dataset Examples

• The appendices show randomly selected examples from PixMo-Cap, AskModelAnything, Points, Points with explanations, CapQA, Clocks, Count, and Docs.Docs examples include charts, tables, diagrams, and other documents; prompts are bold and points appear as pink dots.Source: G. Dataset Examples · p. 18

H. Related Work

• Prior VLMs use CLIP-style encoders, sometimes alongside SSL encoders, while other designs use cross-attention or input pixels directly [26] [41] [59] [69] [85] [92] [105] [112] [131].Cross-attention can add parameters while freezing the LLM and preserving text-only task performance; efficient multimodal models have also emerged under compute constraints [5] [6] [58] [14] [22] [64] [79] [123] [128] [137].Source: H. Related Work · pp. 18–19
• Top-performing multimodal LLMs are proprietary [90] [103], while open systems may omit training details or rely on data generated by proprietary VLMs [15].The authors argue for a fully open state-of-the-art training pipeline that does not use previously trained multimodal LLMs to generate data.Source: H. Related Work · pp. 19–20
• PixMo-CapQA uses detailed PixMo-Cap captions to generate QA pairs, addressing the limited image detail in automatically generated or ground-truth annotations.The passage contrasts this approach with annotation-plus-LLM methods [57] [69] [125] [137] and direct proprietary-VLM annotation [13] [15] [68] [110].Source: H. Related Work · p. 20
• PixMo-Docs supports diverse chart formats through code-based representations, while its document generation considers more than HTML [46] [47] [86] [55].For clock data, the approach uses real watch faces rather than simulator-only renders, enabling varied hands, decorations, colors, and backgrounds [121].Source: H. Related Work · p. 20
• PixMo-Points uses human-collected points rather than masks, yielding 1.98M unique referring-expression instances with an average of 5.5 points per expression.Unlike GRES [66], PixMo-Points covers diverse images and expressions that can refer to many objects, not just a limited category set.Source: H. Related Work · p. 20
• Molmo uses closed text-only LLMs for some dataset generation, arguing this dependency can be replaced by sufficiently capable open LLMs, unlike the circular dependency of using one VLM to build another [37].The authors argue that building open VLMs should proceed in parallel with open-LLM research.Source: H. Related Work · p. 20