PaperlayerView exact revision on arXiv

Source-linked AI summary

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi

arXiv:2409.17146v2cs.CVcs.CLcs.LG

TL;DR

Proprietary VLMs dominate performance, while open models often depend on proprietary data or synthetic outputs, limiting foundational knowledge about building VLMs from scratch. Molmo addresses this gap with fully open weights and PixMo datasets collected without external VLMs, using a standard architecture and tuned training pipeline. Molmo-72B achieves the highest academic benchmark score and ranks second by human preference behind GPT-4o, while outperforming several proprietary systems.

  • Problem

    The strongest VLMs remain proprietary, and many stronger open-weight models rely heavily on proprietary or proprietary-VLM-generated data.

  • Method

    Molmo combines released model weights with PixMo datasets, including detailed captions, free-form image Q&A, and 2D pointing data collected without external VLMs.

  • Results

    Molmo-72B achieves the highest academic benchmark score and ranks second by human preference behind GPT-4o, outperforming Gemini 1.5 Pro and Flash and Claude 3.5 Sonnet.

  • Takeaways & Limitations

    Molmo provides a fully open model and data foundation for studying how performant VLMs can be built without distilling proprietary VLMs.

  • Takeaways & Limitations

    Comparisons are difficult to reproduce because performance can vary by evaluation details and critical prompts or data-processing steps are often unavailable.

Abstract

from arXiv · show

Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.

1. Introduction

Molmo addresses the lack of fully open, high-performing VLMs by pairing released weights with PixMo data collected without external VLMs. Its model and data choices support strong benchmark and human-preference performance across model scales.

  • Motivation: Proprietary VLMs remain the most performant, while stronger open-weight models often rely on proprietary data or synthetic outputs from proprietary systems.This leaves the community without foundational knowledge for building performant VLMs from scratch.
  • Contribution: Molmo releases model weights and vision-language training data without relying on synthetic data from other VLMs.The fully open PixMo collection is presented as the central data contribution.
  • Contribution: PixMo combines dense captions, instruction-following data, 2D pointing data, and additional synthetic datasets targeting skills such as clock and document understanding.The collection includes three annotated datasets and four synthetic datasets, all constructed without VLMs.
  • Data collection: 712k images receive 200+ word captions collected through 60-to-90-second speech descriptions rather than direct long-form writing.The recordings provide an audio receipt that a VLM was not used.
  • Data collection: Over 2.3 million 2D grounding annotations support pointing-based answers and counting, while instruction-following data contains 162k annotations on 73k images.Point annotations are faster to collect than bounding boxes or segmentation masks.
  • Results: Molmo is evaluated on 11 academic benchmarks and human user preference, with Molmo-72B ranking second behind GPT-4o and outperforming several proprietary systems.MolmoE-1B nearly matches GPT-4V, while the 7B models perform between GPT-4V and GPT-4o on both evaluation types.

2. Architecture

Molmo uses a standard vision-language architecture augmented with multiscale overlapping crops and a connector that converts patch features into language-model embeddings. The design preserves global image context while improving fine-grained visual access.

  • Architecture: The architecture combines a pre-processor, ViT image encoder, vision-language connector, and decoder-only language model.The pre-processor creates multiscale, multi-crop images; the connector pools and projects patch features into the LLM space.
  • Model family: Molmo varies the vision encoder and language model while keeping training data and recipe consistent across model variants, apart from learning rates.The tested language models range from fully open OLMo variants to open-weight Qwen2 models.
  • Cropping: Multiple square crops plus a resized full-image overview address the fixed-resolution limits of most ViTs for OCR and detailed captioning.Each crop is processed independently by the vision encoder.
  • Cropping: Overlapping crops give border patches neighboring context while passed features still tile the high-resolution image exactly.Overlap slightly reduces tiled resolution but can be offset with more crops, and it significantly improves results.
  • Vision-language connector: The connector concatenates features from two ViT layers, attention-pools each 2×2 patch window, and maps the pooled vectors into the LLM embedding space with an MLP.Attention pooling outperforms simple feature concatenation.
  • Token arrangement: Vision tokens begin with low-resolution full-image patches, followed by high-resolution crop patches in row-major order, with special tokens marking sequence boundaries and rows.This ordering makes the spatial structure explicit to the language model.

3. Data

PixMo supplies open pre-training and fine-tuning data through detailed image descriptions, edited question-answer examples, and broad skill coverage. Its collection process combines human annotations with language-only model assistance without using external VLMs.

  • Dataset portfolio: PixMo contains seven datasets: three human-annotated datasets and four synthetic datasets created without VLMs.The collection targets pre-training, instruction following, pointing, and additional capabilities.
  • PixMo-Cap: Annotators describe images by speaking for at least 60 seconds, with later collection using a 90-second minimum and one annotator per image.This approach improved efficiency without sacrificing quality in later stages.
  • PixMo-Cap: PixMo-Cap contains 712k distinct images paired with 1.3M transcripts and captions averaging 196 words.Images span approximately 70 topics, and captions are produced from annotators’ spoken descriptions using a language-only LLM.
  • Instruction following: Instruction-following data is collected by having users interactively edit language-only LLM responses, yielding high-quality free-form image answers.The process produces 162k annotations on 73k images.
  • Fine-tuning data: PixMo’s fine-tuning mixture includes human-annotated, generated synthetic, and pre-existing academic datasets, with PixMo-Docs divided into charts, tables, diagrams, and other documents.Figure 4 displays these datasets in proportion to their sampling rates.

4. Training

Molmo pre-trains on detailed PixMo-Cap descriptions and fine-tunes on a weighted mixture of PixMo and academic datasets. The recipe uses length-conditioned captioning, streamlined connector training, style controls, and point-based supervision for grounding and counting.

  • Pre-training: Pre-training generates either a caption or an audio transcript from each image, with prompts specifying the output style and usually providing a length hint.The length hint improves caption and pre-training quality.
  • Pre-training: PixMo-Cap makes a separate connector-only training stage unnecessary, so the connector instead uses a higher learning rate and shorter warmup.Skipping the stage reduces training time and complexity and avoids typical noisy web-scale data requirements.
  • Fine-tuning: Fine-tuning mixes PixMo datasets with open-source academic datasets sampled roughly by the square root of dataset size.Very large synthetic datasets are manually down-weighted, while pointing data is significantly up-weighted because it learns more slowly.
  • Fine-tuning: Style tags and randomly sampled prompts or templates help preserve benchmark skills while reducing dataset-specific answer styles in user-facing responses.Pointing prompts request locations or counts, and outputs use normalized coordinates ordered top-down and left-to-right.
  • Pointing supervision: Pointing supports a point-then-count training strategy in which the model sequentially indicates occurrences before producing the total count.The representation uses plain-text coordinates normalized between 0 and 100.

5. Evaluation

Molmo is evaluated on academic benchmarks and human preference rankings, with results that vary by task and evaluation setting. Its strongest performance is in natural-image understanding and counting, while advanced reasoning remains a weakness.

  • Evaluation setup: Evaluation combines 10 common academic datasets, the harder PixMo-Count benchmark, and a human preference study.The human study uses 15k image-text prompts, approximately 870 annotators, and over 325k pairwise ratings scored with Elo.
  • Evaluation caveats: Academic benchmark results and human evaluation broadly agree, although Qwen2-VL performs strongly academically and comparatively worse in human evaluation.The authors caution that prompting, answer-style alignment, and training-data use can substantially affect comparisons.
  • Overall results: MolmoE-1B nearly matches GPT-4V, while Molmo-72B achieves the highest academic benchmark score and ranks second in human preference behind GPT-4o.The intermediate 7B models perform between GPT-4V and GPT-4o on both evaluation axes.
  • Task-specific results: Molmo matches or outperforms all models on zero-shot RealWorldQA and achieves state-of-the-art results on VQA v2.0.These results specifically concern questions about natural images.
  • Task-specific results: On OCR-centric benchmarks, Molmo surpasses other open models and some proprietary systems but trails slightly behind Qwen2-VL.The OCR benchmarks are ChartQA, DocQA, InfoQA, and TextVQA.
  • Task-specific results: Molmo leads counting benchmarks, achieves competitive AndroidControl accuracy, and lags on MMMU and MathVista reasoning tasks.The reported AndroidControl accuracies are 88.7% for low-level actions and 69.0% for high-level actions.

6. Ablations

Ablations identify overlapping multi-crop vision encoding, length-conditioned captioning, and PixMo data as important design choices. Point-based supervision particularly improves counting, while academic data alone is insufficient for the full task suite.

  • Ablation framework: Ablations measure captioning cap F1 and average accuracy across the 11-benchmark suite.The cap F1 metric is intended to reflect broad image understanding learned during pre-training.
  • Model ablations: CLIP, SigLIP, and fully open MetaCLIP perform roughly similarly, while increasing crops and using overlapping crops generally improve performance.The overlapping design gives patches context from neighboring regions and significantly improves results.
  • Model ablations: Text-only dropout, length-conditioned captioning, and attention pooling improve captioning or downstream benchmark performance over their respective baselines.Length-conditioned captioning is described as a strong pre-training task, and attention pooling improves both reported metrics.
  • Data ablations: Scaling PixMo-Cap to 712k images significantly improves captioning and benchmark metrics, while PixMo-Cap outperforms similarly sized ShareGPT-derived data.Human-annotated captions perform roughly as well as GPT-4o-generated captions on the PixMo-Cap images.
  • Counting ablations: Point-then-count chain-of-thought, predictable spatial ordering, correct point locations, and plain-text coordinates improve counting results.Pointing alone is only slightly worse than point-then-count, whereas random locations or point counts perform worse.
  • Human evaluation: Human evaluation indicates that PixMo-Cap and PixMo-AskModelAnything are important for preferred responses, whereas academic datasets perform extremely poorly alone.GPT-4o captions on the same images also perform well in the ablation evaluation.

A.3. Implementation

The implementation combines distributed PyTorch training with mixed precision, full-precision handling for numerically sensitive components, and loss normalization that avoids token-count bias. Training is reported as stable.

  • Distributed training: The implementation uses PyTorch FSDP based on the OLMo codebase, with SDPA replacing FlashAttention because complex multi-annotation masks are required.SDPA achieves close to the speed of FlashAttention in the authors’ implementation.
  • Numerical precision: Automatic mixed precision runs most operations in bfloat16, but model weights and gradient reduction remain in float32 because half-precision degrades training loss.Layer normalization and RoPE computations are also explicitly performed in full precision.
  • Gradient reduction: FSDP gradients are normalized by the average loss-token count across devices rather than each device’s local token count.This avoids up-weighting examples with fewer loss tokens, such as short responses.
  • Batching and length: Fine-tuning mixes tasks within batches and truncates sequences at a maximum length of 2304 tokens.Truncation mainly affects certain synthetic datasets and occasional outlier examples.
  • Training stability: Training is reported as stable, without loss spikes or NaNs, likely partly because the models are initialized from pre-trained components.

B.1. Pre-Training Task Details

Pre-training pairs images with detailed captions or audio transcripts and uses noisy length hints to control output length. Fine-tuning formats multiple task types, including questions, pointing, counting, and actions.

  • Pre-training inputs: Pre-training uses each image with its caption and one randomly selected audio transcript, training both jointly through multi-annotation training.
  • Pre-training inputs: Prompts distinguish captions from transcripts and usually include a noisy length hint derived from the target character length.The hint is included 90% of the time, while 10% of examples omit it to preserve default captioning.
  • Captioning control: Length hints trade captioning precision against recall, with shorter hints reducing recall but potentially improving accuracy.Using a length hint keeps average caption length within 10 characters of the expected length; ablations use a hint of 65.
  • Captioning data: PixMo-Cap alone is used for pre-training because adding COCO, Localized Narratives, or Visual Genome-derived captions did not improve the captioning metric.
  • Question answering: Fine-tuning formats academic multiple-choice tasks to predict answer labels, while other datasets use direct answers or short answers as specified by task.A-OKVQA includes both multiple-choice and direct-answer versions, and TabWMP is treated as short answer.
  • Pointing and counting: Pointing outputs encode coordinates on a 0-100 scale, support multiple numbered points, and use point names as inline and hover text for interactive responses.Numbering points makes counting easier because the final point number gives the total count.
  • Task-specific formatting: ChartQA synthetic and non-synthetic examples are re-weighted equally because the synthetic subset is larger and observed to be noisier.
  • Action grounding: AndroidControl trains four instruction-to-action configurations using only the instruction and screenshot, with target actions represented as text and coordinates scaled to 0-100.

C. Evaluation Details

The evaluation combines captioning and human-preference protocols, with results organized across diverse image-question categories. Captioning quality is summarized by cap F1, while human evaluation compares anonymized model outputs using multiple outcome categories.

  • Captioning evaluation: Captioning quality is measured with cap F1, the harmonic mean of averaged captioning precision and recall.The evaluation set contains 15,006 manually selected images, disjoint from PixMo-Cap, with up to six audio transcripts per image.
  • Human evaluation: The human evaluation covers ten question categories, including general questions, documents, captioning, counting, charts, and creativity.The listed category counts range from 1,420 creativity examples to 1,525 output-formatting examples.
  • Evaluation protocol: Some evaluations use a 1,500-image random subset of a 2,730-image superset.The authors do not expect this sampling difference to affect results significantly.
  • Human evaluation: Annotators compare anonymized outputs from model A and model B and can select either model, two tie types, or “I don’t know.”The final option is intended for cases where the annotator cannot determine which response is better.

D. Result Details

Results show strong performance across specialized capabilities, with Molmo leading open comparisons in Chatbot Arena and pointing, while crop alignment and task-specific training materially affect outcomes. High-resolution fine-tuning can recover counting performance, and multimodal training introduces measurable text-only trade-offs that text data can partly address.

  • Chatbot Arena: Molmo-72B outperforms fully open and open-weight models on Chatbot Arena’s vision leaderboard but trails some proprietary VLMs.The authors’ own Elo evaluation places Molmo-72B in second place, with rating differences attributed partly to question-type differences.
  • Clock reading: All tested VLMs struggle with in-the-wild clock reading except Molmo, while Molmo-72B underperforms Molmo-7B-D and MolmoE-1B.The benchmark uses out-of-distribution images from COCO, OpenImages, and Clock Movies; the authors suggest real-world clock images and more training could narrow the gap to a specialized model.
  • Clock reading: Synthetic clock training qualitatively generalizes to more complex clock questions and image captioning.The authors illustrate this behavior with an example in the main paper’s Figure 1.
  • Pointing: Molmo demonstrates superior pointing capability, but pointing performance declines when training and test-time crop counts differ.The evaluation set contains 493 manually verified image-pointing question pairs, with segmentation masks generated from ground-truth points.
  • High-resolution fine-tuning: 76.9 to 77.2: increasing training crops from 12 to 36 yields a slight improvement on the 11-avg metric.The high-resolution training comparison varies crop counts at training and inference.
  • High-resolution fine-tuning: 88.5 →87.7 for CountBenchQA and 85.2 →73.9 for PixMo-Count: increasing crops only at inference degrades counting performance.The default model therefore uses 12 crops during both training and counting inference.
  • High-resolution fine-tuning: Brief high-resolution fine-tuning restores counting performance with 36 test-time crops while preserving overall 11-avg performance.The fine-tuned model matches the counting performance of a model trained directly with 36 crops.
  • Text-only benchmarks: Multimodal fine-tuning causes Molmo-7B-D to lose some text-only knowledge, while adding text-only Tulu 39 data improves mathematical reasoning, programming, and average multimodal performance.A 10% down-sampled mixture performs better on most text-only tasks than using the full text-only dataset.

E.1. Discussion of Main Paper Ablations

The ablations show that vision-encoder choice, image resolution, dropout, length conditioning, and data composition materially affect Molmo’s performance. Results also favor PixMo-Cap over distilled alternatives and indicate that optimizer and training details matter.

  • Vision encoder: MetaCLIP, SigLIP, and other web-scale vision encoders perform similarly, while MetaCLIP uniquely makes every Molmo component and dataset open when paired with OLMo.The authors retained OpenAI CLIP as the default because MetaCLIP was evaluated too late for retraining.
  • Vision encoder: DINOv2 performs only slightly worse than vision-language-supervised encoders, with its variant winning 45% of non-tie user-study comparisons.The standard Molmo-7B-D configuration wins the remaining 55% of non-tie matches.
  • Image resolution: More training and testing crops generally improve results, but captioning and counting degrade when test-crop counts differ from training-crop counts.Document-heavy tasks can benefit from more inference crops than training crops; captioning and counting use matched crop counts.
  • Regularization and conditioning: Text-token-only dropout improves captioning, while length conditioning significantly improves captioning and also benefits downstream tasks.The restricted dropout is intended to encourage reliance on vision tokens rather than previous text tokens.
  • Pre-training data: Using ShareGPT4V/o instead of PixMo-Cap performs worse on both reported metrics even when data scale is approximately controlled.Captioning PixMo-Cap with GPT-4o instead produces strong results, which the authors associate with PixMo-Cap’s more diverse image distribution and GPT-4o captioning improvements.
  • Fine-tuning data: The full fine-tuning mixture reaches 76.8% on 11 benchmarks versus 72.2% for academic datasets alone, with PixMo-Docs and counting data explaining much of the gap.Other PixMo fine-tuning datasets mainly add skills and improve conversational user preferences.

F. Data Details

The data details describe PixMo-Points statistics and the infrastructure used to generate synthetic PixMo-Docs data. They also document ablation-table metadata and the use of text-only language models in the generation pipeline.

  • Model ablations: The vision-encoder-layer ablation compares concatenating features from multiple ViT layers with using features from only one layer.The compared layers are near the end and toward the middle of the ViT.
  • Model ablations: The gradient-normalization ablation compares separate component-wise normalization, global normalization, and different pre-training versus fine-tuning schemes.The components are the ViT, connector, and LLM.
  • Model ablations: Table 15 reports additional model ablations, including vision-encoder layers, learning-rate warmup, and gradient normalization.The table caption identifies the gray entries as defaults.
  • PixMo-Docs generation: PixMo-Docs uses text-only LLM coding capabilities to generate rendering programs, then uses another LLM to construct instruction-tuning datasets from those programs.The framework targets text- and figure-heavy images.
  • PixMo-Docs generation: The generation framework supports seven languages or rendering libraries for charts, tables, diagrams, and documents.The listed tools include Matplotlib, Plotly, LaTeX, HTML, Vega-Lite, Mermaid, and Graphviz.

G. Dataset Examples

The examples section catalogs visual samples from PixMo datasets and situates them among prior VLM, synthetic-data, and grounding approaches. It highlights the breadth of document, pointing, and conversational data formats.

  • Dataset examples: The examples include randomly selected images from PixMo-Cap, AskModelAnything, Points, explanation-augmented Points, CapQA, Clocks, Count, and PixMo-Docs.Points are shown as pink dots and prompts are shown in bold.
  • Dataset examples: The appendix examples cover PixMo-Cap, AskModelAnything, Points, Points with explanations, CapQA, Clocks, Count, and several PixMo-Docs formats.The examples include charts, tables, diagrams, and other documents.
  • Dataset examples: PixMo-Docs uses code as an intermediate representation, while PixMo-CapQA derives questions and answers from detailed captions rather than direct proprietary-VLM annotation.The paper positions these choices within efforts to build more open VLM training data.
  • Dataset examples: PixMo-Points uses human-annotated points for arbitrary referring expressions, including multiple objects and no-target cases, rather than segmentation masks.The approach is described as more efficient than collecting segmentation masks and broader than several prior grounding datasets.
  • Dataset examples: PixMo-Docs examples span charts, tables, diagrams, and other document images generated through diverse synthetic pipelines.The related-work discussion contrasts this breadth with prior synthetic chart-generation methods that often support only one or two chart types.
Loading 2409.17146v2…