PaperlayerView exact revision on arXiv

Source-linked AI summary

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, Saining Xie

arXiv:2406.16860v2cs.CV

TL;DR

Cambrian-1 addresses limited exploration of vision components for multimodal LLMs and the saturation of conventional visual-representation evaluations. It uses MLLMs as an evaluation interface, develops vision-centric benchmarking and model components, and reports top performance across diverse benchmarks while releasing open research resources.

  • Problem

    Vision-component design in MLLMs is insufficiently explored, while traditional visual-representation evaluations are becoming saturated and do not reflect diverse real-world perception challenges.

  • Method

    Cambrian-1 evaluates 23 vision backbones through MLLM instruction tuning, analyzes existing benchmarks, introduces CV-Bench and a dynamic spatially-aware connector, and studies balanced instruction-tuning data.

  • Results

    Cambrian-1 achieves top performance across diverse benchmarks, excels in visual-centric tasks, and can match or outperform proprietary models on many existing benchmarks.

  • Takeaways & Limitations

    The release provides model weights, open-source code, datasets, tools, and detailed training and evaluation recipes for research on multimodal systems and visual representation learning.

  • Takeaways & Limitations

    The current model does not use any-resolution processing and may require more visual tokens for ultra-high-resolution or extreme-aspect-ratio images.

Abstract

from arXiv · show

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

1. Introduction

Cambrian-1 studies MLLMs from a vision-centric perspective, motivated by insufficient exploration of vision components and the need for evaluation that reflects real-world visual grounding. It combines systematic representation analysis, a VQA-based benchmark, connector and data-design contributions, and open resources.

  • Vision-component design in MLLMs remains insufficiently explored and disconnected from visual representation learning research.
  • Visual grounding matters for real-world perception, action, and decision-making, while language shortcuts may compensate for weak visual representations.
  • Traditional visual-representation evaluations are saturated and do not capture diverse real-world perception challenges.
  • CV-Bench reformulates traditional vision benchmarks into visual question answering for vision-centric MLLM evaluation.
  • Cambrian-1 studies visual representations, connector design, instruction-tuning data and recipes, and benchmarking as five coordinated pillars.
  • The released Cambrian-1 models achieve top performance across diverse benchmarks and excel on visual-centric tasks, alongside open weights, code, datasets, and recipes.

2. Multimodal LLMs: Preliminaries and Related Work

The paper frames MLLMs as systems whose language models, visual encoders, connectors, data, tuning, and evaluation interact in complex ways. It focuses on broadening visual encoders, improving connector and data practices, and developing more vision-centric benchmarking.

  • MLLM research spans language models, visual encoders, multimodal connectors, data curation, instruction tuning, and evaluation, whose interactions are challenging to understand.
  • Large Language Model: Strong language-model performance can dominate multimodal benchmarks, motivating evaluations that more genuinely assess multimodality.
  • Visual Encoder: The study systematically examines vision backbones spanning language-supervised, self-supervised, segmentation, depth-supervised, and diffusion-based representations.
  • Multimodal Connector: Visual encoders require connectors to map their representations into LLM token space, but MLP projectors face token growth that scales quadratically with image resolution.
  • Instruction Tuning Data: The authors collect VQA and visual-interaction data, study category balancing and mixtures, and develop an internet data-collection engine for instruction tuning.
  • Instruction Tuning: They use pretrained models to study instruction-tuning recipe choices, including connector pretraining and whether to freeze the vision encoder.
  • Evaluation & Benchmarking: The benchmarking analysis categorizes existing evaluations and introduces a vision-centric benchmark by reformulating classic vision tasks.

3. Evaluating Visual Representations through MLLMs

Cambrian-1 evaluates visual representations through MLLMs, exposing benchmark dependence on language priors and introducing CV-Bench for vision-centric assessment. Experiments compare encoders, tuning recipes, and combinations across categorized benchmarks.

  • Evaluation setup: 23 vision backbones are evaluated with two-stage instruction tuning: 1.2M adapter examples followed by 737K instruction-tuning examples.The study uses a common MLLM interface to compare diverse visual representations.
  • Benchmark analysis: Benchmarks form four clusters—General, Knowledge, Chart & OCR, and Vision-Centric—based on correlations among model performance scores.Principal component analysis identifies these benchmark categories.
  • Benchmark analysis: Most benchmarks do not properly measure vision-centric capabilities, and the few that do contain very few samples.This motivates a dedicated vision-centric evaluation set.
  • CV-Bench: CV-Bench contains 2638 manually inspected examples, 3.5× more than RealWorldQA and 8.8× more than MMVP, by repurposing standard vision benchmarks into multimodal questions.It evaluates fundamental 2D and 3D visual understanding, including spatial relationships, object counting, depth order, and relative distance.
  • Instruction tuning recipes: Two-stage training is beneficial, and more adapter data further improves results.The finding summarizes experiments varying connector pretraining data before joint instruction tuning.
  • Visual representations: Language-supervised encoders generally outperform non-language-supervised models, while DINOv2 and encoder combinations remain competitive, especially on vision-centric tasks.High-resolution encoders particularly improve chart and vision-centric performance; self-supervised methods can narrow the gap with sufficient data and tuning.

4. Spatial Vision Aggregator (SVA): A New Connector Design

SVA is a spatially aware connector that aggregates features from multiple vision encoders while preserving spatial structure and reducing visual tokens. Its spatial bias and repeated interaction with LLM layers improve aggregation and condensation of high-resolution visual information.

  • Design motivation: SVA uses learnable latent queries and cross-attention to aggregate multiple vision features while preventing interpolation-related information loss.The design introduces spatial inductive bias and supports repeated aggregation across LLM layers.
  • Spatial aggregation: A learnable C-dimensional token is repeated into an L×L query grid, while encoder feature maps provide cross-attention keys and values.Each encoder feature map has resolution m_kL × m_kL × C, enabling compatible spatial aggregation.
  • Spatial inductive bias: Each query token aligns with a specific sub-region of every encoder’s feature map, preserving spatial structure during cross-attention.The alignment lets tokens aggregate corresponding regions across vision encoders with varying resolutions.
  • Token compression: Aggregating multiple visual features into a single token reduces the number of visual tokens passed to the LLM.This compression targets high-resolution and multi-encoder settings.
  • Deep interaction: SVA repeatedly aggregates vision features across LLM layers, allowing continued access to uncompressed visual information.The number of cross-attention layers and query groups controls aggregation capacity and patterns.
  • Evaluation: SVA consistently outperforms other aggregation baselines, while larger capacity improves performance in ablations.Experiments also assess spatial-based connectors and high-resolution OCR and chart understanding.

5. Instruction Tuning Data for Training MLLMs

Cambrian-1 develops a curated visual instruction-tuning pipeline that collects diverse data, addresses category imbalance, and improves performance despite using fewer samples. It also studies system prompts to preserve conversational ability alongside benchmark performance.

  • 5.2. Data Curation: A threshold between 250k and 350k works best in the data-balancing experiment, exhibiting an elbow effect.
  • 5.1. Data Collection: The collection pipeline combines existing VQA, OCR, and visual-interaction datasets with a targeted internet engine for scarce scientific domains.
  • 5.1. Data Collection: 161k science-related data points were generated, representing 400% more data than the previous combined sources.
  • 5.1. Data Collection: Cambrian-10M contains approximately 9784k instruction-tuning data points collected from diverse sources for current and future MLLM research.
  • 5.2. Data Curation: Cambrian-10M has unbalanced category ratios, motivating experiments on threshold-based balancing and data-mixture composition.
  • 5.2. Data Curation: Different instruction-tuning data ratios materially affect overall performance, with exp-5 identified as the most optimal mixture at controlled size 1.35M.
  • 5.2. Data Curation: Cambrian-7M is a smaller, higher-quality dataset whose careful curation improves performance across categories while making training more efficient.
  • 5.3. Alleviating the “Answer Machine Phenomenon” via System Prompts: Adding system prompts improves chat ability while retaining strong question answering, addressing short-response behavior associated with benchmark-focused tuning.

6. State of the Art Performance

Cambrian-1 combines multiple language backbones, vision encoders, and the Spatial Vision Aggregator in a large-scale instruction-tuning pipeline. It surpasses open-source models and remains competitive with proprietary systems, particularly on high-resolution visual tasks.

  • Cambrian-1 trains models with LLaMA-3-Instruct-8B, Vicuna-1.5-13B, and Hermes-2-Yi-34B language backbones.
  • The vision component combines four encoders through the Spatial Vision Aggregator after connector pre-training on 2.5M adapter data and instruction tuning on Cambrian-7M.
  • Cambrian-1 surpasses open-source models such as LLaVA-NeXT and Mini-Gemini while achieving comparable performance to leading proprietary models on several benchmarks.
  • 576 visual tokens enable strong high-resolution image performance, compared with 2880 tokens used by LLaVA-NeXT and Mini-Gemini.
  • The evaluation discusses fuzzy matching for output accuracy and notes that GPT-4 performance on GQA may reflect unclear training-set overlap.

7. Discussion

Cambrian-1 positions MLLMs as interfaces for evaluating visual representations and reports strong benchmark performance, while emphasizing that current evaluation and data curation remain incomplete. The model also has a stated limitation on ultra-high-resolution inputs.

  • MLLMs are proposed as interfaces for evaluating visual representations because traditional benchmarks are saturated and underrepresent complex real-world perception challenges.
  • Cambrian-1 can match or outperform proprietary models on many existing benchmarks, but benchmark success alone may produce an “answer machine” lacking practical interaction abilities.
  • The paper argues that MLLM development should continue addressing conversational ability, creativity, reliability, and overall user experience beyond benchmark scores.
  • The current model uses moderate visual-token counts and does not support any-resolution processing for ultra-high-resolution or extreme-aspect-ratio images.
  • The paper identifies reinforcement learning from human or environmental feedback as a possible post-training direction beyond supervised fine-tuning.
  • Further improvement of the data-curation pipeline remains an open direction despite Cambrian-10M providing a rich pool for studying MLLM fine-tuning.
  • Cambrian-1 releases model weights, code, datasets, and training and evaluation recipes to support open research in visual representation learning and multimodal systems.

A. Training, Infrastructure, and Implementation

The work combines large-scale TPU training infrastructure with benchmark-correlation analysis and a reformulated computer-vision benchmark. Its implementation relies on substantial sharding and parallelism adaptations for TorchXLA and FSDP.

  • A. Training and Infrastructure: Final Cambrian-1 models are trained in less than 4 days on a TPU-V4-512, while Section 3.4 experiments take less than 24 hours on a TPU-V4-128.
  • A. Training and Infrastructure: TorchXLA with FSDP handles TPU training sharding and parallelism, requiring adaptations because TPUs use static computation graphs.
  • A. Implementation: The implementation rewrites sharding, checkpoint-resume, Transformers FSDP, and gradient-checkpointing components for large-scale TPU training.
  • A. Benchmarking: The correlation matrix is used to analyze and group MLLM benchmarks with respect to different vision backbones.
  • A. Benchmarking: The benchmark correlation analysis finds MMMU less correlated with other benchmarks and assigns it to the knowledge category based on question content.
  • A. Benchmarking: CV-CB reformulates classic 2D and 3D computer-vision benchmarks as question-answering tasks for evaluating MLLM visual capabilities.

C. Cambrian Vision-Centric Benchmark (CV-Bench)

CV-Bench is constructed through semi-automatic generation from established benchmarks followed by expert filtering, producing a verified set of 2D and 3D visual queries. Its overall score averages 2D and 3D accuracies to give them equal importance.

  • Benchmark construction: The benchmark generates task-specific query–answer pairs from COCO, ADE20K, and Omni3D annotations.COCO and ADE20K support 2D-related tasks, while Omni3D supports 3D-related tasks.
  • Benchmark construction: Human experts filter incorrect or ambiguous queries by accepting, modifying, or rejecting generated samples.Manual verification addresses sparse or incorrect annotations before finalization.
  • Benchmark construction: 2,638 image queries remain after the two-stage generation and verification process.The finalized benchmark is intended to improve accuracy and reliability.
  • Evaluation: Overall performance is the average of accuracies from the 2D and 3D task groups.This metric gives equal importance to the two task dimensions.
  • Vision-backbone evaluation: The study evaluates different vision backbones through an MLLM interface and catalogs thirteen models across four representation categories.The categories include language-supervised, self-supervised, class-supervised, and other models.

D.2. Full Results of Different Vision Backbones

The experiments compare vision backbones under different adapter-data and instruction-tuning regimes, including frozen and unfrozen encoders. Results show gains from alignment, increased adapter data, unfreezing, model ensembles, and balanced visual instruction data.

  • Vision-backbone comparison: Thirteen vision encoders are evaluated across language-supervised, self-supervised, class-supervised, segmentation, depth, and diffusion categories.The models are integrated as MLLM vision encoders and tested with multiple adapter-data splits.
  • Vision-backbone comparison: OpenCLIP ConvNeXT-XXL@1024 surpasses the other tested models on DocVQA by over 12% in the 0M adapter-data setting.The result indicates strong performance on an OCR-related benchmark.
  • Adapter-data scaling: 36.21 to 49.94: adding a 0.5M alignment stage improves DFN-CLIP ViT-H/14@378 performance.The comparison uses the 0.5M Adapter Data + 737K Instruction Finetune setting.
  • Adapter-data scaling: 46.79 to 49.72 to 53.09: SigLIP ViT-SO400M/14@384 improves consistently as adapter data increases from 0M to 0.5M to 1.2M.All three values are reported for the 737K instruction-tuning setting.
  • Unfrozen vision encoders: Nearly all models improve on most benchmarks when visual encoders are unfrozen, especially on OCR & Chart and Vision-Centric benchmarks.The percentage changes from frozen to unfrozen encoders are visualized in Figure 17.
  • Model ensembles: Adding a vision-only SSL model improves vision-centric benchmarks while maintaining strong performance in other benchmark categories.The model-ensemble experiment concatenates visual features after interpolating outputs to a fixed token count.

E.4. Full results on data curation experiment

The data-curation experiments examine filtering, data ratios, leakage, and demographic distributions. Filtering balances the data pool, while the authors identify image overlap and pronoun imbalance as relevant considerations for dataset development.

  • Data filtering: Filtering noisy sources such as CLEVR and DVQA reduces pronounced exponential tails in the data distribution.A t value of 250k has the highest average across all benchmarks, while 250k and 350k perform best across many individual benchmarks.
  • Data ratios: Experiment 5 achieves well-rounded performance with its selected data ratio.The data-ratio study emphasizes balancing different aspects of MLLM performance.
  • Training-data composition: The 737K training mix adds 15,501 AI2D, 14,999 DocVQA, and 13,000 DVQA examples to LLaVA 665K.This mix covers all training categories used to study visual representations efficiently.
  • Leakage analysis: 32 test images, or 0.06% of test data across fifteen datasets, match images found by the targeted data engine.The full Cambrian10M dataset contains 7,244 matching test images compared with 1,034 for LLaVA-665k.
  • Leakage analysis: Image overlap does not by itself establish exposure to the exact test image-question pairs in multimodal training.The paper treats image-text pairs as the relevant multimodal data points and calls for future study of image-only leakage.
  • Broader impacts: The reported pronoun distribution is 38.35% male, 17.99% female, and 43.66% neutral.The authors note that biased training data can perpetuate bias and that mitigation may affect generalization or introduce new biases.

F. Implementation Details

Implementation details cover the final adapter-data mixture, SVA configuration, category-dependent attention analysis, and reproducibility materials. The paper also reports a trade-off between unfreezing visual encoders and fine-tuning speed.

  • Model configuration: The final Cambrian models use 2.5M adapter examples combining 1.2M ShareGPT4V captioning data and 1.2M MiniGemini captioning data.The two captioning sources form the reported adapter-data composition.
  • SVA configuration: SVA adds learnable positional encodings and a globally pooled visual feature to guide cross-attention aggregation.Vision feature maps are interpolated before aggregation, with ConvNext handled through its four stages.
  • SVA configuration: Cambrian-8B, Cambrian-13B, and Cambrian-34B use cross-attention strides of 3, 4, and 9, respectively.The final models use D=3 and G=1 while inserting multiple cross-attention layers in the LLM.
  • Attention analysis: Attention contributions vary by image category: ConvNext and SigLIP receive greater influence on high-resolution, text-heavy DocVQA images.On GQA, contributions are relatively uniform; the study also evaluates ScienceQA as a scientific-image category.
  • Efficiency trade-off: Unfreezing visual encoders slows fine-tuning by approximately 50–55% under fixed computational resources.The authors recommend frozen encoders for quicker early iterations and unfreezing for best performance after setup optimization.
  • Reproducibility: The released system and benchmark prompts are intended to support reproduction of the reported evaluations.The prompts are provided in Tables 25 and 26.

G.2. Ablation Study on Fuzzy Matching Vs LLM Judgement

The study compares fuzzy matching with LLM grading for benchmarks requiring response matching. Fuzzy matching is reliable and produces similar accuracies, while few-shot prompting improves the sensitivity of LLM graders to grading accuracy.

  • The evaluation compares model accuracy under fuzzy matching with accuracy obtained using an LLM as grader.The study conducts an ablation on benchmarks requiring fuzzy matching.
  • Few-shot grading examples significantly improve LLM grading accuracy.The graders used are OpenAI GPT-3.5-turbo and GPT-4-turbo, and grading is sensitive to the prompt.
  • The grading prompt requires CORRECT or INCORRECT judgments and tolerates a 0.05 relative error for numerical answers.Examples cover numerical, yes/no, textual, and year-based answers.
  • Fuzzy matching and LLM grading yield similar accuracies on benchmarks requiring response matching.The comparison is reported in Table 24.
  • The authors recommend a more capable model such as GPT-4-turbo for benchmarks with subjective numerical or word responses.This recommendation follows their ablation comparing fuzzy matching with LLM grading.
Loading 2406.16860v2…