Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

• The introduction motivates sensory grounding by arguing that sensory experience supports understanding, real-world action, and decision-making.Source: §1 · p. 3
• Cambrian-1 uses MLLM instruction tuning as a vision-centric protocol to systematically evaluate visual representations, addressing underexplored vision design choices in multimodal models.Source: §1 · p. 3
• The study warns that early reliance on language may shortcut weak visual representations, while existing benchmarks may underserve real-world visual grounding needs.Source: §1 · p. 3
• Traditional linear-probe and fine-tuning evaluations are saturated; CV-Bench converts traditional vision benchmarks into VQA to assess diverse perception challenges.Source: §1 · p. 3
• Its five pillars span vision encoders, a spatially aware token-reducing connector, balanced public instruction data, tuning recipes, and benchmark analysis plus CV-Bench.Source: §1 · p. 4
• Cambrian-1 reports state-of-the-art MLLMs with top performance across diverse benchmarks and visual-centric tasks, and releases weights, code, datasets, and recipes.Source: §1 · p. 4

Large Language Model

• LLMs underpin MLLMs and can improve benchmark performance, but LLM-driven gains may bias evaluation away from visual perception; MMMU is cited as an example of this concern.The section notes improvements on MMMU [144] and AI2D alongside broader LLM-backbone research [10].Source: §2 · pp. 4–5

Visual Encoder

• Most MLLMs use language-supervised CLIP models, while the authors systematically examine self-supervised, segmentation-, depth-, and diffusion-based vision backbones and model ensembles.These alternatives learn representations from visual signals rather than language supervision.Source: §2 · p. 5

Multimodal Connector

• Connectors map visual features into LLM token space; MLP projectors are effective but scale poorly with resolution, motivating designs that handle high-resolution images with fewer tokens.LLaVA-Next requires 2880 visual tokens for one 672px image; the section also identifies resamplers and Q-Formers as connector approaches.Source: §2 · p. 5

Instruction Tuning Data

• Because visual instruction data is scarce online, the authors collect VQA benchmarks and interaction data, study data balancing and category mixtures, and develop an internet data collection engine.Prior work converted VQA benchmarks into instruction-tuning data and reported marked MLLM performance improvements [37] [81] [99].Source: §2 · p. 5

Instruction Tuning

• Using pretrained models, the authors study debated tuning choices, including connector pretraining before joint fine-tuning and whether to freeze the vision encoder.Some proprietary models instead explore end-to-end training from scratch.Source: §2 · p. 5

Evaluation & Benchmarking

• Existing benchmarks cover varied capabilities, but the authors advocate capability-focused aggregates and introduce a vision-centric benchmark from classic vision tasks to address few images and high evaluation variance.They also analyze how well existing benchmarks measure multimodality.Source: §2 · p. 5

3.1. Analyzing the Benchmarks

• Existing benchmarks often fail to measure vision-centric capabilities: some show less than a 5% gap with vision enabled, while others exhibit language bias or require strong visual grounding.The analysis also clusters benchmarks into General, Knowledge, Chart & OCR, and Vision-Centric groups, and finds current vision-centric benchmarks are small and omit depth and spatial awareness.Source: Figure 3 · p. 6, §3.1 · p. 7

3.2. Cambrian Vision-Centric Benchmark (CV-Bench)

• CV-Bench repurposes standard vision benchmarks into manually inspected VQA questions, providing 2,638 examples—3.5× more than RealWorldQA and 8.8× more than MMVP—to test 2D and 3D understanding.Its tasks assess spatial relationships and object counting in 2D, and depth order and relative distance in 3D.Source: §3.2 · pp. 7–8

3.3. Instruction Tuning Recipes

• Two-stage training and more adapter data improve performance across domains; unfreezing the vision encoder generally helps, with language-supervised models benefiting broadly and self-supervised models especially on vision-centric benchmarks.Unfreezing brings only marginal change on knowledge benchmarks and adds computational overhead.Source: §3.3 · p. 9

3.4. MLLMs as a Visual Representation Evaluator

• Language-supervised models outperform non-CLIP models across benchmark categories, especially chart and OCR tasks, while high-resolution encoders and ConvNets boost chart and vision-centric performance, where DINOv2 can outperform some language-supervised models.The authors suggest CLIP’s text-rich training data may explain its chart and OCR advantage; they also find higher resolution is neutral on general VQA and knowledge VQA.Source: §3.4 · pp. 10–11, Figure 6 · p. 10
• With 5M instruction-tuning data and an unfrozen vision backbone, the DINOv2-based MLLM surpasses the CLIP model trained with 0.7M data and narrows the gap to CLIP models under the 5M setting.This comparison tests whether continued fine-tuning can reduce the advantage of language supervision.Source: §3.4 · p. 11, Figure 7 · p. 11

3.5. Combining Multiple Vision Encoders

• Combining multiple encoders through feature concatenation improves performance, with DINOv2 especially helping vision-centric and OCR benchmarks; simple interpolation and equal-weight concatenation can lose information and limit flexible use of high-resolution features.The study interpolates each encoder to 576 visual tokens before concatenating features.Source: §3.5 · p. 12

Spatial inductive bias

• Learnable latent queries cross-attend to multiple vision encoders’ features as keys and values, aggregating them into fewer tokens [37].Each encoder’s output feature map serves as context for the query grid.Source: §4 · pp. 12–13
• SVA aligns each query token with a corresponding feature-map sub-region, preserving spatial structure while accommodating encoders with varying resolutions.The query is an L × L grid of learnable latent tokens, and encoder feature-map resolutions are multiples of L.Source: §4 · pp. 12–13

Multi-layer vision aggregation

• Because high-resolution inputs or more encoders increase the context each token must aggregate, SVA repeats cross-attention across LLM layers to retain access to uncompressed visual features.The repeated aggregation is intended to prevent information loss during compression.Source: §4 · pp. 13–14

Evaluation with multiple vision encoders

• SVA outperforms concatenation [127] and Re-sampler [11] [73] across benchmark categories, with a significant improvement in OCR & chart tasks.The evaluation uses four vision encoders and a Vicuna-1.5-7B base LLM.Source: Table 4 · p. 14, §4 · p. 14

Single-encoder token compression

• With one encoder, SVA outperforms direct interpolation + MLP, C-Abstractor [21], and LDPv2 Projector [35] across categories, especially OCR & Chart and Vision-Centric tasks.The comparison compresses OpenAI CLIP ViT-L from 576 tokens to 36 and omits multi-layer aggregation.Source: §4 · p. 14, Table 5 · p. 15

Hyperparameter ablations

• Increasing SVA capacity through D (cross-attention layers) or G (query groups), and adding cross-attention within LLM layers, improves OCR & chart performance.D and G control aggregation capacity; cross-attention layers within the LLM use D = 1 and G = 1.Source: §4 · pp. 14–15, Table 6 · p. 15

5.1. Data Collection

• The collection combines visual interaction data across seven categories with a small volume of language-only instructions to help preserve conversational abilities.Categories are General conversation, OCR, Counting, Code, Math, Science, and Language-only; the collection draws on existing visual benchmarks and datasets.Source: §5.1 · p. 16
• A targeted data engine generates 161k science-related data points—400% more than prior combined sources—by turning image-caption pairs into VQA instruction Q&A.It selects fields and topics, searches reliable sources such as Wikipedia, and uses an LLM to generate questions and answers from image captions.Source: §5.1 · p. 16
• Cambrian-10M contains approximately 9784k instruction-tuning data points, forming a diverse pool for this work and future research.The dataset composition is visualized in Figure 9.Source: §5.1 · p. 16

5.2. Data Curation

• For Cambrian-10M, source-count thresholds between 250k and 350k work best, with an observed elbow effect in the data-balancing experiment.The study compares thresholds of 150k, 250k, 350k, and 450k.Source: §5.2 · p. 16, Table 7 · p. 16
• At a fixed dataset size of 1.35M, data ratios materially affect performance: OCR gains with more OCR data, but excessive OCR harms general VQA and vision-centric performance.Knowledge-intensive tasks often need a mix of OCR, chart, reasoning, and general perception data; very low science ratios perform poorly.Source: §5.2 · p. 17, Figure 11 · p. 17
• Cambrian-7M improves performance despite having fewer samples, and its balanced curation enables more efficient training.The paper describes Cambrian-7M as smaller but higher quality; Table 8 reports benefits from scaling to Cambrian-10M and further gains across categories after curation.Source: §5.2 · p. 17, Table 8 · p. 18

5.3. Alleviating the “Answer Machine Phenomenon” via System Prompts

• Short-answer-heavy VQA tuning can create an “answer machine” that gives curt replies, motivating MLLM interaction evaluations such as Chatbot Arena [34].The paper argues that benchmark success alone does not ensure meaningful, natural interaction.Source: §5.3 · pp. 18–19
• System prompts preserve benchmark performance while improving longer conversational responses and reasoning tasks through chain-of-thought prompting [134].Prompts specify when a short answer is appropriate, helping the model retain strong question answering while producing more engaging responses.Source: §5.3 · pp. 18–19, Figure 12 · p. 18

6. State of the Art Performance

• Cambrian-1 uses LLaMA-3-Instruct-8B, Vicuna-1.5-13B, and Hermes-2-Yi-34B backbones with a four-model vision component integrated through SVA.The vision encoders are CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518; the connector is pre-trained on 2.5M adapter data and instruction-tuned with the Cambrian-7M data mix.Source: §6 · pp. 19–20
• Cambrian-1 surpasses LLaVA-NeXT and Mini-Gemini on reported benchmarks using 576 image tokens—about 1/5 of their tokens—and achieves comparable performance to proprietary models on several benchmarks.The paper attributes its strength on high-resolution image tasks to SVA.Source: §6 · p. 20
• The authors use fuzzy matching to judge model outputs, treating answers such as “Apple” as correct when the reference is “(a) Apple,” and report reliable judgments after GPT-3.5 ablation.They emphasize post-processing outputs and assessing accuracy.Source: §6 · p. 20
• GPT-4’s GQA test performance is low, possibly because other models were trained on GQA while GPT-4’s training set is unclear.Source: §6 · p. 20

• The authors advocate using MLLMs to evaluate visual representations because existing benchmarks are saturated and inadequately reflect real-world perception challenges.They identify a gap between language-supervised and self-supervised models; language-supervised features can behave like a bag-of-words [126], motivating vision-only models that leverage large-scale datasets [86] while preserving visual grounding [127].Source: §7 · p. 21
• Cambrian-1 can match or outperform proprietary models on many existing benchmarks, but benchmark-only development risks producing an “answer machine” lacking practical interaction capabilities.The authors argue that evaluation should also address conversational ability, creativity, reliability, and user experience, and that alignment with human and societal needs remains an evolving process.Source: §7 · p. 21
• Cambrian-1 uses a moderate number of visual tokens and lacks an any-resolution strategy [30] [78], limiting its handling of ultra-high-resolution images and extreme aspect ratios.For tasks such as V*Bench [137], the authors suggest that increasing image resolution and visual-token count could enable an HD version.Source: §7 · p. 21
• The authors identify reinforcement learning from human or environmental feedback as a promising post-training alternative to supervised fine-tuning, particularly for improving decision-making.They point to recent LLM and MLLM advances as suggesting it may surpass supervised fine-tuning’s limits [38] [105] [111] [142] [147].Source: §7 · p. 21
• Cambrian-10M provides a rich pool for studying MLLM fine-tuning data curation, but the authors describe their higher-quality curation pipeline as an initial step with room to improve.They hope the dataset and work can serve as a foundation for further research into more efficient and effective instruction tuning.Source: §7 · p. 21
• Cambrian-1 provides model weights, open-source code, datasets, and detailed training and evaluation recipes to support open research.The authors report training with TPU-V4 [61], FSDP [151], and TorchXLA, and share infrastructure experiences, technical solutions, implementation, and tutorials for large-scale training.Source: §7 · p. 21

A. Training, Infrastructure, and Implementation

• Final Cambrian-1 models were trained in less than 4 days on TPU-V4-512, while Section 3.4 experiments took less than 24 hours on TPU-V4-128.All models were trained on TPU-V4 pods; evaluation used NVIDIA A6000, A100, and H100 cards.Source: A. Training, Infrastructure, and… · p. 30
• All models were trained using TPU-V4 pods, and evaluation was conducted on NVIDIA A6000, A100, and H100 cards.This distinguishes the training hardware from the evaluation hardware.Source: A. Training, Infrastructure, and… · p. 30
• TPU training with TorchXLA and FSDP [151] required static-graph rewrites, model-resuming support, tutorial fixes, and workarounds for loading models over 30 billion parameters.The authors cite TPU-V4’s 100GB memory constraints and missing or outdated library features as challenges.Source: A. Training, Infrastructure, and… · p. 30
• The authors rewrote TorchXLA FSDP sharding to load very large models, implemented model resuming, and modified Transformers FSDP and gradient checkpointing for large-scale training.They committed to open-sourcing the codebase and publishing a comprehensive tutorial.Source: A. Training, Infrastructure, and… · p. 30

B. Analyzing the Benchmarks

• MMMU was less correlated with other benchmarks in measuring model performance, so the authors grouped it with knowledge-based QAs based on question type.The analysis evaluated one-stage, two-stage finetune-only, and hybrid models; Figure 15 shows the benchmark correlation matrix.Source: MLLM Benchmark Performance Confusion… · p. 31, Figure 15 · p. 31
• The CV-CB filtering figure describes reformulating classic 2D and 3D CV benchmarks as Q&A questions to evaluate MLLM visual capabilities.Figure 16 is labeled “CV-CB Benchmark Filtering.”Source: Figure 16 · p. 32

C. Cambrian Vision-Centric Benchmark (CV-Bench)

• CV-Bench questions are generated from COCO and ADE20K for 2D tasks and Omni3D for 3D tasks, then manually filtered by experts.Reviewers accept, modify, or reject each query, including ambiguous examples.Source: C. Cambrian Vision-Centric Benchmark… · p. 32
• Expert filtering finalized CV-Bench at 2,638 image queries, with improved accuracy and reliability.Queries were accepted, modified, or rejected after manual review.Source: C. Cambrian Vision-Centric Benchmark… · p. 32
• Reviewers reject ambiguous counts and relative-distance questions, and correct cases with clear annotation errors.Sparse annotations can make counts too low; multiple instances can make an object's relative location ambiguous.Source: C. Cambrian Vision-Centric Benchmark… · p. 32
• The benchmark averages task accuracies to give equal importance to 2D and 3D tasks.The described overall metric is the average of the accuracies obtained from these tasks.Source: C. Cambrian Vision-Centric Benchmark… · p. 32

D. Vision Models in MLLMs

• The authors use an MLLM as an interface to evaluate different vision-model capabilities and report model selection, full results, and data split.Source: D. Vision Models in MLLMs · p. 33

D.1. Details of Vision Models

• Thirteen selected vision models span language-supervised, self-supervised, class-supervised, and other model categories.The categories include CLIP-family models, DINOv2, I-JEPA, MAE, MoCo v3, ImageNet22K ViT, Stable Diffusion, SAM, and MiDaS, with bibliography markers given in the passage.Source: §D.1 · p. 33
• The catalog specifies backbone architecture, resolution, token count, and hidden dimension; models with many output patches are interpolated to the specified token length.Table 10 marks interpolation with I.Source: §D.1 · p. 33, Table 10 · p. 33

D.2. Full Results of Different Vision Backbones

• With 0M adapter data plus 737K instruction-tuning data, language-supervised models show superior performance; notably, OpenCLIP ConvNeXT-XXL@1024 surpasses all other models on DocVQA with over 12%, indicating potential to handle OCR-related benchmarks.Source: §D.2 · p. 34
• DFN-CLIP ViT-H/14@378 rises from 36.21 to 49.94 with 0.5M adapter data, while SigLIP ViT-SO400M/14@384 increases from 46.79 to 49.72 to 53.09 across 0M, 0.5M, and 1.2M splits.The adapter splits use no initial adapter phase, 558K LLaVA-1.5 data, or ShareGPT4V-PT data, respectively.Source: §D.2 · pp. 34–35, Table 12 · p. 34, Table 13 · p. 35
• Unfreezing vision encoders improves most models on most benchmarks, especially OCR, chart, and vision-centric tasks; ConvNeXt shows substantial gains.The figure visualizes percentage changes between frozen and unfrozen encoders.Source: §D.2 · p. 36, Figure 17 · p. 37
• After 5M instruction tuning, the performance gap between DINOv2 and CLIP models narrows on general, knowledge, and vision-centric benchmarks.The experiments use 1.2M adapter data; the passage points to Table 15, while the supplied 5M-results table is Table 16.Source: §D.2 · p. 36, Table 16 · p. 36

D.3. Model Ensemble

• Adding a vision-only SSL model to the ensemble improves vision-centric benchmark performance while maintaining strong capabilities in other categories.The ensemble interpolates each encoder’s output to 576 tokens, concatenates outputs along the feature dimension, and uses a larger MLP to project them into the LLM token space.Source: §D.3 · p. 38

E.1. Catalog of Visual Instruction Data

• The visual instruction-data catalog groups datasets by general conversation and VQA, OCR, counting, knowledge, and language-only focus.The catalog summarizes datasets and their references.Source: §E.1 · p. 39

E.2. Additional System Prompts used in Cambrian Data

• Because Cambrian examples require different response formats, the authors add formatting prompts to datasets that do not already specify them.The paper lists the additional prompts in Table 18.Source: §E.2 · p. 39, Table 18 · p. 40

Comprehensive Implementation Details of the Data Engine

• The knowledge-data engine generates topics, retrieves and parses web pages into image-caption-text tuples, filters and standardizes images, then uses an LLM to create image-grounded VQA pairs.Across 30 fields, it generated 3,660 topics and retrieved 36,600 webpages by querying 10 links per topic; it filters texts under 50 words, downloads high-resolution originals, converts formats to PNG, and generated 165K samples.Source: Comprehensive Implementation Details… · pp. 41–42

E.4. Full results on data curation experiment

• Filtering noisy sources makes exponential tails less pronounced; t=250k has the highest average across benchmarks, while t=250k and 350k perform best on many individual benchmarks.The unfiltered pool is dominated by noisy, unbalanced sources such as CLEVR and DVQA.Source: §E.4 · p. 44
• Experiment 5 achieves well-rounded performance, highlighting the importance of choosing a data ratio that balances different aspects of MLLM performance.The full data-ratio study results are presented in Table 20.Source: §E.4 · p. 44

E.5. 737K and 5M Mixes

• The 737K mix adds 15,501 AI2D, 14,999 DocVQA, and 13,000 DVQA examples to LLaVA 665K; the 5M mixes filter Cambrian-10M data at t=150k.The 737K mix covers all categories for studying visual representations efficiently.Source: §E.5 · p. 44

E.6. Test Image Leakage in Visual Instruction Training Data

• The data engine uses difference hashing to compare training and test images and count direct image matches as a potential leakage concern.The analysis reports collisions in Table 21.Source: §E.6 · p. 45
• Across fifteen datasets, the data engine finds 32 overlapping test images (0.06%); Cambrian10M has 7,244 matches versus LLaVA-665k’s 1,034 despite being 15x larger.The authors attribute overlap likely to natural reuse of images across benchmark datasets, rather than targeted test-set inclusion.Source: §E.6 · p. 45, Table 21 · p. 45
• Image overlap alone does not establish test-set training because multimodal examples are image-text pairs; the authors encourage research on image-only leakage effects.They suggest that studying its influence could clarify generalization boundaries and inform dataset-construction practices.Source: §E.6 · p. 46

E.7. Broader Impacts

• The Cambrian dataset’s pronoun distribution is 38.35% male, 17.99% female, and 43.66% neutral.The appendix describes this as a preliminary distribution analysis.Source: §E.7 · p. 46
• Bias mitigation through rebalancing or fairness constraints may reduce bias but risks weakening generalization or introducing new biases, and multimodal interactions make MLLM bias especially complex.The authors argue that openness in model development and data curation can accelerate research on understanding and mitigating potential harms.Source: §E.7 · p. 46

F. Implementation Details

• Final Cambrian models use 2.5M adapter data, including 1.2M ShareGPT4V captioning examples and 1.2M captioning examples used in MiniGemini.The passage reports these component counts as stated.Source: F. Implementation Details · p. 46
• SVA adds learnable positional encodings when k_m > 1 and augments each cross-attention query with a globally pooled vision feature.Most encoder feature maps are interpolated to 576×576; ConvNext’s four stages are interpolated to 96 × 96 and concatenated by channel.Source: F. Implementation Details · p. 46
• Final models use D = 3 and G = 1, with cross-attention strides of 3, 4, and 9 for Cambrian-8B, 13B, and 34B, respectively.The paper describes these placements as a performance-efficiency tradeoff; Section 4 experiments instead use stride 3.Source: F. Implementation Details · p. 46
• Attention contributions vary by image category: encoder contributions are relatively uniform on GQA, ConvNext and SigLIP increase on DocVQA, and SigLIP rises while DINOv2 decreases on ScienceQA.The analysis evaluates Cambrian-8b on GQA, DocVQA, and ScienceQA; Table 22 presents the attention distribution study.Source: F. Implementation Details · pp. 46–47, Table 22 · p. 47
• Unfreezing visual encoders slows fine-tuning by approximately 50–55%; freezing supports quicker early iterations, while unfreezing is recommended for best performance after setup optimization.The recommendation reflects the tradeoff between computational overhead and performance.Source: F. Implementation Details · p. 47

G.1. System Prompts Used in Evaluation

• The paper provides model system prompts and benchmark-evaluation prompts in Tables 25 and 26 to support reproducibility of results.The authors hope sharing these prompts encourages future work to improve benchmark-result reproducibility.Source: §G.1 · p. 47

G.2. Ablation Study on Fuzzy Matching Vs LLM Judgement

• LLM graders are prompt-sensitive, and few-shot examples significantly improve grading accuracy; the study uses GPT-3.5-turbo and GPT-4-turbo.The authors compare fuzzy-matching accuracy with accuracy obtained using an LLM grader.Source: §G.2 · p. 47
• The example grading prompt requires CORRECT or INCORRECT judgments and tolerates a 0.05 relative error for numerical answers.It presents answer and ground-truth pairs as grading examples.Source: §G.2 · pp. 47–48
• Fuzzy matching provides reliable results, with similar accuracies to LLM grading on benchmarks requiring matching.The authors recommend a more capable grader, such as GPT-4-turbo, for benchmarks with more subjective responses, including numbers and words.Source: §G.2 · p. 48, Table 24 · p. 49

H. Potential Misuse & Mitigation Strategies

• Cambrian-1 could generate misleading image descriptions that support false narratives or social-media manipulation of public opinion during elections or other critical events.The paper identifies misinformation as a potential misuse risk.Source: H. Potential Misuse & Mitigation… · p. 51
• Cambrian-1 may hallucinate information unsupported by facts or input, creating danger when users assume its outputs are entirely accurate.The paper warns that this risk is shared with large language models generally.Source: H. Potential Misuse & Mitigation… · p. 51
• Users should critically assess and verify outputs, especially for sensitive or high-stakes uses, and content filtering can flag harmful or misleading material before dissemination.The paper also urges users to cross-check and validate critical outputs.Source: H. Potential Misuse & Mitigation… · p. 51