Source-linked AI summary
Pixtral 12B
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, Sophia Yang
TL;DR
Multimodal benchmarks often underrepresent practical, multi-turn use, and evaluation protocols can change reported model performance. Pixtral 12B introduces a flexible vision encoder and standardized evaluation contributions, achieving strong multimodal performance while retaining text-only capabilities. The model is released under Apache 2.0 and performs strongly against comparable and larger models.
Problem
Existing multimodal benchmarks and evaluation protocols do not fully capture practical use and can penalize substantively correct answers for formatting differences.
Method
Pixtral 12B combines a from-scratch vision encoder supporting native image resolutions and aspect ratios with MM-MT-Bench and Explicit, standardized evaluation procedures.
Results
Pixtral substantially outperforms comparable open models on multimodal benchmarks while maintaining strong text-only reasoning performance and matching or exceeding models such as Qwen2-VL 7B and Llama-3.2 11B.
Takeaways & Limitations
Pixtral provides a versatile open-weight multimodal model for practical image-text interactions, with evaluation resources intended to support fairer comparisons.
Takeaways & Limitations
Multimodal results remain sensitive to prompt design and evaluation metrics, including output-format requirements and exact-match scoring.
Abstract
from arXiv · showhide
We introduce Pixtral-12B, a 12--billion-parameter multimodal language model. Pixtral-12B is trained to understand both natural images and documents, achieving leading performance on various multimodal benchmarks, surpassing a number of larger models. Unlike many open-source models, Pixtral is also a cutting-edge text model for its size, and does not compromise on natural language performance to excel in multimodal tasks. Pixtral uses a new vision encoder trained from scratch, which allows it to ingest images at their natural resolution and aspect ratio. This gives users flexibility on the number of tokens used to process an image. Pixtral is also able to process any number of images in its long context window of 128K tokens. Pixtral 12B substanially outperforms other open models of similar sizes (Llama-3.2 11B \& Qwen-2-VL 7B). It also outperforms much larger open models like Llama-3.2 90B while being 7x smaller. We further contribute an open-source benchmark, MM-MT-Bench, for evaluating vision-language models in practical scenarios, and provide detailed analysis and code for standardized evaluation protocols for multimodal LLMs. Pixtral-12B is released under Apache 2.0 license.
1 Introduction
Pixtral 12B is an open-weight multimodal model designed for image-text understanding, flexible image processing, and practical multi-turn evaluation. It reports strong multimodal and text-only performance while contributing benchmarks and evaluation practices for more standardized assessment.
- Model: Pixtral 12B is an Apache 2.0 multimodal model trained for image-text understanding and multi-turn, multi-image conversation.It is instruction tuned and pretrained on interleaved image-text documents.
- Vision encoder: A from-scratch vision encoder supports images at native resolutions and aspect ratios, with resolution selectable for latency or fine-grained reasoning.The encoder uses ROPE-2D and supports flexible image processing.
- Performance: Pixtral delivers strong multimodal reasoning without sacrificing text-only reasoning performance relative to similarly sized models.It matches or exceeds Qwen2-VL 7B and Llama-3.2 11B on MMMU and MathVista while outperforming most open-source models on MATH and HumanEval.
- Evaluation: MM-MT-Bench addresses the limited practical coverage of short-form and multiple-choice multimodal benchmarks, while its scores correlate highly with LMSys Vision ELO rankings.The paper also identifies prompt and exact-match metric issues and proposes Explicit prompts plus flexible parsing for fairer evaluation.
- Performance: Pixtral surpasses comparable open-source models on MM-MT-Bench and ranks highly on the LMSys Vision leaderboard, including against larger models.The cited comparisons include Llama-3.2 11B, Qwen2-VL 7B, and Llama-3.2 90B.
2 Architectural details
Pixtral combines a vision encoder with a multimodal decoder, using a 400M-parameter vision transformer designed for variable-resolution images and a 12B-parameter language decoder. Relative two-dimensional positional encodings, image-boundary tokens, sequence packing, and causal decoding support flexible multimodal processing.
- Architecture: Pixtral 12B uses a multimodal decoder for reasoning and a vision encoder for ingesting images, built on the Mistral Nemo 12B decoder-only language model.The vision encoder is trained from scratch as a 400M-parameter vision transformer.
- Vision Encoder: Pixtral-ViT processes images across a wide range of resolutions and aspect ratios at their native dimensions.This design supports low-resolution processing for latency constraints and high-resolution processing for fine-grained reasoning.
- Vision Encoder: Break tokens distinguish image rows and mark image ends, while sequence packing concatenates images with a block-diagonal mask preventing cross-image attention leakage.These mechanisms help represent image boundaries and process multiple images efficiently within a batch.
- Vision Encoder: RoPE-2D replaces learned absolute patch embeddings with relative rotary encodings that naturally support variable image sizes.The encoding represents height and width positions through separate frequency components and preserves relative positional structure.
- Multimodal Decoder: The vision encoder connects to the decoder through a two-layer GeLU projection, and image tokens then share the decoder’s causal self-attention pipeline with text tokens.This architecture supports sequences containing text and multiple images within the 128K-token context window.
3 MM-MT-Bench: A benchmark for multi-modal instruction following
MM-MT-Bench evaluates practical multimodal instruction following rather than only image-based multiple-choice accuracy. It uses curated multi-turn conversations spanning common document and visual scenarios, with an independent LLM judge rating responses against references.
- Motivation: MM-MT-Bench addresses the limited user relevance of existing multiple-choice multimodal benchmarks by evaluating instruction-tuned multimodal models.The benchmark is modeled after text-only MT-Bench and focuses on multimodal assistant-style utility.
- Design: MM-MT-Bench contains 92 conversations covering charts, tables, PDF pages, diagrams, and miscellaneous images across one- to four-turn interactions.The benchmark includes 69 single-turn conversations, 18 two-turn conversations, four three-turn conversations, and one four-turn conversation.
- Examples: The benchmark targets image-based extraction, summarization, and reasoning using manually curated images, prompts, and answers verified by a second group of labelers.Prompts require reference to the image input, and multi-turn evaluation provides prior reference answers as conversation history.
4 Results
Pixtral 12B is evaluated against open and closed models using common prompts and metrics, showing strong multimodal and text-only performance. The results also examine prompt formatting, flexible parsing, and vision-encoder design choices.
- Multimodal performance: Pixtral substantially outperforms open models of a similar size and several closed-source models on multimodal benchmarks.All models are re-evaluated with the same prompt and evaluation metric.
- Language performance: Pixtral 12B consistently outperforms comparable open-source models on text-only benchmarks, without compromising text understanding for multimodal capability.The paper presents it as a drop-in replacement for both text and vision tasks.
- Prompt selection: Leading models require prompts that explicitly specify the required output format because vague prompts can penalize substantively correct answers.Explicit prompts substantially improve leading-model performance, while Pixtral generally performs better with them and shows only a minor ChartQA regression.
- Evaluation metrics: Flexible parsing dramatically improves some models, whereas Pixtral 12B changes little and generally continues to outperform other models.The loosest parsing level is included only as an overly generous upper-bound illustration.
- Vision encoder ablations: The Pixtral vision encoder substantially outperforms CLIPA on fine-grained chart and document understanding while matching it on natural-image benchmarks.The ablations compare both encoders using Visual Instruction Tuning with Mistral-Nemo 12B-Instruct as the multimodal decoder initialization.
5 Qualitative examples
Qualitative examples illustrate Pixtral’s use across complex figures, multiple images, charts, and image-to-code tasks. In a chart-analysis comparison, Pixtral gives a complete and accurate response while competing models miss information or trends.
- Qualitative examples: Pixtral supports reasoning over complex figures, multi-image instruction following, chart analysis, and image-to-code conversion.These examples are presented as real-world applications of the model.
- Qualitative examples: On a complex job-jitters chart, Pixtral receives a rating of 8 for a complete and accurate response, while Gemini-Flash-8B extracts wrong information and QwenVL does not elaborate on trends.The comparison is drawn from an MM-MT-Bench example requiring accurate chart understanding, reasoning, and analysis.
6 Conclusion
Pixtral 12B combines strong text-only and multimodal performance with a 400M-parameter vision encoder and a 12B-parameter multimodal decoder. Its instruction following, variable image-size support, and long context window support complex multimodal applications, and it is released under Apache 2.0.
- Conclusion: Pixtral 12B excels in both text-only and multimodal tasks with a 400M-parameter vision encoder and a 12B-parameter multimodal decoder.The model is described as outperforming other open models and matching larger models across benchmarks.
- Conclusion: Variable image sizes, a long context window, and strong instruction following make Pixtral 12B versatile for complex multimodal applications.The paper releases the model under the Apache 2.0 license.
7 Contributors
The appendix presents contributor and implementation material alongside a proof of ROPE-2D’s relative position encoding property. The property expresses positional interactions through relative coordinate differences.
- The appendix lists the Mistral AI Science team contributors.
- The evaluation prompts require image analysis, reasoning, and a final answer in a specified format.
- A judge evaluates assistant responses against reference answers using correctness and helpfulness criteria.
- Conversation history is passed to the judge with reference answers supplied as prior assistant turns.
- ROPE-2D is analyzed for arbitrary features and spatial positions, with the exposition illustrated in four dimensions.
- The resulting positional terms depend on coordinate differences p−r and q−s rather than absolute positions.
C Flexible Parsing Settings
The paper evaluates multimodal model outputs under progressively looser parsing constraints. These settings distinguish exact format compliance from answers that remain recognizable under formatting variation.
- The baseline requires responses to follow prompt instructions exactly and end with "Final Answer: <ANSWER>".
- Flexible Parsing Level 1 additionally accepts responses ending with "Answer: <ANSWER>".
- Flexible Parsing Level 2 strips extra markdown formatting such as bold or italic answer labels.
- Flexible Parsing Level 3 marks a response correct when the ground-truth answer appears anywhere in the response, including number variants with or without commas.
- Flexible Parsing Level 3 is intended as an upper bound because it can mark incorrect answers as correct.
D Robustness to prompting
The appendix tests how prompting and parsing choices affect multimodal evaluation. Pixtral remains stable across prompt variants, while Llama-3.2 and other open models can require model-specific prompts or metric adjustments to recover reported performance.
- D.1 Llama-Specific Prompts: A common prompt is designed to reproduce reported GPT-4o and Claude-3.5 Sonnet results while using the same prompt and metric for all models.
- D.1 Llama-Specific Prompts: Llama-3.2 defaults to "**Answer:** <ANSWER>", omitting "Final" despite explicit instructions.
- D.1 Llama-Specific Prompts: Prompting Llama-3.2 with its default format substantially improves performance, while flexible parsing mitigates regex-related degradation.
- D.1 Llama-Specific Prompts: Llama-specific prompts produce over 15% jumps on Mathvista and MMMU for the 11B variant, while Pixtral performance remains stable and leads Llama-3.2 11B.
- D.1 Llama-Specific Prompts: Table 6 compares default and Llama-specific prompts under the Exact Match metric, showing improved performance for the Llama-3.2 11B variant.
- D.2 Average performance across prompts: Across 10 varied prompts evaluated with Flexible Parsing Level 3, Pixtral outperforms comparable open-source models and surpasses Llama-3.2 90B on Mathvista and ChartQA.
- E Reproducing Reported Numbers: Under a common protocol, some models underperform reported figures, requiring prompt and metric tuning to recover them.
E.1 Summary
The standardized evaluation analysis finds that explicit prompting and flexible parsing can recover reported performance, but model requirements differ. Pixtral maintains strong performance without targeted interventions.
- Frontier and smaller closed-source models can recover or exceed reported figures under a common protocol using Explicit prompts.
- Smaller open-source models typically require model-targeted prompt tuning or metric adjustment to recover or exceed reported performance.
- Pixtral 12B reports strong performance without targeted interventions and remains robust across prompts, LMSys Vision Arena, and MM-MT-Bench.
- The standardized protocol roughly matches or exceeds reported figures, with a small gain from flexible parsing except for Claude Haiku on ChartQA.
- ChartQA prompts are simplified and adapted to expected answer formats, while a unified prompt containing all format specifications reduces performance.
E.4 Llama-3.2
Reported performance for prior models can depend substantially on model-specific prompt formatting and evaluation adjustments. For Llama-3.2 90B, removing answer prefixes improves DocVQA by +4.8, while Pixtral 12B reports strong performance without such interventions.
- Evaluation interventions: Prompting models to produce their default markdown answer format substantially improves their recovered performance under Flexible Level 3 evaluation.The intervention changes the Explicit prompt to request formats such as “**Answer:**” or “*Answer: <ANSWER>*”.
- Evaluation interventions: +4.8 on DocVQA follows stripping “The answer is <ANSWER>” prefixes from Llama-3.2 90B generations penalized by ANLS.The adjustment targets answer-format mismatch rather than model generation content.
- Prompt design: A single unified prompt containing all answer-format specifications reduces performance, motivating separate prompts for different expected answer formats.The evaluation procedure first simplifies the prompt into a one-line instruction.
- Comparison: Pixtral 12B reports strong performance without the model-specific prompt and metric interventions used to recover prior models’ reported results.The comparison is made against an evaluation setup that tunes settings toward each individual prior model.