Pixtral 12B

1 Introduction

• Pixtral is an Apache 2.0-licensed, instruction-tuned model pretrained on large-scale interleaved image-and-text documents, supporting multi-turn, multi-image conversation.Source: §1 · p. 1
• The new vision encoder uses ROPE-2D to process images at native resolution and aspect ratio, enabling low-resolution processing for latency constraints and high-resolution processing for fine-grained reasoning.The encoder is trained from scratch; block-diagonal attention masks enable sequence packing, while ROPE-2D encodings facilitate variable image sizes.Source: §1 · p. 1, Figure 2 · p. 3
• On MM-MT-Bench, Pixtral surpasses comparable open models; on the LMSys Vision Leaderboard it ranks above Llama-3.2 11B [6], Qwen2-VL 7B [23], Claude-3 Opus and Sonnet, and Llama-3.2 90B [6].It is the highest-ranked Apache 2.0 model there; Figure 1 covers MM-MT-Bench and the October 2024 Vision arena.Source: Figure 1 · p. 2, §1 · p. 2
• Pixtral matches or exceeds Qwen2-VL 7B [23] and Llama-3.2 11B [6] on MMMU and MathVista [14], and outperforms most open-source models on MATH [7] and HumanEval [26].It also outperforms Llama-3.2 90B [6] and closed models Claude-3 Haiku and Gemini-1.5 Flash 8B [18] on multimodal benchmarks.Source: §1 · pp. 1–2
• Multimodal evaluation protocols are not standardized, and small setup changes can shift results; under-specified prompts can reduce closed-model scores, while exact match penalizes equivalent formats such as “6.0” versus “6.”Source: §1 · p. 2
• The authors propose explicit format-specifying prompts, analyze flexible parsing, and release evaluation code and prompts to support fair, standardized evaluation.Source: §1 · p. 2

2 Architectural details

• Pixtral 12B pairs a multimodal decoder for high-level reasoning with a vision encoder for image input.Source: §2 · p. 3

2.1 Multimodal Decoder

• Pixtral 12B builds on Mistral Nemo 12B, a 12-billion-parameter decoder-only language model with strong knowledge and reasoning performance.Source: §2.1 · p. 3

2.2 Vision Encoder

• The 400-million-parameter Pixtral-ViT is trained from scratch to process images across varied resolutions and aspect ratios.Unlike tile-based approaches, it adapts to images at their native resolution and aspect ratio, supporting multimodal tasks from classification to optical character recognition.Source: §2.2 · pp. 3–4
• Pixtral-ViT uses image-break and image-end tokens, gated feedforward layers, sequence packing with block-diagonal masks, and RoPE-2D positional encodings.RoPE-2D encodes relative height and width positions, enabling variable image sizes without interpolating learned position embeddings.Source: §2.2 · pp. 3–4

2.3 Complete architecture

• A two-layer GeLU network projects vision-encoder outputs to the decoder’s embedding size, and causal self-attention enables multi-image conversations.The decoder treats image tokens like text tokens, including RoPE-1D positional encodings.Source: §2.3 · p. 4

• MM-MT-Bench evaluates multimodal instruction following, addressing the limits of multiple-choice benchmarks and extending the text-only MT-Bench approach.The benchmark aims to measure a model’s utility as a multimodal assistant or chatbot.Source: §3 · p. 5

Design

• MM-MT-Bench has a 0.91 Pearson Correlation Coefficient with LMSys-Vision ELO ratings.The passage reports this correlation as evidence of the benchmark’s alignment with the ratings.Source: §3 · p. 5
• The benchmark contains 92 conversations across charts, tables, PDF pages, diagrams, and miscellaneous images, with conversations spanning one to four turns.It includes 69 single-turn conversations, 18 two-turn conversations, four three-turn conversations, and one four-turn conversation.Source: §3 · p. 5
• An independent LLM judge scores each turn from 1 to 10 for correctness and completeness, using the full conversation history.Models are queried in parallel over all turns, with reference answers for earlier turns supplied as conversation history.Source: §3 · p. 5

Examples

• The benchmark’s manually curated image prompts and answers cover extraction, summarization, and reasoning, and each prompt requires the image to answer correctly.A second group of labelers verified the answers.Source: §3 · p. 5

4.1 Main Results

• Pixtral substantially outperforms open models around its scale and several closed-source models on multimodal benchmarks, leading comparable-size models on MM-MT-Bench and approaching the largest open models on LMSys Vision Arena.The comparison uses shared prompts and metrics; relaxed evaluation results for Qwen2-VL 7B and Llama-3.2 11B are also reported.Source: §4.1 · p. 6, Table 2 · p. 6
• Pixtral 12B consistently outperforms comparable-size open-source models on text-only benchmarks, without compromising text understanding for multimodal capability.The authors describe it as a drop-in replacement for text-only deployments as well as vision tasks.Source: §4.1 · p. 6, Table 3 · p. 6

4.2 Prompt selection

• Explicit prompts that specify answer formats substantially improve leading models over vague prompts that leave the required response format unclear.Pixtral generally performs better with explicit prompts, with only a minor regression on ChartQA; some smaller models perform worse, possibly due to a mismatch with benchmark training prompt styles.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 5 · p. 7

4.3 Sensitivity to evaluation metrics

• More flexible parsing dramatically improves some models’ scores, while Pixtral benefits little and generally continues to outperform other models under flexible metrics.Flexible Level 3 counts a response correct if the reference appears anywhere in the generation, making it an overly generous illustrative upper bound.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Vision Encoder Ablations

• With visual instruction tuning, Pixtral’s encoder substantially outperforms CLIPA on fine-grained chart and document understanding while matching it on natural-image VQAv2.CLIPA was tested at 224 × 224 pixels and with 25 crops totaling 1120 × 1120 pixels; Pixtral’s flexible encoder used variable resolutions up to 1024 pixels.Source: Figure 6 · p. 8, §4.4 · p. 9

5 Qualitative examples

• Examples demonstrate Pixtral’s use for complex-figure reasoning, multi-image instruction following, chart understanding and analysis, and image-to-code conversion.Source: §5 · p. 9
• On an MM-MT-Bench job-jitters chart example, Pixtral 12B’s complete, accurate response earned rating 8, while Gemini-1.5 Flash-8B (0827) gave wrong information and QwenVL-7B did not elaborate on trends.Source: §5 · p. 9

6 Conclusion

• Pixtral 12B combines strong text and multimodal performance with variable image sizes and a long context window, and is released under Apache 2.0.Its architecture uses a 400M-parameter vision encoder and a 12B-parameter multimodal decoder.Source: §6 · p. 9

A Prompts

• The authors open-source evaluation prompts selected to reproduce reported GPT-4o and Claude-3.5 Sonnet performance.Source: A Prompts · p. 19

A.1 MMMU and Mathvista

• The MMMU and Mathvista prompt requests detailed image description and reasoning before a formatted final answer, using a single option letter when choices are provided.Source: §A.1 · p. 19

A.2 ChartQA

• The ChartQA prompt requires a concise final answer, preserving graph labels and formatting numbers, ratios, and percentages according to explicit rules.Source: §A.2 · p. 19

A.3 VQAv2

• The VQAv2 prompt requires a minimal single-word, number, or short-phrase answer, with numbers unit-free and Yes/No answers restricted to those words.Source: §A.3 · p. 19

A.4 DocVQA

• The DocVQA prompt instructs models to answer using a single word or phrase.Source: §A.4 · p. 19

A.5 MM-MT-Bench Judge Prompt

• The MM-MT-Bench judge prompt directs an impartial correctness-and-helpfulness rating, with the conversation history and reference answers supplied as assistant turns through teacher-forcing.Source: §A.5 · pp. 19–20

B Relative Position Encoding Property of ROPE-2D

• ROPE-2D’s position rotations depend on coordinate differences, (p − r)·θ1 and (q − s)·θ2, for positions within image height H and width W.The appendix illustrates the property for d = 4 and states that extension to higher dimensions is straightforward.Source: B Relative Position Encoding Property… · p. 20

C Flexible Parsing Settings

• The parsing levels progressively relax exact-answer-format requirements to investigate how model performance varies under looser constraints.Source: C Flexible Parsing Settings · p. 21
• The baseline requires exact compliance with prompt instructions and a response ending in “Final Answer: <ANSWER>.”Source: C Flexible Parsing Settings · p. 21
• Flexible Parsing Level 1 also accepts responses ending in “Answer: <ANSWER>.”Source: C Flexible Parsing Settings · p. 21
• Flexible Parsing Level 2 catches cases where models add extra markdown formatting and strips it; this formatting is particularly prevalent in Llama-3.2 models [6].Source: C Flexible Parsing Settings · p. 21
• Flexible Parsing Level 3 accepts a response if the ground-truth answer appears anywhere, making it an upper bound that may mark incorrect answers as correct.For single-letter answers, it searches for “is <A>,” “are <A>,” or “<A>”; for numbers, it searches with and without commas.Source: C Flexible Parsing Settings · p. 21

D.1 Llama-Specific Prompts

• In Section 4.1, a common prompt allowed reproduction of reported GPT-4o and Claude-3.5 Sonnet figures, requiring models to end responses with "Final Answer: <ANSWER>".The full prompts are in Appendix A.Source: §D.1 · p. 21
• Llama-3.2 defaults to “**Answer:** <ANSWER>” despite the explicit instruction, and flexible parsing mitigates performance loss from regex mismatches.The Llama-specific prompt instead asks for that default output format.Source: §D.1 · p. 21
• Llama-specific prompts improve Llama-3.2 performance by over 15% on both Mathvista and MMMU, while Pixtral remains stable and leads Llama-3.2 11B.Table 6 compares the default and Llama-specific prompts under Exact Match.Source: Table 6 · p. 21, §D.1 · p. 21

D.2 Average performance across prompts

• For prompt-variation tests, Mistral Large v2 generated 10 rewrites with varied wording and explicit instructions, and all models used Flexible Parsing Level 3.The experiment reports average results across the prompts.Source: §D.2 · p. 22
• Pixtral typically shows lower performance variance between prompts than other models.The standard deviations are shown in gray in Table 7.Source: §D.2 · p. 22, Table 7 · p. 22
• Across 10 prompt variants, Pixtral outperforms comparable-size models and surpasses Llama-3.2 90B on Mathvista and ChartQA.Table 7 reports mean performance and standard deviations; all models use Flexible Level 3 parsing.Source: Table 7 · p. 22, §D.2 · p. 22

E Reproducing Reported Numbers

• The common protocol evaluated all models with the same metric and prompt, allowing frontier models to achieve their reported performance.The protocol is described in Section 4.1.Source: E Reproducing Reported Numbers · p. 22
• Under a common protocol, some models substantially underperformed reported figures, and recovering them required tuning each model’s prompt and metric.The results are shown in Table 8.Source: E Reproducing Reported Numbers · p. 22

E.1 Summary

• Frontier and smaller closed-source models recover or exceed reported figures under the common protocol by precisely following the explicit prompts.The explicit prompts are described in Appendix A.Source: §E.1 · p. 22
• Smaller open-source models generally need model-targeted prompt tuning or metric adjustments to recover or exceed reported performance.Source: §E.1 · p. 22
• Pixtral 12B follows prompt instructions and performs strongly without targeted interventions, with robust results across prompts and strong performance on LMSys Vision Arena and MM-MT-Bench.The prompt robustness results are reported in Appendix D; the benchmark results are discussed in Section 4.1.Source: §E.1 · p. 22

E.2 Closed models: Claude-3 Haiku and Gemini-Flash-8B

• The standardized protocol roughly matches or exceeds reported figures, with small gains from flexible parsing; Claude Haiku on ChartQA requires Level 3 parsing to approach its reported performance.Source: §E.2 · p. 22

E.3 Qwen2-VL 7B

• For Qwen2-VL 7B, evaluation uses a one-line instruction and answer-format-specific prompts, such as requesting a floating-point answer to two decimal places.Analogous prompts are used for integer and multiple-choice answers.Source: §E.3 · pp. 22–23
• A unified prompt containing all answer-format specifications reduces Qwen2-VL 7B performance.Source: §E.3 · p. 23

E.4 Llama-3.2

• Llama-3.2 defaults to markdown-formatted answers; requesting that format in the Explicit prompt lets the models recover reported performance under Flexible Level 3 evaluation.The section notes substantial improvement after changing the Explicit prompt to request the default format.Source: §E.4 · p. 23
• +4.8 DocVQA points result from stripping “The answer is <ANSWER>” prefixes from Llama-3.2 90B generations penalized by ANLS.Source: §E.4 · p. 23

E.5 Llava-OneVision 72B

• Llava-OneVision 72B is evaluated with a simplified one-line instruction and separate prompts for each expected answer format.The section reports that a single unified prompt containing all format specifications reduces performance.Source: §E.5 · p. 23
• For Llava-OneVision 72B, a single prompt combining all answer-format specifications reduces performance.Source: §E.5 · p. 23

E.6 Molmo

• Molmo evaluation uses answer-format-specific prompts and adjusts long-answer scoring for default “\n\n<ANSWER>” endings; VQAv2 post-processing maps number words to digits.For example, “Two” is remapped to “2.”Source: §E.6 · p. 23
• Pixtral 12B reports strong performance without the model-specific evaluation interventions used to recover prior models’ reported performance.Table 8 describes tuning evaluation settings toward individual models and highlights Pixtral 12B alongside strong closed-source models as reporting strong performance without such interventions.Source: Table 8 · p. 24