LFM2 Technical Report

Model family and deployment

• The released portfolio includes dense 350M–2.6B checkpoints with 32K context, an 8.3B MoE with 1.5B active parameters, multimodal and retrieval variants, and task-specific LFM2-Nanos.Open weights, quantized variants, and deployment guides target Transformers, llama.cpp, ExecuTorch, and vLLM.Source: §1 · p. 2

Backbone

• Hardware-in-the-loop architecture search selects a compact hybrid backbone combining short-range, input-aware gated convolutions with grouped-query attention, tuned for quality under device speed and memory budgets.The resulting dense models span 350M–2.6B parameters with 32K context, alongside an MoE variant, LFM2-8B-A1B.Source: §1 · p. 2

Training

• LFM2 uses 10–12T-token pre-training, long-context mid-training, tempered decoupled Top-K distillation to avoid support mismatch, and a three-stage post-training pipeline.The training approach aims to improve small-model quality without prohibitive cost.Source: §1 · p. 2

Native multimodality

• LFM2-VL pairs the language backbone with a SigLIP2 vision encoder and lightweight connector for flexible on-device accuracy-latency tradeoffs, while LFM2-Audio processes continuous audio features and generates text or discrete audio tokens.The audio-language model supports low-latency speech assistance and transcription or translation in one model.Source: §1 · p. 2

Information retrieval

• LFM2-ColBERT adds a dense module to create a low-latency query and document encoder, using late interaction with max-similarity for multilingual retrieval.The model is designed for high-performance retrieval across multiple languages.Source: §1 · p. 2

Performance

• On CPUs, LFM2 dense checkpoints deliver up to ∼2× prefill and decode speedups versus similarly sized baselines while maintaining or improving benchmark accuracy; LFM2-8B-A1B attains 3–4B-class quality at ∼1.5B active parameters.The report characterizes these as strong quality, latency, and memory tradeoffs for edge deployment.Source: §1 · p. 3

2.1 Architecture Optimization Process

• Architecture candidates are assessed on downstream quality, batch-1 latency, and peak memory, with budget-violating designs discarded and feasible candidates ranked on the quality–latency–memory Pareto frontier.Quality uses an internal suite of 50+ evaluations; latency includes TTFT, decode latency, and prefill throughput, and memory is measured at 4K and 32K contexts.Source: §2.1 · p. 4
• Hardware-in-the-loop search selects gated short convolutions for most layers and a minority of GQA blocks, matching or exceeding baseline quality while improving decode, prefill, and long-context memory.Under identical on-device budgets, adding linear-attention, state-space, or extra convolution operators does not improve aggregate quality and typically worsens device metrics.Source: §2.1 · p. 5

2.2 LFM2 Dense Models

• Dense LFM2 models combine gated short-convolution blocks, a minority of GQA blocks, and SwiGLU MLPs, using pre-norm RMSNorm and RoPE with QK-Norm in attention.The short-convolution operator applies input-dependent multiplicative gating around a depthwise 1D convolution, providing fast local mixing with CPU cache behavior; GQA shares keys and values across head groups.Source: §2.2 · pp. 5–6
• LFM2 releases dense 350M, 700M, 1.2B, and 2.6B checkpoints with 32,768-token context windows and a 65,536-token byte-level BPE vocabulary.The tokenizer is designed for encoding efficiency across English, Japanese, Arabic, Korean, Spanish, French, and German, and includes special tokens for fill-in-the-middle, tool calling, and ChatML.Source: §2.2 · p. 6

2.3 LFM2 MoE

• LFM2-8B-A1B has 8.3B total and 1.5B active parameters, retaining the hybrid backbone while replacing dense MLPs with sparse MoE MLPs in most layers.The first two layers remain dense; subsequent MoE layers have 32 SwiGLU experts and select Top-k=4 experts per token.Source: §2.3 · pp. 6–7

2.4 Inference Performance

• Across smartphone and laptop CPUs, LFM2 generally achieves substantial throughput improvements over similarly sized open baselines.On the Galaxy S25, LFM2-350M delivers 2–3× higher prefill throughput than Granite-4.0-350M; relative trends on the Ryzen AI 9 HX 370 largely mirror the smartphone results.Source: Table 2 · p. 7, §2.4 · pp. 7–8, Table 3 · p. 8

3.1 Training Data

• Dense models use a mixture of roughly 75% English, 20% multilingual, and 5% code; LFM2-8B-A1B uses 60% English, 25% multilingual, and 15% code.Multilingual priorities include Japanese, Arabic, Korean, Spanish, French, and German; 50% of code examples use fill-in-the-middle (Bavarian et al., 2022).Source: §3.1 · p. 9

3.2 Training Stages

• Dense checkpoints train for 10T tokens at 4,096-token context, then 1T higher-quality tokens at 32,768-token context; LFM2-8B-A1B uses 12T initial tokens before the same mid-training stage.The MoE mid-training uses sources with naturally long context and an accelerated learning-rate decay schedule.Source: §3.2 · p. 9

3.3 Decoupled Top-K Knowledge Distillation

• The decoupled, tempered Top-K objective uses LFM1-7B teacher logits with Top-K=32, separating Top-K probability-mass matching from conditional probability matching.Temperature is applied only to the conditional Top-K distributions; the binary membership term and teacher Top-K mass remain untempered to avoid support mismatch. The overall distillation term is balanced with next-token cross-entropy; at τ = 1 it is a lower bound to full forward KL, while τ > 1 optimizes a tempered surrogate.Source: §3.3 · pp. 9–10

4.1 Overview

• The post-training pipeline uses SFT, preference alignment, and model merging to teach chat-template behavior and improve capabilities such as RAG and function calling.SFT uses a large general-purpose mixture; preference alignment directly optimizes against offline data including on-policy SFT-checkpoint samples; merging evaluates and combines candidates to improve robustness and performance.Source: §4.1 · p. 10

4.2.1 SFT Data Mixture

• SFT uses approximately 5.39 million samples for the 350M–2.6B models and 9.24 million for 8B-A1B, integrating multilingual examples across task categories.Its mixture includes 26.2% mathematical reasoning, 16.3% general-purpose, 11.3% instruction following, and 10.2% code. Off-policy preference data contributes only chosen responses to make supervised data closer to the policy targeted by preference optimization.Source: §4.2.1 · pp. 10–11

4.2.2 Training Protocol

• SFT retains the 32,768-token context length and trains for three epochs, using 10% embedding input dropout during training but no internal dropout layers.The protocol uses cosine learning-rate decay, AdamW, bfloat16 activations and gradients, and ZeRO-2 sharding across eight H100-80GB GPUs per run.Source: §4.2.2 · p. 12

4.3.1 Preference Dataset Creation

• Preference data construction starts from approximately 1 million conversations and yields approximately 700,000 preference conversations after response selection and filtering.For each conversation, five SFT-checkpoint responses are sampled; an LLM jury selects chosen and rejected responses, favoring on-policy responses in ties, and CLAIR refines chosen responses for instruction-following data.Source: §4.3.1 · p. 13

4.3.2 Length-Normalized Direct Alignment

• The preference-alignment objective combines length-normalized relative and absolute loss components, with length-normalized DPO and APO-zero as special cases.A joint example uses margin m = 0.1 and absolute-component weight λ = 0.2; the absolute component can regularize generation probabilities.Source: §4.3.2 · p. 13

4.4 Stage 3: Model Merging

• Model merging evaluates parameter-space techniques in parallel and selects the best checkpoints without additional training.Tested techniques include model soup, task arithmetic, TIES-Merging, DARE, and DELLA.Source: §4.4 · p. 14

4.5 Evaluation

• LFM2 models are competitive across knowledge, instruction-following, and mathematical-reasoning benchmarks, and dominate the throughput-versus-average-score Pareto frontier.The frontier uses prefill throughput on 4K-token prompts and decode throughput with a 4K-token prefix, profiled on Samsung S25.Source: §4.5 · pp. 14, 16, Table 6 · p. 15, Table 7 · p. 15
• LFM2-1.2B scores 74.89% on IFEval, while LFM2-2.6B reaches 79.56%; LFM2-2.6B also scores 82.41% on GSM8K.LFM2-1.2B's IFEval score surpasses Qwen3-1.7B's 73.98%, and LFM2-2.6B outperforms Llama-3.2-3B (71.43%) and SmolLM3-3B (72.44%) on IFEval. LFM2-1.2B scores 58.30% on GSM8K and 55.04% on MGSM; LFM2-2.6B scores 74.32% on MGSM. LFM2-8B-A1B improves over LFM2-2.6B by 11.46 points on MMLU-Pro (37.42%), +10.6 points on MATH 500, and +8 points on MATH Level 5; it scores 25.85% on IFBench. LFM2-1.2B retains 94.4% of its English GSM8K performance on MGSM (58.30% vs. 55.04%) and 84.5% of its MMLU performance on MMMLU (55.23% vs. 46.73%).Source: §4.5 · p. 14, Table 6 · p. 15, Table 7 · p. 15

5.1 Architecture

• NaFlex SigLIP2 supports variable resolutions and aspect ratios, while a PixelUnshuffle-plus-MLP connector projects image features into LFM2’s language space and cuts visual tokens fourfold.Inputs up to 512 × 512 pixels use a single-frame regime; larger images use dynamic tiling, with 2–10 tiles and an optional thumbnail. The fourfold reduction saves memory and compute with slight downstream performance degradation.Source: §5.1 · p. 16, §5.1.1 · pp. 16–17, Figure 5 · p. 17

5.2 Training Protocol

• Training moves from connector alignment with the vision encoder and language backbone frozen to joint multimodal mid-training of all three components.Mid-training uses a 32,768-token context and disables high-resolution tiling for sample efficiency; an annealing schedule reduces text-only data from 100% to 30% over the first 5% of steps. Connector pre-training is retained despite ablations showing the schedule removes its necessity.Source: §5.2.1 · p. 18, §5.2.2 · p. 18
• The final stage applies multimodal instruction SFT with high-resolution tiling enabled, then linearly merges top-performing candidate checkpoints into the final model.SFT variants use about 50B tokens with text–image ratios from 5% to 30%, at 32,768-token context length.Source: §5.2.3 · p. 18, §5.2.4 · p. 18

5.3 VLM Training Data Mixtures

• Training data spans captioning, OCR, document and general VQA, and diverse multimodal SFT tasks, with translated vision-language and SFT data supporting nine target languages.Lower-quality SFT subsets are moved to mid-training; the report notes that further multilingual gains require more culturally contextual and locally sourced data.Source: §5.3 · p. 19

5.4 Evaluation

• 51.83 on MM-IFEval lets LFM2-VL-3B match or exceed open-source models below 4B, while its OCRBench score is 822.LFM2-VL-1.6B scores 46.35 on MM-IFEval versus InternVL3.5-1B’s 36.17; LFM2-VL-450M scores 657 on OCRBench versus SmolVLM2-500M’s 562.Source: Table 8 · p. 20, Table 9 · p. 20, §5.4 · p. 21
• 76.55 on SEEDBench and 71.37 on RealWorldQA make LFM2-VL-3B stronger than all baselines, including the larger Qwen2.5-VL-3B, on those benchmarks.Across general vision benchmarks, LFM2-VL-450M also beats SmolVLM2-500M on MMStar (+3.14) and MMBench (+4.47), while LFM2-VL-1.6B beats InternVL3.5-1B on RealWorldQA (+8.63).Source: §5.4 · p. 19, Table 9 · p. 20, Table 8 · p. 20
• 75.84 (+1.51) on translated MMBench and 81.52 (+3.92) on translated MMMB put LFM2-VL-3B ahead of Qwen2.5-VL-3B.It also scores 62.70 on MMLU and 53.37 on MMMLU, outperforming SmolVLM2-2.2B by +35.81 and +21.21, respectively, while closely matching Qwen2.5-VL-3B.Source: Table 9 · p. 20, §5.4 · p. 21
• Reducing the vision-token budget minimally degrades general vision benchmarks, but causes larger drops on tasks requiring high-resolution perception.Figure 6 shows that users can tune compute cost to visual difficulty and resource constraints; fewer tokens limit fine-grained detail on high-resolution tasks.Source: Figure 6 · p. 21, §5.4 · p. 21

6.1.1 Encoder

• The 1.5B model extends the LFM2-1.2B backbone with a continuous log-mel input encoder and a separate discrete audio-code output path.The encoder downsamples 16 kHz audio 8×, processes it through 17 FastConformer layers, and projects embeddings to the backbone dimension; each embedding represents 0.08 seconds of audio.Source: §6.1 · p. 22, §6 · p. 22, §6.1.1 · p. 22

6.1.2 Audio Decoder / 6.1.3 Audio Detokenizer

• An RQ-Transformer generates eight Mimi RVQ codes per frame, while a compact detokenizer converts the frames to waveform audio for real-time edge synthesis.The RQ-Transformer has 115M parameters versus 1.2B for the backbone; the 35M-parameter detokenizer predicts complex STFT coefficients that are converted to 24 kHz audio.Source: §6.1.2 · p. 23, §6.1.3 · p. 23

6.2 Inference

• Two stateful inference modes share the backbone: interleaved mode alternates text and audio for low-latency conversational responses, while sequential mode switches modalities as needed.Interleaved generation emits six text tokens followed by twelve audio tokens; sequential generation uses control tokens to switch between text and audio, supporting ASR, TTS, and mixed outputs.Source: §6.2 · p. 23, §6.2.1 · p. 24, §6.2.2 · p. 24

6.3 Training details

• Training uses separate text and audio cross-entropy losses, weights the first Mimi codebook most heavily, and combines modality losses according to their token counts.Training proceeds through connector alignment with the backbone and encoder frozen, then full-model mid-training on about 100 billion mixed text and audio tokens.Source: §6.3 · pp. 24–25

6.4 Audio Training Data Mixtures

• Audio training mixes transcription, language classification, TTS, and audio-chat data, totaling about 472,000 hours per epoch across input and output modalities.Synthetic conversations include spoken-dialogue cues and varied user voices, and datasets are augmented and filtered for quality; the resulting mixture targets robustness across speech patterns, dialogue structures, and voices.Source: §6.4 · pp. 25–26

6.5 Evaluation

• LFM2-Audio-1.5B performs competitively against Qwen2.5-Omni-3B on ASR and VoiceBench, and its ASR word error rates approach those of ASR-only models such as Whisper.The report also notes that the similarly sized Ultravox-v0.5-Llama-3.2-1b performs similarly on VoiceBench but does not support audio generation.Source: Table 12 · p. 26, §6.5 · p. 26, Table 11 · p. 26

7.1 Architecture and Design

• The 353M-parameter model uses a hybrid backbone and 128-dimensional token projections, with MaxSim matching each query token to its most similar document token.Inputs are constrained to 512 document tokens and 32 query tokens for retrieval.Source: §7.1 · p. 27

7.2 Training Methodology

• Training distills cross-encoder relevance scores into the late-interaction model using a mean-squared-error objective and min-max-normalized teacher scores.The model starts from the pre-trained LFM2-350M checkpoint, and the distillation phase uses English data.Source: §7.2 · p. 28, §7.3 · pp. 29–30

7.3 Evaluation

• LFM2-ColBERT-350M achieves 0.661 mean NDCG@10 across monolingual NanoBEIR tasks and languages, with Arabic at 0.490 versus 0.309 for GTE-ModernColBERT-v1.The baseline scores 0.680 in English, while LFM2-ColBERT-350M scores 0.661; the English-to-Arabic performance gap is 26% for LFM2-ColBERT-350M versus 55% for the baseline.Source: §7.3 · p. 28, Figure 8 · p. 29
• English-only distillation still yields zero-shot multilingual transfer, including English queries against French documents at 0.551 NDCG@10, or 97.7% of monolingual French performance.European-language transfer is stronger (0.50+ NDCG@10) than Arabic-Japanese transfer (0.375); the report relates this pattern to linguistic distance and pre-training language volumes.Source: §7.3 · pp. 29–30, Table 13 · p. 29
• On English NanoBEIR tasks, LFM2-ColBERT-350M scores FEVER 0.949, HotpotQA 0.895, Natural Questions 0.746, SciFact 0.804, and FiQA 0.591 NDCG@10.The benchmark covers 13 retrieval tasks, including fact verification, question answering, and entity retrieval.Source: §7.3 · pp. 28–29, Table 15 · p. 30

7.4 Inference Performance

• At batch size 32, LFM2-ColBERT-350M reaches 1,350 queries/s versus 1,300 for GTE-ModernColBERT-v1, while both reach approximately 1,100 documents/s.At batch size 64, query throughput is 1,420 versus 1,370 queries/s, despite LFM2-ColBERT-350M having 2.37 times more parameters.Source: Figure 9 · p. 31, §7.4 · p. 31

8.1 Modern Open Weight Foundation Model Families

• LFM2 targets the efficiency–quality gap in on-device models by co-designing architecture and training for device constraints while retaining generalist capabilities.Unlike recent general-purpose model families that primarily optimize broad capability and datacenter serving, LFM2 adopts an edge-first objective.Source: §8.1 · p. 31

8.2 Alternative Architectures for Sequence Modeling

• LFM2 uses a minimal hybrid of gated short convolutions and a minority of GQA blocks, since pure sub-quadratic models can lose long-range skills and memory capacity.Under identical on-device efficiency budgets, adding SSMs, linear attention, or further convolutions did not improve quality; hybrid attention addresses limitations reported for pure sub-quadratic architectures.Source: §8.2 · pp. 32–33

8.3 Efficient Training for Small Models

• LFM2’s deterministic decoupled Top-K distillation makes temperature apply only to the conditional Top-K term, avoiding support mismatch when distilling larger models.It makes explicit a decomposition related to concurrent Sparse Logit Sampling while targeting unavailable tail logits and storage constraints.Source: §8.3 · p. 33

9.1 Key Contributions

• Up to 2× faster prefill and decode on CPUs than similarly sized attention-heavy baselines, while matching or improving benchmark accuracy.The family spans dense models from 350M to 2.6B parameters and an 8.3B MoE model with 1.5B active parameters; all support 32K context.Source: §9.1 · p. 33, §9 · p. 33
• Hardware-in-the-loop search yields a compact hybrid backbone using gated short convolutions in most layers and a few GQA blocks for global context.The edge-first design is paired with tempered, decoupled Top-K distillation to reduce storage and bandwidth while avoiding support mismatch, and a three-stage post-training recipe to improve instruction following, RAG, function calling, and multilingual robustness.Source: §9.1 · p. 33

9.2 Limitations and Future Work

• The deployment search targets batch-size-1 CPU and mobile SoC inference, not optimal large-batch server settings or any particular accelerator family.LFM2 variants are also capacity-limited against frontier-scale systems and perform best on short-to-medium-context, task-oriented edge workloads.Source: §9.2 · p. 34
• LFM2-VL handles images up to 512×512 pixels directly and uses tiling plus token reduction at higher resolutions, but lacks explicit grounding and native video support.Future work includes lightweight grounding, higher resolutions, video and sensor extensions, and reinforcement-learning-style post-training for visual reasoning and long-horizon instruction following.Source: §9.2 · p. 34
• LFM2-Audio focuses on speech, but non-speech audio and overlapping multi-speaker speech are not systematically evaluated.Training is dominated by English and other high-resource languages; future work includes richer multilingual and non-speech data, realistic streaming studies, and on-device personalization.Source: §9.2 · p. 34
• LFM2-ColBERT shows weaker cross-lingual transfer for Arabic and East Asian languages, and its 512-token document limit restricts long-form retrieval.Future work includes targeted training for distant language pairs, domain-specific adaptation via projection-layer training with the backbone frozen, and longer document encoding while managing MaxSim computation and index storage.Source: §9.2 · p. 35

9.3 Closing Remarks

• LFM2 is designed to run on commodity CPUs and mobile SoCs, enabling on-device assistants, RAG workflows, and speech interfaces without continuous connectivity.The paper argues that local processing can reduce raw-data transmission and reliance on remote inference, while open weights and deployment recipes make LFM2 a practical baseline for edge-first models.Source: §9.3 · p. 35

9.4 Availability

• All models, including Liquid-Nanos, are released with open weights and deployment packages for ExecuTorch, llama.cpp, and vLLM.The release supports Snapdragon SoCs, AMD Ryzen processors, and standard CPUs, with deployment guides and quantization profiles.Source: §9.4 · p. 35

A.1 Forward KL

• The teacher’s Top-K set T(xc) contains its K highest-probability tokens for context xc, and its complement is A \ T(xc).Source: §A.1 · p. 44
• The forward-KL chain rule separates Top-K membership and conditional-distribution terms over the Top-K set and its complement.The decomposition includes a Bernoulli membership term and conditional KL terms for both sets.Source: §A.1 · p. 44
• Because teacher logits outside T(xc) are unavailable, the objective drops the non-negative complement KL term to form a lower bound.Source: §A.1 · p. 44
• Temperature is applied only to the teacher and student conditional distributions within T(xc), leaving the Bernoulli membership term and teacher Top-K mass weight untempered.Source: §A.1 · p. 44
• The lower-bound term matches the amount of probability mass on the Top-K set, while the conditional term matches its distributional shape.Source: §A.1 · p. 44
• Naively truncating the teacher distribution before full-vocabulary temperature scaling makes the tempered KL grow as τ², whereas conditioning both models on T and tempering there removes the support mismatch.With full logits, both tempered distributions instead converge to uniform over A and the KL tends to zero; conditional distributions converge to uniform over T.Source: §A.1 · p. 45

A.2 Reverse KL variant

• The Top-K reverse-KL objective drops the unavailable, non-negative term for tokens outside the Top-K set, yielding a computable lower bound.The derivation starts from the reverse-KL chain rule.Source: §A.2 · p. 45
• Temperature is applied only to the conditional Top-K term; the outer binary term remains untempered.This temperature placement is specified for the lower-bound variant and the teacher-weighted surrogate.Source: §A.2 · p. 45
• The teacher-weighted surrogate is not a strict lower bound, but empirically removes the incentive for the student to shrink its probability on the Top-K set.It is introduced to avoid incentivizing the student to down-weight P_S(T | x_c).Source: §A.2 · p. 45

• All methods merge safetensors model state dictionaries tensor-by-tensor while preserving each parameter’s original dtype.Source: B Model Merging Techniques · p. 45

Model Soup

• Model Soup merges models by weighted averaging of corresponding parameters using normalized weights (Wortsman et al., 2022).Source: B Model Merging Techniques · p. 45

Task Arithmetic

• Task arithmetic merges fine-tuned models by adding weighted task vectors—their parameter deltas from a base model—so each capability’s contribution can be controlled independently (Ilharco et al., 2022).Source: B Model Merging Techniques · p. 46

TIES-Merging

• TIES-Merging reduces task-vector interference through magnitude-based top-k% sparsification, majority-sign election, and averaging only parameters agreeing with the elected signs (Yadav et al., 2023).The disjoint-merge denominator normalizes by the sum of weights agreeing with the majority sign.Source: B Model Merging Techniques · p. 46

DARE

• DARE replaces magnitude-based pruning with random task-vector dropout and rescales retained parameters by 1/(1 − p); it is combined with task arithmetic or TIES consensus (Yu et al., 2024).Source: B Model Merging Techniques · p. 46

DELLA

• DELLA assigns higher dropout probabilities to lower-magnitude parameters, with p_min = p − ϵ/2 setting the average dropout rate to p; retained parameters are rescaled before TIES-style merging (Deep et al., 2024).Its magnitude-based sampling is combined with sign election and disjoint merging of parameters that agree with the majority sign.Source: B Model Merging Techniques · p. 46

C Evaluation Details

• The custom text-only evaluation harness extracts final answers more reliably than restrictive open-source parsers that can miss correct answers due to formatting variation.The harness is designed specifically to handle differences in answer formatting.Source: C Evaluation Details · p. 47

MMLU

• MMLU uses five-shot chat-format examples, selects the highest-logit token with a constrained-choice fallback, and scores probabilities over A/B/C/D.The fallback is intended to improve robustness to tokenizer differences.Source: MMLU · p. 47

MMLU-Pro

• MMLU-Pro uses five-shot validation-set CoT examples in chat format, removes N/A options, and applies a priority-based custom answer parser before string-match scoring.Generation is greedy with a maximum of 8192 tokens.Source: MMLU-Pro · p. 47

GPQA Diamond

• GPQA Diamond evaluates each question under 10 answer orderings, uses the MMLU parsing approach, and averages accuracy across the permutations.Source: GPQA Diamond · p. 48

IFEval / IFBench / Multi-IF

• IFEval averages four strict/loose prompt- and instruction-level metrics; IFBench uses strict/loose accuracy, while Multi-IF averages accuracy across three turns.These benchmarks use their official reference implementations; Multi-IF also truncates prompts exceeding the model context length.Source: IFEval · p. 48, IFBench · p. 48, Multi-IF · p. 48

GSM8K / GSMPlus

• GSM8K uses randomly sampled five-shot training examples and loose accuracy, whereas GSMPlus follows the GSM8K implementation but uses strict matching only.GSM8K also reports strict accuracy; loose scoring is used because strict scoring performs poorly with Gemma 3 models.Source: GSM8K · p. 49, GSMPlus · p. 49

MMMLU / MGSM

• MMMLU evaluates seven language subsets zero-shot without a chat template and averages across languages; MGSM uses five examples per language and averages loose accuracy across six languages.MGSM also reports strict and loose accuracy per language.Source: MMMLU · p. 50, MGSM · p. 50

D Multilingual Vision Evaluations

• The multilingual benchmarks were translated from English into Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, and Spanish using GPT-4.1-mini.Source: D Multilingual Vision Evaluations · p. 51
• Detailed per-language evaluation results are presented in Table 16.Source: D Multilingual Vision Evaluations · p. 51, Table 16 · p. 51
• Table 16 covers 2–4B VLM performance on multilingual MMBench and MMMB.The table caption states that results were obtained using VLMEvalKit (Duan et al., 2024).Source: Table 16 · p. 51