PaperlayerView exact revision on arXiv

Source-linked AI summary

Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, Hakim Hacid, Hamza Yous, Brahim Farhat, Ibrahim Khadraoui, Mugariya Farooq, Giulia Campesan, Ruxandra Cojocaru, Yasser Djilali, Shi Hu, Iheb Chaabane, Puneesh Khanna, Mohamed El Amine Seddik, Ngoc Dung Huynh, Phuc Le Khac, Leen AlQadi, Billel Mokeddem, Mohamed Chami, Abdalgader Abubaker, Mikhail Lubinets, Kacper Piskorski, Slim Frikha

arXiv:2507.22448v1cs.CL

TL;DR

Transformer sequence-length costs motivate more efficient hybrid designs. Falcon-H1 combines attention and Mamba-based SSMs in parallel; its evaluations report superior performance with less data and up to 8x faster long-context inference.

  • Problem

    Transformer attention has quadratic complexity in sequence length, motivating alternatives that improve computational or memory efficiency while retaining performance.

    Source: §1 · p. 3

  • Method

    Falcon-H1 combines attention and Mamba-2 heads in parallel, with independently adjustable head counts in its hybrid mixer.

    Source: §1 · p. 3

  • Results

    Falcon-H1 reports superior performance using 2.5T to 18T tokens and up to 8x faster inference in long-context scenarios.

    Source: §7 · p. 53

  • Takeaways & Limitations

    Falcon-H1-1.5B-Deep delivers capabilities competitive with leading 7B–10B models and is presented as suited to edge deployments.

    Source: §7 · p. 53

    Scaling contexts beyond 256k and enhancing data remain future-work priorities.

    Source: §7 · p. 53

Abstract

from arXiv · show

In this report, we introduce Falcon-H1, a new series of large language models (LLMs) featuring hybrid architecture designs optimized for both high performance and efficiency across diverse use cases. Unlike earlier Falcon models built solely on Transformer or Mamba architectures, Falcon-H1 adopts a parallel hybrid approach that combines Transformer-based attention with State Space Models (SSMs), known for superior long-context memory and computational efficiency. We systematically revisited model design, data strategy, and training dynamics, challenging conventional practices in the field. Falcon-H1 is released in multiple configurations, including base and instruction-tuned variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters. Quantized instruction-tuned models are also available, totaling over 30 checkpoints on Hugging Face Hub. Falcon-H1 models demonstrate state-of-the-art performance and exceptional parameter and training efficiency. The flagship Falcon-H1-34B matches or outperforms models up to 70B scale, such as Qwen3-32B, Qwen2.5-72B, and Llama3.3-70B, while using fewer parameters and less data. Smaller models show similar trends: the Falcon-H1-1.5B-Deep rivals current leading 7B-10B models, and Falcon-H1-0.5B performs comparably to typical 7B models from 2024. These models excel across reasoning, mathematics, multilingual tasks, instruction following, and scientific knowledge. With support for up to 256K context tokens and 18 languages, Falcon-H1 is suitable for a wide range of applications. All models are released under a permissive open-source license, underscoring our commitment to accessible and impactful AI research.

1. Introduction

Falcon-H1 combines Transformer-style attention and Mamba-based state-space models in a parallel hybrid architecture. The family pairs this design with varied model sizes, multilingual and long-context support, and training strategies designed for efficiency.

1. Introduction

  • Falcon-H1-34B-Instruct achieves competitive or superior results to Qwen3-32B, Qwen2.5-72B-Instruct, and LLaMA3.3-70B-Instruct at approximately half their size.Source: §1 · p. 3
  • The hybrid mixer places attention and Mamba-2 heads in parallel, with independently adjustable head counts to tune their ratio.The paper says this design enables faster inference, lower memory use, and strong generalization across tasks.Source: §1 · p. 3
  • The family includes base, instruction-tuned, and quantized variants at 0.5B, 1.5B, 1.5B-deep, 3B, 7B, and 34B parameters.Source: §1 · p. 3
  • Falcon-H1 supports 18 languages and describes scalability to more than 100 languages through its multilingual tokenizer.Source: §1 · p. 4
  • Falcon-H1-0.5B performs on par with typical 7B models from 2024, while Falcon-H1-1.5B-Deep rivals leading 7B–10B models.The introduction presents these smaller models as intended for low-resource and edge deployments.Source: §1 · p. 4
  • The models support up to 256K context length for applications including long-document processing, multi-turn dialogue, and long-range reasoning.Source: §1 · p. 4
  • Falcon-H1 uses a redesigned training approach to maximize the value of high-quality but limited data and a customized µP recipe for scaling across model sizes.Source: §1 · p. 4

2. Architecture

This section presents Falcon-H1’s architectural and tokenizer design studies, covering its parallel attention–SSM setup and ablations of key configuration choices. It also examines depth–throughput trade-offs and tokenizer data, vocabulary, and splitting choices that inform the final design.

2. Architecture

  • The hybrid block runs attention and SSM on the same input in parallel, concatenates their outputs before projection, and permits flexible channel ratios.Proxy-model ablations and coarse grid searches from 300M to 1.5B parameters informed the final architecture configuration.Source: §2 · p. 4, Figure 1 · p. 5

2.1 Channel Allocation

  • Falcon-H1 uses semi-parallel blocks with roughly 2:1:5 SSM, attention, and MLP channel allocation; more attention channels hurt loss more than SSM–MLP reallocations.The optimal SSM/MLP allocation was relatively flat near its optimum, allowing slight variation by model size.Source: §2.1 · pp. 5–7, Figure 2 · p. 7

2.2 SSM-Specific Parameters Ablations

  • Larger SSM state dimensions improved validation accuracy while throughput declined, whereas larger SSM head dimensions improved loss and efficiency.At sequence length 2048, the final models used (ng, dstate) = (1, 256); Falcon-H1-34B used ng = 2 to suit its parallelism. Head dimensions below 32 reduced GPU utilization, while dhead ≥64 maintained optimal efficiency.Source: §2.2 · pp. 8–9, Figure 3 · p. 9
  • A convolution kernel size of 4 minimized validation loss, and SSD chunk sizes of 128 or 256 formed an efficiency plateau; the models use kernel size 4 and chunk size 256.Chunks below 64 incur launch overhead, while chunks above 256 make the prefix-sum kernel memory-bound.Source: §2.2 · p. 9, Figure 4 · p. 10
  • Resetting the SSM hidden state at document boundaries prevents cross-document leakage without added compute or memory overhead.A constant −80 bias before exponentiation effectively zeros the transition at a boundary while preserving gradient flow and numerical stability.Source: §2.2 · p. 10

2.3.1 RoPE Base Frequency

  • Falcon-H1 uses RoPE base frequency b = 1011; increasing b improved evaluations when sequence length was extended, and large b enables extension without RoPE modifications.A sweep on Falcon-H1 0.5B found the loss curve flattens at larger b, with 1011 roughly corresponding to its optimum.Source: §2.3.1 · p. 11, Figure 5 · p. 12

2.3.2 Width–Depth Trade-offs

  • Depth adds sequential computation and activation-memory demands, while width increases parallel computation and peak-memory footprint.The section also describes a fixed-parameter loss comparison across layer counts, but provides no numerical outcome in the supplied passages.Source: §2.3.2 · pp. 11–12, Figure 5 · p. 12
  • W1536L87 outperformed W2560L30 in quality, but reduced training throughput and slowed inference by 25–30% relative to the shallowest configuration.The deep variant also matched or outperformed 3.0B and 7.0B reference models twice to five times its size; the released 1.5B variants are width-balanced with 24 layers and deeper with 66 layers.Source: §2.3.2 · p. 13

2.4 Tokenizer

  • Tokenizer vocabulary size was scaled with model size to balance memory footprint and performance, rather than applying one vocabulary size across all scales.The tokenizer studies examined compression, expressiveness, and full-scale downstream outcomes because proxy metrics alone do not capture all effects on model performance.Source: §2.4 · p. 13, §2.4.1 · p. 13

2.4.1 Empirical Studies

  • Tokenizer corpus-size effects were non-monotonic and depended on vocabulary size: 1GB or 14GB performed best for 65k, while 14GB led on both metrics for 135k.For 65k, 1GB gave the best compression and 14GB maximized bytes per token; performance degraded at 40GB. The study varied corpus sizes of 1GB, 14GB, and 40GB and vocabularies of 65k and 135k.Source: §2.4.1 · p. 13
  • Differences between modern splitting regex patterns were small, so the study recommends using a well-established, up-to-date tokenizer’s pattern.The comparison used GPT-4o, LLaMA-3, and GPT-2 patterns at a fixed 131k vocabulary size.Source: §2.4.1 · p. 14
  • Splitting both digits and punctuation consistently improved HumanEval code generation and supported more semantically meaningful tokenization across languages, despite fertility and bytes-per-token metrics not reliably predicting downstream performance.Without punctuation splitting, full-width punctuation in Chinese and Japanese can merge with adjacent characters; the study concludes tokenizer choices should be guided by model performance rather than proxy metrics alone.Source: §2.4.1 · pp. 14–16
  • Adding common LATEX commands to the tokenizer vocabulary improved performance across most of four math benchmarks in a controlled comparison of two 1B Falcon-Mamba models.The models trained on the same 280GT math-heavy mixture, including a 40GT decay stage; the augmented vocabulary gave common mathematical syntax native tokens.Source: §2.4.1 · pp. 16–17, Figure 8 · p. 16

2.4.2 Final Tokenizer Design and Implementation

  • The final design uses BPE trained on a multilingual corpus covering over 121 languages, with 32K, 65K, 130K, and 261K vocabularies tailored to model scale.It includes digit and punctuation splitting, manually added common LATEX commands, and 1,024 reserved special tokens for downstream customization.Source: §2.4.2 · p. 17, Table 5 · p. 17

3. Pretraining

This section describes Falcon-H1’s pretraining data, optimized mixtures and schedules, context extension, training methods, and distributed setup. It covers training dynamics and stability, weight decay, hyperparameter transfer, and scaling and parallelism strategies for efficient training and inference.

3.1 Pretraining Data

  • More than 20T tokens were assembled in the corpus, with up to 18T used for training; English web data retained about 11T tokens, and multilingual web data exceeded 3,000 GT.The multilingual web corpus covered 17 pretrained languages; Falcon-H1-0.5B was English-only.Source: §3.1 · p. 17, §3.1.1 · p. 18

3.1.1 Data Sources

  • Rewriting curated raw data was favored over ungrounded synthetic generation to structure knowledge, reduce noise, and improve training stability and efficiency.Synthetic textbooks were also generated from Wikipedia topic hierarchies covering over 30K topics.Source: §3.1.1 · pp. 19–20
  • 32K-, 128K-, and 256K-targeted long-context data used fill-in-the-middle and section-reordering tasks, while long-context extension followed learning-rate decay for greater compute efficiency.Falcon-H1-34B and 7B used 60 GT at both 32K and 128K, and 25 GT at 256K; 3B and 1.5B used the same token counts at 32K and 128K.Source: §3.1.1 · p. 20, §3.1.2 · p. 23

3.1.2 Data Strategy

  • 15% web data in Falcon-H1-34B and 12.35% in Falcon-H1-7B; the smaller models used a constant data mixture throughout pretraining.The 34B and 7B mixtures were iteratively refined, while the smaller models leveraged dynamic data preparation from larger-model training.Source: §3.1.2 · p. 21, Table 6 · p. 21
  • Multi-epoch upsampling kept scarce high-quality corpora influential, and the authors’ memorization-window analysis suggested repeated reuse could preserve generalization.They validated sources by training 0.5B models from scratch and used deterministic sequential loading to support reproducible comparisons and mixture updates.Source: §3.1.2 · pp. 20, 22
  • Introducing all complexity levels from the outset—the anti-curriculum approach—outperformed curricula that reserved high-quality data for later, when sufficient high-quality data was available.The authors hypothesize that early, continuous exposure helps models develop features needed for complex tasks.Source: §3.1.2 · pp. 22–23

3.2.1 Training Stability

  • Loss spikes occurred even in a pure Mamba2 baseline; wider models spiked while deeper variants trained smoothly, and clipping positive dt removed spikes.Attenuating dt enabled relatively high learning rates without observed spikes, while preserving its full parameter range.Source: §3.2.1 · pp. 23–24

3.2.2 Effective Learning Rate and Effective Weight Decay

  • With no weight decay, matrix-layer parameter norms grew indefinitely; with weight decay, they stabilized, and normalized norms were close across layers.The stabilized norm depends on learning rate, weight decay, and batch size; the section leaves the batch-size dependence for future work.Source: §3.2.2 · pp. 24–25
  • ELR primarily determines training noise, while parameter norms follow the measured scaling and ELR/EWD directions are orthogonal on log-scaled sweeps.The authors caution that these conclusions are rough approximations from limited Falcon-H1 experiments.Source: §3.2.2 · pp. 25–27, Figure 11 · p. 27
  • Internal tests found power scheduling achieved better loss than learning-rate-tuned WSD, while EPS improved convergence over PS but assumes parameter norms should not be scaled.The authors say systematic study is needed to determine how weight-norm control should be incorporated into training schedules.Source: §3.2.2 · pp. 27–28

3.2.3 Maximal Update Parametrization (µP) with Tunable Multipliers

  • Falcon-H1 relocates µP scaling from learning rate and weight decay to forward multipliers, allowing models of different sizes to share those optimizer settings and tune multipliers at a base size for transfer.The authors also report tuning 35 multipliers; sensitivity was strongest for ELR multipliers, followed by forward, EWD, and vector-layer LR multipliers.Source: §3.2.3 · p. 29, Figure 12 · p. 31

3.2.4 Other Aspects: Batch Scaling, Rampup, Warmup

  • Batch scaling with square-root learning-rate adjustment better preserves optimal learning than no batch scaling, though robust hyperparameter transfer across batch sizes requires further study.The adjustment is described as suitable for Adam; the authors note that parameter-norm scaling with batch size also needs consideration.Source: §3.2.4 · p. 31
  • Rampup without batch scaling had better early loss than rampup with scaling, but scaling eventually outperformed it at longer training durations; the optimal warmup duration was 0.1GT.No rampup produced the worst loss and amplified instability; the rampup-period scaling benefit persisted after the runs returned to the same hyperparameters.Source: §3.2.4 · pp. 31–33, Figure 13 · p. 32

3.3 Pretraining Infrastructure

  • Mambatron uses redesigned context parallelism and new Mixer Parallelism for Falcon-H1’s hybrid attention-SSM architecture, and the series was trained on 4,096 NVIDIA H100 GPUs.Dynamic node allocation enabled simultaneous training of six models; the passage says Mixer Parallelism accelerates training and inference throughput.Source: §3.3 · p. 33

3.3.1 Scaling Dynamics of Data Parallelism

  • With global batch size fixed, increasing data-parallel workers reduces gradient-accumulation computation while communication latency grows, causing throughput to deviate from ideal linear scaling.The passage attributes diminishing gains to communication overhead becoming dominant as the computation term shrinks.Source: §3.3.1 · pp. 33–34
  • To scale data parallelism without destabilizing convergence, the authors cap DP size to manage communication and increase global batch size only to a stability-preserving throughput threshold.Although increasing global batch size proportionally with NDP could maintain linear scaling with constant K, large deviations from the tuned batch size can harm convergence and final performance.Source: §3.3.1 · p. 34

3.3.2 Mixer Parallelism (MP)

  • Mixer Parallelism partitions tensor-parallel groups between attention and Mamba operations so they run concurrently, then synchronizes their outputs with an all-reduce.The authors report improved training throughput, memory efficiency, and inference efficiency, particularly for small batches and short sequences.Source: §3.3.2 · p. 34
  • Interleaved Mixer Parallelism achieves a 1.43x speedup over the baseline by effectively balancing computational load, demonstrating the reported superiority of the interleaved strategy in the comparison.Source: §3.3.2 · p. 34
  • Mixer Parallelism accelerates inference for small batches and short generated sequences, but its advantage diminishes and reverses for larger batches and longer generations.This pattern holds across model sizes.Source: §3.3.2 · p. 35

3.3.3 Context Parallelism (CP)

  • Context Parallelism shards each sequence into contiguous device-local chunks, using RingAttention for attention, chunk-wise state passing for SSM layers, and boundary activations for causal convolution.RingAttention circulates K/V tensors so each rank uses O(Q) memory rather than O(T); SSM communication passes a single state tensor per boundary, keeping bandwidth linear in GPU count.Source: §3.3.3 · p. 36

4. Post-trainining

Falcon-H1 post-training combines expanded, skill-focused data with staged supervised fine-tuning and offline preference optimization. The authors report that stopping preference optimization near one epoch yielded better results than the tested alternatives.

4.1 Post-training Data

  • The expanded post-training corpus targets reasoning, mathematics, science, instruction following, and function calling, using open and proprietary sources with benchmark decontamination.The authors report that data quality and structure matter more to post-training performance than volume alone.Source: §4.1 · p. 37

4.2 Supervised Fine-Tuning (SFT)

  • SFT data mixtures and training duration were tested to balance domains, and released checkpoints can be further adjusted for specific use cases.The authors describe SFT as more sensitive to data mixture than pretraining.Source: §4.2 · p. 37
  • Supervised fine-tuning used a 16k-context stage followed by a 128k stage, which was omitted for the 0.5B model.Each stage ran for 3 GT; the 16k stage used warmup-stable-decay, followed by a constant minimum learning rate at 128k.Source: §4.2 · pp. 37–38

4.3 Direct Preference Optimization (DPO)

  • Stopping DPO at approximately one epoch yielded better results than completing two epochs or using a one-epoch linear schedule.The stopping point was determined empirically.Source: §4.3 · p. 39

5. Evaluation

Falcon-H1 models are evaluated across base and instruction-tuned benchmarks, with comparisons standardized across models. The reported results emphasize reasoning-focused strengths across scales and increasing throughput advantages at long contexts, alongside task-specific trade-offs.

5.1 Base Models

  • Falcon-H1-0.5B leads the reported Math, Science, and Code benchmarks, while Falcon-H1-1.5B-Deep rivals 7B–10B models and exceeds its shallower counterpart on MATH-lvl5 (+4.38 points) and MMLU-Pro (+5.54 points).The deep variant is highly competitive in Code and Multilingual tasks, though Qwen3-1.7B beats it on HumanEval and GSM8k.Source: §5.1 · pp. 39, 41, Table 14 · p. 40, Table 15 · p. 41
  • Falcon-H1-3B leads on MATH-lvl5 (25.83) and MGSM (64.00) after 2.5T training tokens, while Falcon-H1-7B leads on MMLU (77.38), MATH-lvl5 (34.67), and GPQA (36.58).The 3B result had not plateaued at training’s end; the 7B model also leads on MBPP, MBPP+, and MGSM.Source: §5.1 · pp. 41–42, Table 16 · p. 42, Table 17 · p. 43
  • Falcon-H1-34B leads on BBH (69.36), MATH-lvl5 (40.71), GPQA (42.70), HumanEval and HumanEval+, and MGSM; Qwen2.5-72B and Llama3.1-70B outperform it on MMLU, ARC-C, and HellaSwag.Source: §5.1 · pp. 42–43
  • Across base-model scales, Falcon-H1 reports leadership in complex reasoning, mathematics, science, and code while challenging models trained on substantially more data.The authors also report that performance had not plateaued at the end of pretraining.Source: §5.1 · p. 43

5.2 Instruct Models

  • Falcon-H1-0.5B-Instruct leads on five Math benchmarks and several Science, Code, and instruction-following tasks; the 1.5B-Deep-Instruct leads across most evaluated domains and is competitive with leading 7B models.The 0.5B-Instruct evaluation excludes multilingual tasks because it was trained only on English data.Source: §5.2 · pp. 44, 46, 47, Table 20 · p. 46, Table 21 · p. 47
  • The instruction-tuned 3B, 7B, and 34B models show broad strengths in reasoning, science, code, multilingual tasks, and knowledge-intensive domains, with competitors ahead on some math or preference benchmarks.The authors characterize the series overall as parameter-efficient across evaluated scales.Source: §5.2 · pp. 47, 48, 49, 52
  • At 131k tokens, Falcon-H1-34B-Instruct scores 62.21 on RAG, the top result, but is surpassed on extreme-length recall and long-form question answering.The authors attribute the gap to training-data composition and identify more curated long-context data as an improvement opportunity.Source: Table 25 · p. 51, §5.2 · p. 51

5.3 Model Efficiency

  • At the longest sequence lengths tested, Falcon-H1-34B achieves up to 4x higher input throughput and an 8x speedup in output throughput, while Qwen2.5-32B has marginal throughput advantage at shorter contexts.Source: §5.3 · p. 52

6. Model Integrations

The report summarizes Falcon-H1’s ecosystem integrations and notes that the supported-tool list was still expanding at release.

6. Model Integrations

  • The integrations list was described as continually expanding, with current information available on the project website.Source: §6 · p. 53
  • Some integrations had been validated internally, while official support awaited merging into main libraries at release.Source: §6 · p. 53
  • Table 26 summarizes the key platforms and tools supported at the time of the report’s release.The report points readers to the project website for the latest information.Source: Table 26 · p. 53, §6 · p. 53

7. Conclusion

The conclusion presents Falcon-H1 as a hybrid Mamba-Transformer model family aiming for strong performance with resource efficiency. It also identifies data, context scaling, and reasoning depth as areas for future work.

7. Conclusion

  • Falcon-H1 combines a hybrid Mamba-Transformer architecture with a customized training strategy and multilingual design, and reports up to 8x faster inference in long-context scenarios.The models use 2.5T to 18T training tokens, and the 1.5B-Deep model is described as competitive with leading 7B–10B models.Source: §7 · p. 53
  • Future work will refine base and instruction-tuned models with broader, more diverse high-quality datasets and develop knowledge compression and contexts beyond 256k.Source: §7 · p. 53
  • The authors plan further strategic scaling and techniques to deepen core reasoning capabilities.Source: §7 · p. 54

8. Authors

The core contributors listed are Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, Younes Belkada, Dhia Eddine Rhayem, Guillaume Kunsch, and Hakim Hacid.

8. Authors

A. Languages used for training Falcon-H1 tokenizers

The table pairs language codes with their corresponding languages.

A. Languages used for training Falcon-H1 tokenizers

  • Table 27 lists language codes and their corresponding languages.Source: Table 27 · p. 65

B. Scalar stochastic dynamics with weight decay

The analysis models AdamW updates for a single parameter to separate stationary noise and signal effects, then describes the authors’ stagewise procedure for tuning µP multipliers and transferring settings.

B. Scalar stochastic dynamics with weight decay

C. Tuning µP multipliers

  • The authors tune all 35 µP multipliers in stagewise micro-sweeps and manually inspect training curves for spikes, noise, and anomalies.Source: C. Tuning µP multipliers · p. 67
  • Near-optimal multiplier sensitivity is estimated by approximating loss as quadratic in the logarithm of each multiplier.Source: C. Tuning µP multipliers · p. 67
  • The coordinate choices ELR and EWD are intended to reduce coupling between learning-rate and weight-decay multipliers for matrix-like layers.Source: C. Tuning µP multipliers · p. 68
  • The authors did not scale µP multipliers with model depth, citing simplicity and time constraints.Source: C. Tuning µP multipliers · p. 68
  • For different training durations, the effective power scheduler can be activated later to adapt to late-stage curriculum learning or long-context extension.Source: C. Tuning µP multipliers · p. 68

D.1 Multilingual Evaluations - Base Models

The appendix presents base-model performance comparisons on multilingual tasks at four parameter scales, with scores reported as percentages.

D.1 Multilingual Evaluations - Base Models

  • Table 28 compares base models on multilingual tasks at the 1B–2B scale using percentage scores.Source: Table 28 · p. 69
  • Table 29 compares base models on multilingual tasks at the 3B–4B scale using percentage scores.Source: Table 29 · p. 70
  • Table 30 compares base models on multilingual tasks at the 7B–12B scale using percentage scores.Source: Table 30 · p. 71
  • Table 31 compares base models on multilingual tasks at the 34B scale using percentage scores.Source: Table 31 · p. 72

D.2 Multilingual Evaluations - Instruct Models

The section compares Instruct-model performance on multilingual tasks across four parameter scales, using percentage scores and marking the best and second-best average scores.

D.2 Multilingual Evaluations - Instruct Models

  • The 1B–2B-scale table compares Instruct models on multilingual tasks using percentage scores and marks the best and second-best average scores.Source: Table 32 · p. 73
  • The 3B–4B-scale table compares Instruct models on multilingual tasks using percentage scores and marks the best and second-best average scores.Source: Table 33 · p. 74
  • The 7B–12B-scale table compares Instruct models on multilingual tasks using percentage scores and marks the best and second-best average scores.Source: Table 34 · p. 75
  • The 34B-scale table compares Instruct models on multilingual tasks using percentage scores and marks the best and second-best average scores.Source: Table 35 · p. 76

D.3 Long-context Evaluations - Instruct Models

The section reports Instruct-model performance on HELMET tasks at five sequence lengths, from 8192 to 131072 tokens.

D.3 Long-context Evaluations - Instruct Models

  • The HELMET task performance table reports results at sequence length 8192.Source: Table 36 · p. 77
  • The HELMET task performance table reports results at sequence length 16384.Source: Table 37 · p. 78
  • The HELMET task performance table reports results at sequence length 32768.Source: Table 38 · p. 78
  • The HELMET task performance table reports results at sequence length 65536.Source: Table 39 · p. 79
  • The HELMET task performance table reports results at sequence length 131072.Source: Table 40 · p. 79

E.1 Synthetic English Data Topics

The section identifies the scope of the synthetic English data topics and describes the programming-language coverage and filtering used for the HQ code corpus.

E.1 Synthetic English Data Topics

  • Table 41 lists topics crawled from starting nodes in the Wikipedia category tree.Source: Table 41 · p. 80

E.2 Programming Languages

  • The included programming-language inventory comprises 67 languages, with its names listed across the appendix passages.Source: §E.2 · pp. 80–81

E.3 Code Quality Classifier

  • The code quality classifier supports 19 programming languages, and only these languages are included in the HQ code corpus.Source: §E.3 · p. 81

Related papers

Loading 2507.22448v1…