Source-linked AI summary
Granite Code Models: A Family of Open Foundation Models for Code Intelligence
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, Yan Koyfman, Boris Lublinsky, Maximilien de Bayser, Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Yi Zhou, Chris Johnson, Aanchal Goyal, Hima Patel, Yousaf Shah, Petros Zerfos, Heiko Ludwig, Asim Munawar, Maxwell Crouse, Pavan Kapanipathi, Shweta Salaria, Bob Calio, Sophia Wen, Seetharami Seelam, Brian Belgodere, Carlos Fonseca, Amith Singhee, Nirmit Desai, David D. Cox, Ruchir Puri, Rameswar Panda
TL;DR
Software development needs code LLMs that are productive across more than code generation while remaining practical to deploy. Granite Code introduces models spanning multiple sizes and coding tasks, achieving strong performance across open-source benchmarks, with repository-level packing left for future work.
Problem
Smaller code-focused models can perform well on code generation, but may lag on other coding tasks such as fixing and explanation, while larger generalist models are expensive to deploy.
Method
Granite Code presents decoder-only foundation models in Base and other variants across four sizes, designed for enterprise software development and diverse code-related tasks.
Results
Granite Code models consistently reach state-of-the-art performance among open-source code LLMs across code generation, explanation, and bug-fixing tasks.
Takeaways & Limitations
The 3B-to-34B Granite Code family supports applications ranging from complex application modernization to on-device memory-constrained use cases.
Takeaways & Limitations
The models were not trained with repository-level file packing, which the authors leave as future work to improve repository-level code generation.
Abstract
from arXiv · showhide
Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and LLM-based agents are beginning to show promise for handling complex tasks autonomously. Realizing the full potential of code LLMs requires a wide range of capabilities, including code generation, fixing bugs, explaining and documenting code, maintaining repositories, and more. In this work, we introduce the Granite series of decoder-only code models for code generative tasks, trained with code written in 116 programming languages. The Granite Code models family consists of models ranging in size from 3 to 34 billion parameters, suitable for applications ranging from complex application modernization tasks to on-device memory-constrained use cases. Evaluation on a comprehensive set of tasks demonstrates that Granite Code models consistently reaches state-of-the-art performance among available open-source code LLMs. The Granite Code model family was optimized for enterprise software development workflows and performs well across a range of coding tasks (e.g. code generation, fixing and explanation), making it a versatile all around code model. We release all our Granite Code models under an Apache 2.0 license for both research and commercial use.
1 Introduction
Granite Code targets enterprise software development with models spanning sizes and coding tasks, addressing gaps in deployment cost, non-generation capabilities, transparency, and licensing. Extensive evaluations report strong performance across coding and mathematical benchmarks.
- Enterprise software development needs code LLMs for generation, explanation, fixing, testing, documentation, modernization, vulnerability detection, and translation.
- Current code LLMs trade off deployment cost against breadth: large generalists are expensive, while smaller code-focused models may lag on fixing and explanation.
- Enterprise adoption is also constrained by limited transparency about training data and processing, plus license terms that complicate usage.
- Granite Code introduces Base and Instruct model variants in four sizes: 3B, 8B, 20B, and 34B.
- The models use two-phase pretraining over 116 programming languages, followed by high-quality code and natural-language data to improve reasoning.
- 33.2% vs 21.3%: Granite-8B-Code-Base outperforms CodeGemma-8B by almost 12 points on HumanEvalPack despite using fewer training tokens.
- Granite models show strong results across coding tasks and sizes, while Granite-8B-Code-Base also exceeds most similarly sized models on mathematical benchmarks.
2 Data Collection
Granite Code training data is assembled from public code and language sources, then filtered, deduplicated, and screened for harmful content, personal information, and malware.
- The data pipeline crawls and filters code, deduplicates it, applies HAP and PII filtering, and adds high-quality natural-language data.
- Code is retained from 116 programming languages after language identification based solely on file extension, followed by quality filtering rules.
- Exact SHA256 and fuzzy deduplication remove identical or near-identical code files to reduce data bias.
- HAP filtering removes documents exceeding a distribution-based threshold derived from keyword occurrences, including comments.
- PII is redacted into replacement tokens or synthetic IP addresses, and ClamAV scans remove malware from source code.
- Additional datasets cover web, mathematical, academic, and instruction-tuning text to improve language understanding and mathematical reasoning.
3 Model Architecture
Granite Code uses decoder-only Transformer architectures across four model scales, with scale-specific attention, normalization, positional, and context-length choices. The 34B model is created through depth upscaling of the 20B model.
- Granite Code models use Transformer decoder architectures with pre-normalization applied before attention and MLP blocks.
- 3B: The 3B model uses RoPE, multi-head attention, SwiGLU, RMSNorm, and a 2048-token context length.
- 8B: The 8B model adds grouped-query attention for a performance–inference-efficiency tradeoff and uses a 4096-token context length.
- 20B: The 20B model uses learned absolute positions, multi-query attention, GELU, LayerNorm, and an 8192-token context length.
- 34B: The 34B model is formed by depth-upscaling the 20B model, concatenating truncated original and duplicate networks into an 88-layer model.
4 Pretraining
Granite Code pretraining combines large-scale code training with a second phase of code and natural-language data, using causal language modeling and fill-in-the-middle objectives. A weighted objective with α = 0.5 supports strong completion and infilling performance.
- Pretraining is organized into two phases, with code-only training first and mixed code, technical, mathematical, and web data second.
- Granite Code models are trained on 3.5T to 4.5T tokens of code and code-related natural-language data using byte-pair encoding.
- The causal language-modeling and fill-in-the-middle objectives train models for both standard completion and context-based token insertion.
- 4.2 Training Objective: The overall training loss is a weighted combination of causal language modeling and fill-in-the-middle objectives.
- 4.2 Training Objective: α = 0.5 yields state-of-the-art performance on both code completion and code infilling tasks.
- 4.2 Training Objective: Fill-in-the-middle is used during pretraining but dropped during instruction finetuning by setting α = 1.
- Training uses AdamW, scheduled learning rates, large token batches, fused attention and optimizer kernels, and mixed parallelism on A100 and H100 clusters.
5 Instruction Tuning
Granite Code instruction tuning combines permissively licensed code, math, language, and commit-based instruction data to improve instruction following while supporting efficient finetuning. The pipeline uses optimization and memory-saving techniques that enable long-context finetuning of large models.
- Instruction tuning uses permissively licensed data to enhance instruction-following capabilities for enterprise applications.
- The instruction-tuning mixture includes commit-based code data, math datasets, code-instruction datasets, and language-instruction datasets.
- Three training epochs use a cosine scheduler, 250 warmup steps, and an initial learning rate of 10−5.
- Embedding noise improved overall answer quality for the instruction model.
- FlashAttention 2, Padding-Free Transformer, and full activation checkpointing reduce finetuning memory use and redundant computation.These techniques enabled Granite-20B-Code finetuning with 8K context on one node using 8×A100 GPUs within a few hours.
6 Evaluation
Granite Code models are evaluated across diverse coding tasks, languages, and repository settings using standardized comparisons. They show strong performance across model sizes, often matching or outperforming similarly sized or larger open-source models, while retaining some task-specific limitations.
- Evaluation scope: The evaluation covers code generation, explanation, fixing, editing, math reasoning, repository completion, infilling, and data-science workflows across multiple benchmarks.Comparisons use common scripts and environments, with benchmark-specific decoding and sampling settings.
- Multilingual generation: Granite-8B-Code-Base performs best on 16 of 18 MultiPL-E languages compared with similarly sized CodeLlama-7B.Granite-34B-Code-Base also outperforms CodeLlama-34B on most languages.
- Python generation: At mid parameter ranges, Granite models beat CodeLlama-7B and CodeLlama-13B by approximately 5% and 15% on average across MBPP and MBPP+.Granite-34B-Code-Base differs from CodeLlama-34B by only 0.9% on average across both benchmarks.
- Data-science tasks: Granite-8B-Code-Base achieves the best average DS-1000 performance at 34.5%, outperforming similarly sized models across seven libraries.Granite-3B-Code-Base also exceeds CodeGemma-2B by more than 12 points on average.
- Repository-level generation: Granite-20B-Code outperforms StarCoder2-15B and CodeLlama-34B on all four RepoBench metrics across Python and Java.The result holds despite the models not being trained with repository-level file packing.
- Repository-level generation: On CrossCodeEval, no model is best across every language and metric, indicating that uniformly high multilingual performance remains challenging.Granite-20B-Code-Base beats CodeLlama-13B on Python, Java, and C# but trails it on TypeScript.
- Infilling: Granite Code models significantly outperform StarCoder and StarCoder2 on SantaCoder-FIM across model sizes, but scaling from 8B to 34B yields no improvement.The result supports smaller models for FIM code-completion tasks.
6.2 Code Explanation and Fixing
The evaluation tests Granite models on code explanation and fixing across six programming languages, extending assessment beyond code synthesis. Granite base and instruction-tuned models perform strongly, with instruction tuning improving results across languages.
- Code explanation: Granite Code base models significantly outperform other state-of-the-art base code models on HumanEvalExplain.Granite-8B-Code-Base beats CodeLlama-34B by 9.3% on average and approaches CodeLlama-70B.
- Code explanation: Granite-34B-Code-Instruct reaches an average HumanEvalExplain score of 41.9%, close to CodeLlama-70B-Instruct at 41.1%.Instruction tuning significantly improves the performance of all base models across languages.
- Code fixing: On HumanEvalFix, Granite base models again significantly outperform other base models, while the 8B and 20B instruct models lead models below 34B parameters.Instruction tuning improves performance for almost all evaluated models.
- Code fixing: Granite-8B-Code-Instruct consistently outperforms the compared open-source instruction-tuned general language models.The comparison is presented in Figure 3.
- Overall findings: The results show that Granite base and instruct models support code generation as well as code fixing and explanation.These capabilities are evaluated as distinct tasks across six programming languages.
6.3 Code Editing and Translation
Granite Code models perform strongly on instructional code editing and translation tasks, with better edit quality than competing models and easily parseable translation outputs.
- Code Editing: Granite Code models achieve better CanItEdit pass rates with fewer unnecessary code changes than CodeGemma and CodeLlama.Pass@1 measures functional correctness, while ExcessCode measures conciseness and precision of edits.
- Code Translation: CodeLingua evaluates instruction-tuned models by Pass@1 translation accuracy across five target languages.The benchmark combines programs from Avatar and CodeNet and supplies unit tests for translated programs.
- Code Translation: Granite and CodeGemma produce nicely formatted translation outputs that are easier to parse than CodeLlama generations.Extracting only relevant code from CodeLlama outputs requires non-trivial post-processing.
- Code Translation: Go is the hardest target language to translate to, while C has the highest translation success rate among Granite models.
6.4 Code Reasoning, Understanding and Execution
Granite Code models show competitive code reasoning and execution performance across CRUXEval and mathematical benchmarks, with larger models and the 8B model often performing strongly.
- Code Reasoning and Execution: Granite Code performance on CRUXEval-I and CRUXEval-O increases as model size scales from 3B to 34B parameters.CRUXEval tests input prediction and output prediction over 800 Python functions and input-output pairs.
- Code Reasoning and Execution: At 7B–8B parameters, CodeGemma-7B outperforms all models on both CRUXEval tasks, while no 3B model is consistently best.Granite-3B-Code-Base beats CodeGemma-2B on CRUXEval-I but trails it on CRUXEval-O; Granite-34B-Code-Base shows the reverse pattern against CodeLlama-34B.
- Mathematical Reasoning: Llama-3-8B-Base generates invalid programs on tool-aided MATH+Py and GSM8K+Py tasks, producing very low scores.
- Mathematical Reasoning: Granite-8B-Code-Base achieves top-1 or top-2 performance across four chain-of-thought and two tool-aided mathematical reasoning tasks.The evaluation includes MATH, GSM8K, SAT, OCW, MATH+Py, and GSM8K+Py.
6.6 Calling Functions and Tools
Granite Code models improve substantially on function and tool calling as they scale, while instruction tuning strengthens performance and larger models improve robustness to code perturbations.
- Function and Tool Calling: Overall BFCL accuracy rises from 25.65% for Granite-3B-Code-Base to 57.12% for Granite-34B-Code-Base.BFCL evaluates function calling across simple, multiple, parallel, and parallel-multiple function-call categories using AST and executable evaluation.
- Function and Tool Calling: Granite-8B-Code-Instruct beats CodeLlama-7B-Instruct by 22% on AST Summary, 14% on Execution Summary, and 12% on Overall accuracy.
- Function and Tool Calling: Instruction tuning raises Granite-8B-Code overall accuracy by 17.88% over Granite-8B-Code-Base.The figure reports more noticeable instruction-tuning improvements for Granite models than for the compared base models.
- Robustness: Granite-34B-Code-Base consistently outperforms CodeLlama-34B across all four ReCode perturbation categories.ReCode perturbs docstrings, function names, and code to assess robustness of code generation.
7 Conclusion
The paper presents Granite Code as a versatile open model family for enterprise software development, spanning sizes, tasks, and deployment settings.
- Conclusion: Granite Code models range from 3B to 34B parameters and support tasks including generation, bug fixing, explanation, documentation, and repository maintenance.The family is positioned for complex application modernization and memory-constrained on-device use cases.
- Conclusion: Across diverse code-related tasks and programming languages, Granite Code models consistently reach state-of-the-art performance among open-source code LLMs.The conclusion reports average performance matching or exceeding CodeGemma, StarCoder2, and Llama3 on generation, explanation, and bug-fixing tasks.
- Conclusion: The models are released under the Apache 2.0 license for research and commercial use.
A Programming Languages
The passage enumerates a broad set of programming languages and code-related technologies, spanning established, modern, domain-specific, and tooling-oriented entries.
- The list includes established languages such as ABAP, Ada, Assembly, C, COBOL, FORTRAN, Java, and JavaScript.
- It also names modern and specialized languages including Clojure, Cuda, Dart, Elixir, Elm, Haskell, Julia, Kotlin, Lean, and Lisp variants.
- The enumeration covers code-related technologies and formats such as ANTLR, CMake, Dockerfile, GraphQL, HTML, JSON, JSX, and Jupyter.