Source-linked AI summary
Arcee's MergeKit: A Toolkit for Merging Large Language Models
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
TL;DR
The open-source LLM ecosystem has produced many specialized checkpoints that cannot readily share capabilities, creating a need for practical model-merging infrastructure. MergeKit provides an extensible library for merging models across hardware settings and supports community-developed strategies. Its examples and reported use cases show merged models achieving enhanced, generalized capabilities, while merging models with different initializations remains less widely adopted.
Problem
Many separately fine-tuned LLMs are specialized for individual tasks, limiting their ability to leverage related models’ strengths and creating deployment challenges.
Method
MergeKit is an extensible open-source toolkit that executes diverse model-merging strategies efficiently across memory-constrained CPUs and accelerated GPUs.
Results
Merged models outperform Meditron-7B across medical benchmarks, while LERP- and SLERP-merged models outperform Llama2-7B chat on general benchmarks.
Takeaways & Limitations
Model merging supports versatile models that perform across multiple tasks or adapt to new domains without retraining from scratch.
Takeaways & Limitations
Merging methods for models with different initializations have not yet achieved the wide adoption and success of common-initialization methods.
Abstract
from arXiv · showhide
The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances in transfer learning, the process of fine-tuning pretrained models for specific tasks, has resulted in the development of vast amounts of task-specific models, typically specialized in individual tasks and unable to utilize each other's strengths. Model merging facilitates the creation of multitask models without the need for additional training, offering a promising avenue for enhancing model performance and versatility. By preserving the intrinsic capabilities of the original models, model merging addresses complex challenges in AI - including the difficulties of catastrophic forgetting and multitask learning. To support this expanding area of research, we introduce MergeKit, a comprehensive, open-source library designed to facilitate the application of model merging strategies. MergeKit offers an extensible framework to efficiently merge models on any hardware, providing utility to researchers and practitioners. To date, thousands of models have been merged by the open-source community, leading to the creation of some of the worlds most powerful open-source model checkpoints, as assessed by the Open LLM Leaderboard. The library is accessible at https://github.com/arcee-ai/MergeKit.
1 Introduction
Open-source LLMs have proliferated into specialized checkpoints, but separately fine-tuning and deploying them limits cross-task knowledge sharing. MergeKit addresses this gap as an extensible library for merging models into multitask systems.
- Open-source checkpoints span pretrained and instruction-tuned models across domains, but separately storing, deploying, and training them limits shared task knowledge.
- Model merging combines parameters from multiple models into one model, supporting multitask and continual learning while reducing catastrophic forgetting.
- MergeKit provides a centralized, open-source library for executing community-formulated merging strategies on memory-constrained CPUs and accelerated GPUs.
2 Background & Related Work
Model merging ranges from averaging or interpolating compatible checkpoints to task-vector, permutation, fusion, and architectural composition methods. MergeKit extends this ecosystem with passthrough and Franken-MoE methods, while practical evidence shows merged models can perform competitively across domains despite weaker adoption for different initializations.
- Foundations: Model merging integrates pretrained models into a unified model that retains their strengths, building on weight averaging, mode connectivity, and linear interpolation.
- Task-vector methods: Task Arithmetic, TIES, Model Breadcrumbs, and DARE manipulate task vectors to combine more models without degrading capabilities.
- Interpolation methods: SLERP extends weight averaging by interpolating between checkpoints along a spherical path that can have a lower loss barrier than direct linear interpolation.
- MergeKit extensions: MergeKit’s passthrough method builds unusually large models by piecewise combining layers, while Franken-MoE constructs sparse mixture-of-experts models from dense models.
- Different initializations: Methods for different initializations use permutation symmetry, neuron alignment, optimal transport, or feature correlation, but they have not achieved the adoption of common-initialization merging.
- Practical use cases: Merged models demonstrate competitive specialized-domain performance and support continuous or multitask learning without retraining from scratch.
3 Library Design: Key Design Principles
MergeKit is designed as an accessible, extensible toolkit for applying and sharing model merges across users and hardware environments. Its reusable architecture, resource-efficient execution, tutorials, and GUI support both experimentation and practical deployment.
- MergeKit provides tutorials and notebooks to guide users through effective toolkit use and adoption.
- 3.1 User-Centric Design: Intuitive Interface and YAML Configuration Control: YAML configuration files let users define complex merge operations without coding, making recipes repeatable, shareable, and remixable.
- 3.1 User-Centric Design: Intuitive Interface and YAML Configuration Control: Configuration files specify merge methods, input models, and parameters globally or per component, including layer-varying interpolated gradients.
- 3.2 Modularity: Plug-and-Play Components: MergeKit emphasizes composable, interchangeable components that can be extended, customized, experimented with, or reused in standalone scripts.
- 3.3 Scalability: Efficiency and Performance Optimization: Out-of-core tensor loading and a directed acyclic graph minimize working memory, redundancy, and resource use during model merging.
- 3.4 Mergekit Graphical User Interface (GUI): MergeKit-GUI runs on a Hugging Face-hosted A100 GPU and lets users upload configurations, select techniques, and execute merges interactively.
4 Extensibility of MergeKit
MergeKit is built for community extension: contributors can add merging strategies through defined interfaces and computational-graph components. Experiments show that supported methods can combine medical and general capabilities across benchmarks.
- 4 Extensibility of MergeKit: The project invites the community to develop novel merging strategies and enhancements that refine MergeKit.
- 4 Extensibility of MergeKit: New merge methods implement the interface in merge_methods/base.py and may require updates to graph.py, plan.py, or architecture.py.
- 4 Extensibility of MergeKit: The MergeKit-GUI demonstration illustrates the toolkit’s user-facing interface for model merging.
- 4.1 Practical Example: Applying Model Merging in Medical Domain: All merged models outperform Meditron-7B across USMLE, MedMCQA, and PubMedQA medical benchmarks.
- 4.1 Practical Example: Applying Model Merging in Medical Domain: LERP- and SLERP-merged models outperform the Llama2-7B chat model on general benchmarks.
- 4.1 Practical Example: Applying Model Merging in Medical Domain: Within the medical domain, SLERP appears to outperform the other evaluated merging methods.
- 4.1 Practical Example: Applying Model Merging in Medical Domain: The experiments indicate that model merging can produce more generalized models with enhanced capabilities across diverse applications.
5 Conclusion and Future Work
MergeKit is an open-source tool for integrating LLMs while documenting model-merging literature and supporting continued community-driven expansion.
- MergeKit integrates large language models through an open-source tool.
- The paper presents MergeKit’s functionalities and reviews recent model-merging literature from an engineering perspective.
- MergeKit supports adding new merging techniques and encourages contributions from the open-source community.
- The project is intended to continuously incorporate new methodologies through collaborative development.
Ethical Considerations
The project frames its open-source mission around democratized access, global collaboration, and ethical responsibility, including fairness, accountability, transparency, and bias mitigation.
- MergeKit aims to democratize access to advanced AI technologies while maintaining ethical integrity.
- The toolkit is intended to strengthen collaboration among researchers, developers, and practitioners worldwide.
- The project emphasizes fairness, accountability, and transparency in the open-source community.
- The authors highlight proactive identification and mitigation of biases in merged models.