PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Optimization of Model Merging Recipes

Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, David Ha

arXiv:2403.13187v2cs.NE

TL;DR

Model merging is cost-effective but often depends on human intuition and domain knowledge to design effective recipes. This paper uses evolutionary search across parameter and data-flow spaces to discover cross-domain model combinations, producing Japanese language, Math, and Vision models with state-of-the-art benchmark results. The approach also yields a 7B Japanese LLM that surpasses some previous 70B models and a VLM strong on Japanese culture-specific content.

  • Problem

    Model merging is cost-effective but relies on human intuition, model-selection instincts, and domain knowledge, limiting systematic discovery across diverse open models and benchmarks.

  • Method

    Evolutionary Model Merge automatically searches parameter space and data-flow space to discover effective combinations of selected foundation models.

  • Results

    The method produces Japanese Math and Vision-language models with state-of-the-art results across several benchmarks, including a 7B Japanese LLM surpassing some previous 70B Japanese LLMs.

  • Takeaways & Limitations

    Evolutionary model composition can discover non-trivial cross-domain combinations and provide an alternative to expensive foundation-model development.

  • Takeaways & Limitations

    The merged models inherit source-model limitations, can produce logically incoherent responses, and were not instruction-tuned or aligned, so outputs may be factually flawed.

Abstract

from arXiv · show

Large language models (LLMs) have become increasingly capable, but their development often requires substantial computational resources. While model merging has emerged as a cost-effective promising approach for creating new models by combining existing ones, it currently relies on human intuition and domain knowledge, limiting its potential. Here, we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing for optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models like a Japanese LLM with Math reasoning capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with significantly more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. This work not only contributes new state-of-the-art models back to the open-source community, but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.

1 Introduction

The paper frames model merging as a cost-effective alternative to additional training, but argues that human-designed recipes limit systematic discovery. It introduces evolutionary model merging across parameter and data-flow spaces, demonstrating cross-domain Japanese models with strong benchmark performance.

  • Motivation: Model merging combines existing models without additional training, offering a cost-effective route to new models.Its growing use reflects the potential to democratize model development.
  • Motivation: Human-designed merging recipes depend on intuition, model-selection instincts, and domain knowledge, limiting systematic exploration across diverse models and benchmarks.The paper motivates evolutionary search as a more systematic alternative.
  • Contributions: Evolutionary Model Merge automatically discovers model combinations by searching both parameter space and data-flow space.The framework targets user-specified capabilities while leveraging existing open models.
  • Contributions: Cross-domain merging combines models from disparate domains, including non-English language with Math or Vision, to create capabilities difficult for human experts to design.The approach is presented as a way to move beyond leaderboard-defined task combinations.
  • Results: The generated Japanese Math LLM and Japanese VLM achieve state-of-the-art performance across various benchmarks without explicit optimization for those tasks.The VLM is designed for culturally specific Japanese content, while the LLM provides Math reasoning capability.
  • Impact: The paper open-sources EvoLLM-JP and EvoVLM-JP as competitive Japanese foundation models developed without gradient-based training.This supports further research on alternative approaches to foundation-model development.

2 Methods

The method uses evolutionary algorithms to optimize model merging in two orthogonal spaces: parameter mixing and layer-routing choices. It can combine these spaces into a unified search while retaining intact layers for data-flow merging.

  • Framework: The framework decomposes model merging into parameter-space and data-flow-space configuration spaces, then integrates them through evolutionary optimization.Its goal is to generate a merged model that outperforms the selected individual models.
  • Parameter Space: Parameter-space merging combines weights from models sharing an architecture, using task-vector analysis to target their task-specific strengths.The approach enhances TIES-Merging with DARE.
  • Data Flow Space: Data-flow-space merging preserves layer weights and evolves the inference path that tokens follow across layers from different models.A token can be routed from a layer in one model to a layer in another.
  • Data Flow Space: The initial data-flow search is restricted to serial, non-adaptive paths represented by sequences of layer indices under a fixed step budget.The method defers more flexible configurations to future work.
  • Search Space: Assuming M total layers and T inference steps, the unrestricted data-flow search has size (M + 1)^T, while the indicator-array parameterization reduces it to 2^T.The reduced space remains large but is described as tractable for evolutionary search.
  • Data Flow Space: Data-flow routing can expose layers to input distributions unlike those seen during their original use, potentially producing unexpected outputs.Preliminary studies found that swapping neighboring language-model layers reduced performance.
  • Integrated Strategy: The parameter-space and data-flow-space methods can be applied sequentially, including multi-objective combinations using methods such as NSGA-II.Parameter merging can first create objective-specific models before data-flow merging expands relevant performance metrics.

3 Results

The results focus on evolutionary discovery of models that combine capabilities across domains rather than optimizing only the narrow task sets common in merged-model benchmarks. The demonstrations produce a Japanese Math LLM and a culturally aware Japanese VLM.

  • Motivation: Most merged models optimize narrow leaderboard task sets, whereas the study searches for combinations spanning non-English language, Math, and Vision.The motivation is to discover solutions that may be difficult for human experts to design.
  • Demonstrations: Evolutionary Model Merge produces a Japanese LLM with Math reasoning by combining a Japanese LLM and an English Math LLM.This experiment applies the method described in the paper's methods section.
  • Demonstrations: The same approach produces a Japanese VLM proficient in culturally specific content by merging a Japanese LLM with an English VLM.The model targets Japanese culture-specific image understanding.

3.1 Evolving Japanese Math LLM

The study evolves merges of a Japanese language model and mathematics models, evaluating Japanese mathematical reasoning and general Japanese proficiency while isolating test data from optimization. The resulting merges improve mathematical performance and retain strong Japanese-language capability, with analyses indicating important contributions from the Japanese model and evolved scaling and layer-selection choices.

  • Source Models: The source set combines shisa-gamma-7b-v1 with WizardMath-7B-V1.1 and Abel-7B-002, all fine-tuned from Mistral-7B-v0.1.
  • Dataset: Evaluation uses 250 Japanese MGSM test samples, while evolutionary search uses the remaining 1069 GSM8k-derived samples to avoid test-set overfitting.
  • Baseline Performance: The source models score at or below 30.0 accuracy on MGSM-JA because the Japanese model has limited mathematical proficiency and the math models have insufficient Japanese command.
  • Experimental Results: The evolved models achieve JP-LMEH scores of 70.5 and 66.2, surpassing source models and a previous 70B Japanese LLM despite using 7B to 10B parameters.
  • Experimental Results: Compared with the source Japanese model, evolved models improve MGSM, JSQuAD, and JAQKET, although DFS performance drops on JNLI and MARC under different evaluation protocols.
  • Analysis: CMA-ES assigns substantial contributions across all three models, while DFS achieves best performance with the PS-merged model and Japanese model and uses alternating layer paths.Removing DFS scaling parameters causes a performance decline exceeding 20 percent.

3.2 Method Behavior Analysis

The analysis compares evolutionary merging with unoptimized merging and fine-tuning, tests robustness and scaling, and examines design choices underlying DFS performance.

  • Comparison with Unoptimized Model Merging: TIES-Merge and Frankenmerging decreased MGSM-JA and JP-LMEH performance, while DARE-TIES performed relatively better among unoptimized baselines.Frankenmerging produced an MGSM-JA score of 0, whereas DARE-TIES slightly improved MGSM-JA over the source models.
  • Comparison with Fine-tuning: Fine-tuning reached at most 43.2 MGSM-JA, below the proposed method’s score above 50, and often reduced JP-LMEH performance.The comparison used the same 1069 Japanese-translated GSM8k samples and tested multiple learning rates, epochs, LoRA, and full-parameter fine-tuning.
  • Source-Model Selection: Adding irrelevant source models caused limited degradation up to four additions, and performance did not collapse even with eight additional models.The distraction experiments were conducted for parameter-space merging but not DFS because of DFS’s large inference-time GPU memory requirement.
  • Scaling to Larger Models: At 13B, evolutionary merging significantly improved MGSM-JA over both source models, demonstrating scalability, while task-specific score trends remained similar across 7B and 13B models.Across both sizes, JSQuAD, JAQKET, and MGSM improved while JNLI and MARC slightly decreased relative to the original Japanese model.
  • Analysis on DFS Merging: Reversing DFS source-model order changed accuracy by more than 10 points, showing that layer order is a critical design choice.The initialization favors the first model early in search, reducing nonsensical layer stacks but sacrificing automatic model-order swapping.
  • Analysis on DFS Merging: Including the scaling matrix W was important for DFS, although simple scaling may fail when source models undergo substantially longer fine-tuning.The authors suspect prolonged fine-tuning increases cross-model incompatibility, potentially through permutations in weight matrices.

3.3 Evolving Japanese VLM

The method evolves the LLM component of a VLM while keeping its vision encoder and projection network fixed, and evaluates the resulting models on Japanese and culture-specific benchmarks.

  • Architecture: A VLM combines a vision encoder, an LLM for text generation, and a projection network mapping image features into the LLM’s embedding space.The vision encoder is fixed during training, while the projection network and optionally the LLM are trained on vision-language data.
  • Method: The method treats the VLM’s LLM component as a standalone language model and applies model-merging methods while fixing the vision encoder and projection network.The experiment merges shisa-gamma-7b-v1 with LLaVA-1.6-Mistral-7B, which share the Mistral-7B-v0.1 base model.
  • Datasets: The new benchmark includes JA-VG-VQA-500 and 50 questions over 42 culturally specific Japanese images in JA-VLM-Bench-In-the-Wild.The latter dataset was human-filtered and designed to require more nuanced responses than the 500-sample Japanese VQA test set.
  • Evaluation: The evolutionary merging search used a non-overlapping Japanese Visual Genome VQA subset to avoid leakage into JA-VG-VQA-500 evaluation.The evaluation compared against LLaVA-1.6-Mistral-7B and Japanese Stable VLM using deterministic decoding and Japanese-language-aware ROUGE-L scoring.
  • Experimental Results: Merged VLMs outperformed both baselines on Japanese culture-specific content and showed enhanced performance on JA-VG-VQA-500.Simple merging without evolutionary search did not achieve similarly strong performance, and qualitative results were generally more detailed and correct.

4 Discussion

The discussion presents evolutionary model merging as a general approach for discovering cross-domain model combinations and reports promising applications, while noting limitations inherited from source models and current model-selection requirements.

  • Discussion: Evolutionary techniques automatically discover non-trivial ways to merge open-source models from disparate domains, including language, mathematics, vision, and diffusion.The approach searches combinations that may be difficult for human experts to design.
  • Discussion: The method produced Japanese language and vision-language models with state-of-the-art results across multiple benchmarks without explicit optimization for those tasks.The Japanese LLM also exceeded some previous state-of-the-art 70B-parameter Japanese LLMs.
  • Discussion: The method currently requires users to select the source models supplied as ingredients for evolutionary search.The authors identify searching for candidate source models from a larger population as future work.
  • Further Applications and Impact: Evolutionary merging has also been applied to diffusion image generation, including combining SDXL-Lightning with conventional SDXL fine-tunes developed under different protocols.This application illustrates integration of models with varying development protocols.
  • Limitations: The authors acknowledge that merged models inherit source-model limitations, including occasional incoherent responses and potentially factually flawed outputs without instruction fine-tuning or alignment.These limitations constrain reliability of the current models.
  • Ethical and Societal Impact: The work is presented as a proof of concept whose broader use may require continued verification and refinement because merged models can exhibit unexpected behaviors, biases, or reduced interpretability.The authors connect these concerns to applications intended for mission-critical or wide public use.

Data Availability Statement

The paper reports that all datasets used in its experiments are publicly available, with sources listed for language-model and vision-language experiments.

  • Data Availability Statement: All datasets used in the paper are publicly available, with detailed sources provided.The statement covers the datasets used throughout the reported experiments.
  • LLM Experiments: LLM experiments use publicly available optimization data from the Japanese GSM8K test set and test data from the Grade School Math dataset.The sources are identified through Hugging Face and GitHub, respectively.
  • VLM Experiments: VLM experiments use JA-VG-VQA-500 and JA-VLM-Bench-In-the-Wild as benchmark datasets.Both datasets are available on Hugging Face.
  • VLM Experiments: The VLM optimization data come from a subset of the Japanese Visual Genome VQA dataset available under the train split of JA-VG-VQA-500.The paper directs readers to Section 3.3 for detailed VLM dataset references.

A Evaluation Details

The evaluation details identify the model-name mappings used in Table 1 and describe a Japanese LM Evaluation Harness configuration designed for direct comparison with established reports and leaderboards.

  • A Evaluation Details: Table 1 maps GPT-3.5, GPT-4, Llama 2 70B, Japanese Stable LM 70B, and Swallow 70B to their specific model versions.These mappings clarify which model checkpoints underlie the reported comparisons.
  • A Evaluation Details: The Japanese LM Evaluation Harness uses Stability AI Japan’s fork configured according to its convention.This configuration is compatible with Stability AI Japan’s report and Rinna leaderboards.
  • A Evaluation Details: Table 6 expands Table 2 to compare the authors’ models with a wider range of models.Some comparison numbers come from the Stability AI Japan report and Rinna leaderboard.
  • A Evaluation Details: 7B to 10B-parameter models achieve higher scores than all existing models below 70B parameters and scores on par with state-of-the-art 70B models.This comparison is reported for the authors’ models in Table 6.

B Evolving for License Specific Open-Source Models

The authors address licensing constraints by developing an alternative model using only true open-source components and report comparable evaluation results for that release.

  • B Evolving for License Specific Open-Source Models: EvoLLM-JP uses WizardMath-7B-V1.1, whose Non-Commercial, Research-only Microsoft License leads the release to adopt the same license.The authors state that this license is not truly open-source.
  • B Evolving for License Specific Open-Source Models: A separate experiment using only MIT- or Apache 2.0-licensed models produced the similarly performing EvoLLM-JP-A.This experiment was conducted to address the licensing issue while preserving applicability of the method.
  • B Evolving for License Specific Open-Source Models: EvoLLM-JP-A merges shisa-gamma-7b-v1, Arithmo2-Mistral-7B, and Abel-7B-002, all released under MIT or Apache 2.0 licenses.The model’s results are included in Table 6 for comparison.
  • B Evolving for License Specific Open-Source Models: 52.4 MGSM-JA score and 69.0 Japanese Language Model Evaluation Harness score are reported for EvoLLM-JP-A.The scores use the protocol described in Section 3.1.

C Case Study

The case studies test whether EvoLLM-JP and EvoVLM-JP combine Japanese-specific knowledge with mathematical reasoning or visual question answering. They show notable successes on culturally grounded examples, alongside a Japanese-math failure that exposes a transfer limitation.

  • EvoLLM-JP: A Japanese-math case requiring Setsubun knowledge and mathematical reasoning was answered correctly only by the merged model.WizardMath-7B-V1.1 failed even when the question was translated into English.
  • EvoLLM-JP: All models failed a Japanese box-volume problem involving wall thickness, while WizardMath-7B-V1.1 solved the original English version correctly with 72 cubic inches.EvoLLM-JP-v1-7B produced 351 cubic inches, shisa-gamma-7b-v1 produced 360, and the translated Japanese reasoning remained incorrect.
  • EvoVLM-JP: The VLM case studies evaluate Japanese-specific knowledge and VQA abilities across culturally grounded questions.Table 9 presents responses from the evaluated VLMs.
  • EvoVLM-JP: For Koi-nobori, LLaVA-1.6-Mistral-7B failed, whereas Japanese-Stable-VLM and EvoVLM-JP identified the tradition, with EvoVLM-JP giving the more fluent explanation.The comparison attributes the difference to Japanese cultural knowledge and highlights EvoVLM-JP’s fluency.
  • EvoVLM-JP: EvoVLM-JP alone identified Japan’s green traffic light as “blue” in Japanese, while the other VLMs gave incorrect or inconsistent color responses.The result reflects the Japanese cultural and linguistic convention described in the case study.
Loading 2403.13187v2…