Source-linked AI summary
Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment
Yasuaki Sumita, Tomoharu Iwata, Toshiyuki Tanaka
TL;DR
Base models tend to generate more diverse outputs, whereas aligned models tend to produce higher-quality outputs, and existing diversity methods have limitations in cost or semantic diversity. The paper merges aligned-model lower layers with base-model higher layers without additional training; across experiments, the split adjusts diversity and quality, and the merged models generally compare favorably with temperature sampling. The authors report that reversing the merge order yields results largely consistent with their layer-depth hypothesis.
Problem
Existing diversity methods either require costly fine-tuning or, like temperature sampling, mainly improve lexical rather than semantic diversity.
Source: §1 · p. 1Method
Results
Merged models had higher semantic diversity than the aligned model for almost all split positions, and their diversity–quality trade-off was better than temperature sampling in most settings.
Source: §4.1.2 · p. 5, §4.3.2 · p. 6, §5 · p. 8Takeaways & Limitations
Varying the layer split provides an operating point on the diversity–quality trade-off according to task requirements.
Source: §5 · p. 8The controllable diversity–quality range is limited to that of the base and aligned models, so merged-model quality and diversity depend on those models.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Alignment is known to change the output characteristics of large language models (LLMs). Base models tend to generate more diverse outputs, whereas aligned models tend to produce higher-quality outputs. The appropriate balance between the two depends on the task. Existing methods either require costly fine-tuning or, like temperature sampling, mainly improve lexical diversity rather than semantic diversity. We introduce a depth-aware FrankenMerging approach for controlling the diversity--quality trade-off between an aligned model and its base model without additional training. This approach combines the lower layers of an aligned model with the higher layers of its base model based on the hypothesis that lexical diversity is more closely associated with lower layers and semantic diversity with higher layers. Experiments showed that varying the number of layers taken from each model adjusts both diversity and quality. In most cases, the merged models exhibited higher semantic diversity than the aligned model, and at some split positions, they showed higher lexical diversity than both original models. They also demonstrated a better diversity--quality trade-off than temperature sampling in most settings. A supporting ablation, in which we reversed the merging order, yielded results largely consistent with the hypothesis.
1 Introduction
Base models tend to be more diverse, while aligned models tend to produce higher-quality outputs, so the preferred balance depends on the task. The paper proposes depth-aware FrankenMerging without additional training and reports that layer splits adjust diversity and quality, with results largely consistent with its layer-depth hypothesis.
1 Introduction
- Existing approaches either require costly fine-tuning or, like temperature sampling, mainly increase lexical rather than semantic diversity.The paper distinguishes lexical diversity as variation in vocabulary and semantic diversity as variation in content.Source: §1 · p. 1
- The approach controls the diversity–quality trade-off without additional training by combining an aligned model’s lower layers with a base model’s higher layers.It varies the split position to adjust the balance between the models.Source: §1 · pp. 1–2
- The experiments evaluate diversity and quality across translation, summarization, story generation, and dialogue generation, comparing split positions with temperature sampling.A reversed-order ablation tests the layer-depth hypothesis.Source: §1 · p. 2
- Varying the layer split adjusts both diversity and quality, and the approach provides a better diversity–quality trade-off than temperature sampling in most settings.The reversed-order ablation is largely consistent with the hypothesis that lexical and semantic diversity are associated with different layer depths.Source: §1 · p. 2
2 Related Work
Prior work examines diversity metrics and methods for increasing output diversity, while model merging has largely focused on performance. The paper introduces depth-aware layer merging to address diversity directly.
2.1 Diversity in Natural Language Generation
- Prior work distinguishes lexical diversity, often assessed with n-gram metrics, from semantic diversity, better assessed with embedding-based metrics.Studies also report that alignment’s effects on diversity depend on the diversity aspect and evaluation protocol.Source: §2.1 · p. 2
2.2 Improving Output Diversity in LLMs
- Diversity-enhancement methods use decoding or fine-tuning, but sampling methods primarily improve word-level diversity while fine-tuning approaches incur high computational costs.Temperature sampling flattens the token probability distribution; fine-tuning approaches modify training procedures.Source: §2.2 · p. 3
2.3 Model Merging
- Model merging integrates parameters from multiple pretrained models, and FrankenMerging selectively combines layers without modifying their parameters.The cited merging studies focus on model performance, not how layer depth relates to output diversity.Source: §2.3 · p. 3
- No previous work addressed model merging specifically to enhance output diversity or considered how layer depth relates to it.This study introduces depth-aware FrankenMerging to account for that relationship.Source: §2.3 · p. 3
3 Approach
The approach constructs a new model by taking an initial segment of layers from an aligned model and the remaining layers from its base model. The split is motivated by a hypothesis linking lexical and semantic diversity to different layer depths.
3 Approach
- The method combines the aligned model’s layers 1 through i with the base model’s layers i + 1 through L to create a new model.The models share an architecture and number of layers; i is selected from 1 through L − 1.Source: §3 · p. 3
- The design hypothesizes that lexical diversity is more closely associated with lower layers and semantic diversity with higher layers.Using the base model’s higher layers is expected to bring its semantic diversity into the merged model, while changing the split position is expected to adjust the diversity–quality balance.Source: §3 · p. 4
4 Experiments
Across four generation tasks and two model families, the experiments tested whether layer-split merging controls diversity and quality. The results show a better diversity–quality trade-off than temperature sampling in most settings, while reversed-order results were largely consistent with the layer-role hypothesis.
4.1.1 Setup
- The evaluation measured per-input lexical and semantic diversity across translation, summarization, story generation, and dialogue generation using 10 outputs for each of 500 inputs.Lexical diversity was one minus Self-BLEU; semantic diversity used cosine distance between sentence embeddings.Source: §4.1.1 · p. 4
4.1.2 Result and Discussion
- Merged models exceeded the aligned model in semantic diversity for almost all split positions and surpassed both original models in lexical diversity at some positions across many tasks.In Qwen 2.5 story generation, semantic diversity also exceeded the base model at intermediate split positions.Source: Figure 2 · p. 5, §4.1.2 · p. 5
4.2.1 Setup
- Quality evaluation covered translation and summarization, using COMET and BERTScore respectively, with scores averaged over 10 outputs per input.The authors note that reference similarity may not reliably indicate quality in story and dialogue generation, where many outputs can suit one input.Source: §4.2.1 · p. 5
4.2.2 Result and Discussion
- Quality changed with the split position: it generally rose toward aligned-model quality in LLaMA 2, while Qwen 2.5 merged models outperformed its aligned model at every position.The Qwen 2.5 aligned model performed worse than its base model in both evaluated tasks, likely because models received plain-text prompts without chat templates.Source: §4.2.2 · pp. 5–6, Figure 3 · p. 6
4.3.1 Setup
- The trade-off comparison varied temperature from 0.0 to 2.0 and summarized each method’s diversity–quality points with Hypervolume against a reference point.Confidence intervals and paired bootstrap tests used 10,000 resamples, with Holm correction for multiple comparisons.Source: §4.3.1 · p. 6
4.3.2 Result and Discussion
- Merged models had larger Hypervolume than temperature sampling in most settings, with significant advantages across lexical-diversity comparisons and several semantic-diversity comparisons.For lexical diversity, the advantage was significant for both model families and both tasks; for semantic diversity, it held for LLaMA 2, while Qwen 2.5 exceeded aligned-model sampling and was comparable to base-model sampling.Source: §4.3.2 · p. 6, Table 1 · p. 7
4.4.1 Setup
- The ablation reversed the layer sources, placing base-model layers 1 through i before aligned-model layers i + 1 through L, and compared diversity across split positions.Source: §4.4.1 · p. 7, §4.4.2 · p. 7
4.4.2 Result and Discussion
- Reversing the merging order generally reduced lexical diversity relative to the proposed order and kept semantic diversity at or below the base model, broadly supporting the depth-dependent hypothesis.The proposed order had significantly higher lexical diversity for 27–30 of 31 split positions in LLaMA 2 and 22–26 of 27 positions in three Qwen 2.5 tasks, but 9 of 27 in Qwen 2.5 translation.Source: §4.4.2 · pp. 7–8, Figure 5 · p. 8
5 Conclusion
The study presents depth-aware FrankenMerging as a way to adjust the diversity–quality trade-off, with layer counts selecting operating points across tasks. The merged models generally outperform temperature sampling on this trade-off, especially for lexical diversity.
5 Conclusion
- Across four tasks, varying the layer counts taken from each model selects an operating point on the diversity–quality trade-off.The merged models also showed a better trade-off than temperature sampling in most settings, particularly for lexical diversity.Source: §5 · p. 8
- The approach combines an aligned model’s lower layers with a base model’s higher layers to control the diversity–quality trade-off.Source: §5 · p. 8
- Reversing the merging order produced results largely consistent with the hypothesis linking lexical diversity to lower layers and semantic diversity to higher layers.The authors note this may reflect a depth-dependent effect on output diversity.Source: §5 · p. 8
Limitations
The study’s evidence and the method’s controllable range are bounded by its evaluated models and layer-depth analysis. Task-specific settings and real-application adjustment also remain open areas.
Limitations
- The experiments focus on two model families, each with approximately 7B parameters: LLaMA 2 and Qwen 2.5, which were aligned with different procedures.Source: Limitations · p. 8
- The controllable quality–diversity range is limited to that of the base and aligned models, so merged-model outcomes depend on those source models.The authors expect this limitation to be resolved as base-model performance improves.Source: Limitations · p. 9
- The layer-depth analysis compares two merging orders; finer-grained layer replacements could clarify individual layer roles.Source: Limitations · p. 9
- The appropriate hyperparameter value depends on task and purpose, and evaluating automatic adjustment in real applications remains future work.The authors suggest adjustment using target-task validation data.Source: Limitations · p. 9
- Because the approach modifies the model rather than decoding, it can be combined with sampling methods and diversity-oriented fine-tuning.The authors expect such combinations to further improve output diversity.Source: Limitations · p. 9
Ethical Considerations
The paper reports using models and datasets under their licenses and terms of use for research purposes. It also cautions that merged models may not retain all safety behavior acquired through alignment.
Ethical Considerations
- The authors state that models and datasets were used according to their licenses and terms of use, for research purposes.Source: Ethical Considerations · p. 9
- Merged models may not fully retain aligned models’ safety behavior, so practitioners should evaluate safety alongside diversity and quality before application use.The stated concern follows from using base-model higher layers in the merged model.Source: Ethical Considerations · p. 9
- WritingPrompts contains Reddit prompts and stories that may include offensive content; the generated outputs were not released.Source: Ethical Considerations · p. 9
A Experimental Settings
The appendix provides settings not described in Section 4.
A Experimental Settings
- The appendix supplies experimental settings that Section 4 does not describe.Source: A Experimental Settings · p. 11
A.1 Models
The study used base and aligned models from Hugging Face Hub for LLaMA 2 and Qwen 2.5.
A.1 Models
- The model pairs were LLaMA 2 (base and chat) and Qwen 2.5 (base and instruct), all obtained from Hugging Face Hub.Source: §A.1 · p. 11
A.2 Merging
The models were merged with MergeKit’s passthrough method, which uses selected layers without averaging, rescaling, or additional training.
A.2 Merging
- MergeKit’s passthrough method merges model layers without weight averaging, rescaling, or additional training.Source: §A.2 · p. 11
- The merged model takes input embeddings from the first slice’s model and final normalization and language-modeling head from the second slice’s model.Source: §A.2 · p. 11
- Merged models retain the original weight data types and use the base model’s configuration except for maximum context length and EOS tokens.All conditions use the base model tokenizer, which shares its vocabulary with the aligned model.Source: §A.2 · p. 11
A.3 Data
The appendix describes the datasets and preprocessing, prompting, generation, evaluation, and software and hardware used in the experiments.
A.3 Data
- The four task datasets were WMT24++ for translation, XL-Sum for summarization, WritingPrompts for story generation, and DailyDialog for dialogue generation.Summarization articles were capped at 1,024 base-tokenizer tokens; story prompts had their leading category tags removed, and dialogue inputs were speaker-prefixed with the final utterance removed.Source: §A.3 · pp. 11–12
A.4 Prompts
- All conditions used plain-text prompts without chat templates, while tokenizers added their default special tokens.Source: §A.4 · p. 12
A.5 Generation
- Generation used vLLM, with M outputs per input in one request and no post-processing; at temperature 0.0, one greedy output was replicated M times.Source: §A.5 · p. 12
A.6 Evaluation
- Evaluation used Self-BLEU with up to four-gram overlap, Qwen3-Embedding-0.6B for sentence embeddings, COMET for translation, and BERTScore F1 with roberta-large.Source: §A.6 · p. 12
A.7 Software and Hardware
- Each run used one NVIDIA RTX A6000 GPU with 48 GB of memory.Table 4 lists the software versions.Source: §A.7 · p. 12, Table 4 · p. 13