Source-linked AI summary
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, Sunghun Kim
TL;DR
Scaling LLMs efficiently can require complex training and inference changes. The paper introduces DUS, combining depthwise scaling with continued pretraining, and reports strong performance for SOLAR 10.7B and SOLAR 10.7B-Instruct. The study also notes unresolved DUS hyperparameter choices and substantial computational demands.
Problem
Existing LLM scaling methods can require non-trivial changes to training and inference frameworks, creating a need for simpler efficient scaling approaches.
Method
DUS scales a pretrained model depthwise and applies continued pretraining, while SOLAR 10.7B-Instruct adds instruction and alignment fine-tuning.
Results
SOLAR 10.7B outperforms comparable models across benchmarks, while SOLAR 10.7B-Instruct significantly outperforms Mixtral-8x7B-Instruct across evaluation metrics.
Takeaways & Limitations
DUS provides a simple approach for scaling high-performing LLMs from smaller models, with SOLAR models maintaining computational efficiency and broad framework compatibility.
Takeaways & Limitations
The optimal DUS hyperparameters remain undetermined because hardware limitations and the time and cost of continued pretraining restricted broader experiments.
Abstract
from arXiv · showhide
We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.
1 Introduction
The paper addresses the challenge of scaling LLMs efficiently without the complex training and inference changes required by many existing approaches. It introduces DUS and releases SOLAR models that show strong benchmark performance while retaining simple framework compatibility.
- Existing scaling methods can require non-trivial training and inference changes, motivating simpler ways to scale LLMs efficiently.
- DUS scales the base model’s layer count and continues pretraining without MoE modules or framework changes.
- SOLAR 10.7B has 10.7 billion parameters and outperforms Llama 2 and Mistral 7B across various benchmarks.
- SOLAR 10.7B-Instruct is fine-tuned for complex instruction-following tasks and significantly outperforms Mixtral-8x7B-Instruct across evaluation metrics.
- SOLAR 10.7B is released under the Apache 2.0 license to support wider access and application by researchers and developers.
2 Depth Up-Scaling
DUS constructs a deeper transformer by combining truncated copies of a pretrained base model, then applies continued pretraining to recover performance. The design avoids additional MoE machinery and targets efficient integration with existing frameworks.
- DUS uses pretrained base-model weights, depthwise scaling, and continued pretraining to build a larger LLM.
- The base model uses a 32-layer Llama 2 architecture initialized with pretrained Mistral 7B weights.
- Depthwise scaling removes the final m layers from the original model and the initial m layers from a duplicate, then concatenates the remaining models.
- For n = 32, s = 48, and m = 8, the resulting scaled model fits the targeted 7–13 billion parameter range under hardware constraints.
- The scaled model initially loses performance, but continued pretraining produces rapid recovery.
- Unlike MoE scaling, DUS requires no gating networks, dynamic expert selection, distinct training framework, or specialized CUDA kernels.
3 Training Details
SOLAR 10.7B-Instruct is fine-tuned through instruction tuning followed by alignment tuning. The training pipeline includes open-source data and synthesized math-focused datasets, including preference tuples derived from rephrased answers.
- Fine-tuning proceeds in two stages: instruction tuning and alignment tuning.
- Instruction tuning trains the model to follow QA-format instructions using open-source datasets and Synth. Math-Instruct.
- Alignment tuning further optimizes the instruction-tuned model toward human or strong-AI preferences using sDPO.
- Synth. Math-Alignment uses rephrased questions as prompts, rephrased answers as chosen responses, and original answers as rejected responses.
4 Results
SOLAR 10.7B is evaluated against other models and through ablations of instruction tuning, alignment tuning, and model merging. Results show strong benchmark performance, while dataset choices and merging affect task-specific behavior and hyperparameter exploration remains limited.
- Main Results: SOLAR 10.7B outperforms pretrained models of similar sizes, while SOLAR 10.7B-Instruct achieves the highest H6 score despite being smaller than Mixtral 8x7B-Instruct and Qwen 72B.H6 averages six benchmark tasks: ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, and GSM8K.
- Instruction Tuning Ablations: 69.21 H6 for SFT v2 changes little from 69.15 for SFT v1, but GSM8K rises from 52.24 to 57.32 while ARC, HellaSwag, and TruthfulQA decline.SFT v2 adds OpenOrca to the Alpaca-GPT4 data used by SFT v1.
- Instruction Tuning Ablations: 70.88 H6 is achieved by SFT v4, which adds Synth. Math-Instruct to SFT v1 and scores higher than SFT v3 across all tasks.SFT v3 also uses Synth. Math-Instruct, but starts from the SFT v2 dataset combination.
- Instruction Tuning Ablations: Merging SFT v3 and SFT v4 retains SFT v4’s high non-GSM8K scores while exceeding both models on GSM8K.The authors describe merging models with different task strengths as a promising way to improve general performance.
- Alignment Tuning Ablations: 73.06 H6 for DPO v1 improves on the 70.03 SFT base score, but GSM8K falls to 58.83; adding Synth. Math-Alignment raises GSM8K to 60.27 without negatively affecting other task scores.The authors conclude that adding Synth. Math-Alignment benefits H6.
- Alignment Tuning Ablations: Different merge methods have little effect on H6 or individual-task scores, while performance gaps between SFT base models do not always carry over after alignment tuning.The merge-method comparison uses Average with several weights and SLERP; the base-model comparison contrasts DPO v2 and DPO v3.
5 Conclusion
SOLAR 10.7B and SOLAR 10.7B-Instruct are 10.7-billion-parameter models built with depth up-scaling, showing strong NLP performance while maintaining computational efficiency.
- SOLAR 10.7B and SOLAR 10.7B-Instruct are depth up-scaled models with 10.7 billion parameters.
- The models outperform Llama 2, Mistral 7B, and Mixtral-7B-Instruct on essential NLP tasks while maintaining computational efficiency.
- The results indicate that DUS can scale highly performant LLMs from smaller ones.
- Further exploration of DUS could improve the approach and support another path for efficiently scaling LLMs.
Limitations
The study identifies limitations involving DUS hyperparameter exploration, computational demands, bias, energy use, and task-specific fine-tuning requirements.
- The choice to remove m = 8 layers was driven primarily by hardware limitations, and its optimality for performance remains undetermined.
- The time and cost of continued pretraining limited more comprehensive comparative experiments.
- SOLAR’s computational demands for training and inference might limit use by people with restricted computational resources.
- The model remains vulnerable to training-data biases that could produce skewed outcomes in certain situations, while its energy consumption raises environmental concerns.
- The fine-tuned variant still requires task-specific fine-tuning for optimal performance in specialized applications, and this process can be resource-intensive and not always effective.
Ethics Statement
The paper describes ethical safeguards for SOLAR, including low evaluated data contamination and stated attention to responsible experimental and operational practices.
- SOLAR 10.7B-Instruct showed low levels of data contamination in the reported evaluations.
- The experiments were designed to avoid potential ethical pitfalls.
- The paper states that SOLAR complies with privacy norms, respects intellectual property, and aims to ensure the absence of algorithmic bias.
- The authors characterize SOLAR’s ethical framework as robust and comprehensive.
A Contributions
The study introduces SOLAR 10.7B and its instruction-tuned variant, emphasizing broad benchmark performance, instruction following, and open commercial access.
- SOLAR 10.7B is released as a depthwise scaled and continually pretrained 10.7-billion-parameter model.
- The Apache 2.0 license permits commercial use and integration into diverse products and services.
- SOLAR 10.7B outperforms Llama 2 and Mistral 7B across reasoning, mathematics, and MMLU benchmarks.
- SOLAR 10.7B-Instruct is fine-tuned to improve adherence to complex instructions.
B.2 Mixture of Experts
Mixture-of-Experts models can improve efficiency and output diversity but introduce implementation challenges. DUS instead scales model depth without dynamic components and uses continued pretraining.
- MoE models can improve output diversity and computational efficiency, especially with sparse computation.
- MoE implementation is complicated by dynamic routing, load imbalance, and hardware constraints involving static tensor shapes.
- DUS scales models vertically rather than horizontally and avoids dynamism, reducing complexity relative to MoE.
- Continued pretraining is used to recover performance after scaling, because scaling without further pretraining degrades performance.
- Instruction tuning fine-tunes models on instruction, input, and output data to support more controlled task behavior.
- DPO directly increases the probability of positive responses without the reinforcement-learning complexity of RLHF.
B.6 Data Contamination
The paper identifies benchmark data contamination as a concern and distinguishes guideline, raw-text, and annotation contamination.
- Data contamination can occur through exposure to benchmark guidelines, raw benchmark text, or benchmark annotations during training.
C Additional Information
The additional information contains filtered FLAN-derived task names and contamination-test reporting for SOLAR 10.7B-Instruct.
- Table 8 lists task names used to filter FLAN-derived datasets such as OpenOrca.
- Table 9 reports contamination-test results for SOLAR 10.7B-Instruct, using SOLAR 10.7B as the reference model.
- All four tested benchmark datasets fall below the contamination threshold, while GSM8K has a noticeably higher value than the other datasets.