Source-linked AI summary
Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
Wei Lu, Rachel K. Luu, Markus J. Buehler
TL;DR
Domain adaptation requires strategies that introduce specialized knowledge while retaining existing capabilities, yet systematic comparisons remain limited. This paper evaluates continued pretraining, supervised and preference-based fine-tuning, and model merging across model families, finding strong domain performance from advanced optimization combined with merging. It also identifies scope limits involving model scale, prompting, and real-world validation of generated designs.
Problem
Domain-specific fine-tuning must add specialized knowledge while preserving earlier capabilities, but systematic comparisons are limited and training resources and original datasets are often unavailable.
Method
The study systematically evaluates CPT, SFT, DPO, ORPO, and SLERP model merging across Llama and Mistral variants using consistent training and benchmark experiments.
Results
SLERP-merged models generally achieve the highest benchmark accuracy, especially with DPO and ORPO, while the best non-merged strategy differs across Llama and Mistral models.
Takeaways & Limitations
CPT and SFT provide foundations for domain adaptation, while DPO or ORPO followed by SLERP merging is a viable approach for strong domain performance in larger models.
Takeaways & Limitations
The study keeps prompting consistent despite possible model-specific prompt responses, and bioinspired designs require further research for real-world validation.
Abstract
from arXiv · showhide
The advancement of Large Language Models (LLMs) for domain applications in fields such as materials science and engineering depends on the development of fine-tuning strategies that adapt models for specialized, technical capabilities. In this work, we explore the effects of Continued Pretraining (CPT), Supervised Fine-Tuning (SFT), and various preference-based optimization approaches, including Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO), on fine-tuned LLM performance. Our analysis shows how these strategies influence model outcomes and reveals that the merging of multiple fine-tuned models can lead to the emergence of capabilities that surpass the individual contributions of the parent models. We find that model merging leads to new functionalities that neither parent model could achieve alone, leading to improved performance in domain-specific assessments. Experiments with different model architectures are presented, including Llama 3.1 8B and Mistral 7B models, where similar behaviors are observed. Exploring whether the results hold also for much smaller models, we use a tiny LLM with 1.7 billion parameters and show that very small LLMs do not necessarily feature emergent capabilities under model merging, suggesting that model scaling may be a key component. In open-ended yet consistent chat conversations between a human and AI models, our assessment reveals detailed insights into how different model variants perform and show that the smallest model achieves a high intelligence score across key criteria including reasoning depth, creativity, clarity, and quantitative precision. Other experiments include the development of image generation prompts based on disparate biological material design concepts, to create new microstructures, architectural concepts, and urban design based on biological materials-inspired construction principles.
1 Introduction
The paper frames domain adaptation as a systematic comparison of fine-tuning and optimization strategies that add specialized knowledge while preserving prior capabilities. It also examines model merging as a route to nonlinear capability gains and uses structured scientific data processing to support complex problem solving.
- Research motivation: Domain-specific fine-tuning must add new knowledge while retaining capabilities learned during earlier training stages.Training from scratch is often infeasible because of cost and unavailable original datasets.
- Training strategies: LoRA, CPT, SFT, and preference-based optimization represent complementary strategies for adapting models to specialized domains.CPT introduces domain knowledge, while SFT supports task performance and downstream interaction requirements.
- Model merging: Model merging combines differently trained models and can produce nonlinear interactions that yield functionalities absent from either parent model.The paper presents merging as potentially improving accuracy and adaptability for complex real-world challenges.
- Research gap: Systematic exploration remains limited because LLM training is complex, expensive, and time-consuming.Developers often focus on a single approach that produces acceptable results rather than comparing strategies consistently.
- Study approach: The study investigates fine-tuning and optimization strategies through consistent experiments and processes raw, distilled, and conversational data into interconnected insights.The overview emphasizes question-answer or instruction-response pair generation from scientific information.
2 Results and Discussion
Across Llama-3.1 and Mistral-7B variants, SLERP merging generally produced the strongest benchmark performance, especially after preference optimization. The analyses also report nonlinear gains over parent-model averages, strategy-dependent clusters, and reduced benefits from noisier CPT data.
- Model performance: SLERP-merged Llama-3.1 variants, especially those combined with DPO or ORPO, generally achieved the highest benchmark accuracy.The best non-merged strategy was Instruct-CPT-SFT-DPO.
- Model performance: SLERP-merged Mistral-7B variants likewise generally achieved the highest accuracy, while Base-CPT-SFT was the best non-merged strategy.Instruct-CPT-SFT performance was very similar to Base-CPT-SFT.
- Training duration: The Instruct model improved consistently across five CPT epochs and peaked at epoch 5, whereas the Base model fluctuated and scored highest at epoch 1.The original Figure 5 merges used CPT results from epoch 5.
- Synergistic effects: Merged-model performance substantially exceeded the simple average of parent-model scores for both Llama-3.1 and Mistral-7B, indicating nonlinear synergistic effects.The reported comparison uses observed merged performance against an expected score from averaging the two parent models.
- Synergistic effects: SLERP’s curved interpolation is presented as preserving parameter-space structure while enabling nonlinear combinations that can produce behaviors unavailable to either parent model.The authors also discuss possible regularization and mitigation of catastrophic forgetting as contributing mechanisms.
- Strategy analysis: Clustering separated models into performance-based groups, with multi-stage strategies—especially ORPO—forming higher-actual-score clusters across both model families.Hierarchical clustering grouped models with similar training strategies into related subclusters.
- Data quality: Using the larger, noisier CPT dataset decreased performance, while the Mistral variant trained on the original integrated dataset reached 0.81 versus 0.80 for the extended dataset.The authors associate the decline with lower-quality, more variable text and leave further dataset-composition analysis for future work.
3 Conclusions
The study finds that fine-tuning strategy and model scale jointly shape domain-specific LLM performance: larger models benefit from advanced optimization and merging, while the smallest model shows limitations. The conclusions also cover diminishing returns from pretraining, realistic materials applications, and directions for further study.
- Model scale: Larger 7B and 8B models show novel capabilities under advanced optimization, whereas the 1.7B SmolLM model does not.The authors describe this pattern as suggesting a scale threshold for benefits from techniques such as SLERP merging.
- Fine-tuning strategies: CPT-SFT is critical across models, while additional DPO or ORPO and model merging mainly improve larger architectures.Model merging is reported as effective primarily in larger models and detrimental in the smallest model considered.
- Performance outcomes: Fine-tuning consistently improves performance, with Llama achieving the highest absolute scores while SmolLM records the greatest relative improvement.The comparison distinguishes absolute performance from improvement over each model’s base model.
- Pretraining scale: More pretraining generally yields better performance but reduces the relative improvements available from later training.The token comparison is approximate because the model architectures, strategies, and datasets differ.
- Future work and limitations: The conclusions identify prompt engineering, dataset improvement, emergent-behavior mechanisms, and model-size thresholds as areas for future research.The study also notes that larger datasets are not necessarily beneficial and that the reported approaches have higher computational cost than simpler LoRA-based methods.
- Applications: Consistent human-AI conversations and image-prompting experiments extend assessment beyond question answering into reasoning, structured outputs, and materials design.Applications include biological-material microstructures, futuristic architecture, and urban or cityscape design.
4 Materials and methods
The study assembles domain-specific training corpora from biological-materials and spider-silk literature, combining original, newly collected, and extended datasets. Spider-silk collection used a multi-stage process to identify, screen, and retrieve papers for training.
- 4,323 spider-silk papers were downloaded as PDFs for training through publisher APIs, manual downloads, and library requests.
- 4,520 spider-silk papers were initially collected from Web of Science, then processed through DOI lookup, duplicate and relevance screening, and retrieval.
- The extended dataset adds 3,826 biological-materials papers retrieved from broader search terms and varied text formats.
- The standard training dataset combines 1,034 biological-materials papers with 4,323 spider-silk papers.
- Training generally used the original and spider-silk datasets, while selected experiments added the extended dataset to examine larger, more varied data.
4.2 Dataset processing and preparation
PDFs were converted into markup with Marker to create the best-quality raw dataset for continued pretraining.
- Marker converts scientific-paper PDFs into markup format for the best-quality raw dataset used in CPT.
4.3 Dataset distillation and preparation for supervised learning
Scientific papers were transformed into structured training data through extraction, summarization, quantitative-property collection, and JSON organization. Different fine-tuning stages used distinct combinations of raw text, question-answer pairs, and preference triples.
- Raw scientific-paper text was processed with general-purpose LLMs to produce question-answer pairs and other higher-quality learning data.
- The structured representation included titles, insights, facts, and details describing papers’ findings, observations, methods, and technical elements.
- The pipeline extracted research questions, corresponding answers, comparisons, quantitative material properties, summaries, and structured article details.
- The spider-silk example records nanomembrane thicknesses of 250 nm to 440 nm and an ultimate engineering strain above 220%.
- Rejected answers for DPO/ORPO were generated by instructing an LLM to produce scientifically incorrect responses.
- For each paper, five distinct data sets produced around 21,000 spider-silk and 5,000 bio-inspired-materials examples.
- CPT used high-quality structured data with raw text, SFT used question-answer pairs, and DPO/ORPO used prompt-response-rejected-response triples.
4.4 Benchmark development and prompting
Benchmarking used constrained prompts for multiple-choice and true/false questions, with deterministic decoding and answer normalization. The questions covered biological materials across factual, conceptual, numerical, and experimental topics.
- The benchmark prompt instructed models to select the correct answer with one word or letter.
- Evaluation used deterministic inference without sampling at T = 0 to ensure consistent results.
- A custom answer checker converted model outputs into a single choice letter or T/F label.
- Questions covered biological-material properties, applications, production, biology, materials science, gene studies, and methodologies.
- The benchmark included factual, conceptual, numerical-calculation, experimental-technique, and scenario-based questions.
4.5 Benchmark development for model assessment
The study develops two domain-specific benchmarks for assessing LLMs on spider silk and broader biological or bio-inspired materials. The benchmarks combine knowledge assessment with questions targeting reasoning, logic, creativity, and quantitative understanding.
- Two benchmarks cover spider silk and general biological or bio-inspired materials.The datasets are named the Spider silk benchmark and the Bioinspired/biological materials benchmark.
- Spider silk benchmark: 159 spider-silk question-answer pairs comprise 105 basic and 54 advanced questions.The basic set assesses knowledge and understanding, while the advanced set addresses logic, reasoning, and related scenarios.
- Spider silk benchmark: The spider-silk benchmark spans descriptive, conceptual, analytical, numerical, comparative, and experimental question categories across materials, biology, applications, genes, production, and methodology.
- Spider silk benchmark: The spider-silk benchmark is designed to assess knowledge recall alongside logic, reasoning, and creativity.Each entry records the question, answer, type, knowledge category, assessment area, and any referenced paper.
- Bio-inspired/biological materials benchmark: The expanded biological-materials benchmark retains categories for general, specific, numerical, and non-biological questions while doubling the previous question count.These categories cover broad trends, article-specific phenomena, literature-based quantitative values, and distinctions between synthetic and biological materials.
4.6 Training approach
The training pipeline combines continued pretraining, supervised fine-tuning, and preference optimization, using domain-specific data alongside model-specific instruction mixtures. DPO learns from chosen and rejected responses, whereas ORPO integrates preference alignment into supervised fine-tuning without a reference model.
- Data and infrastructure: Llama and Mistral training uses the same data sources but adjusts the SFT and DPO/ORPO mixture to match SmolLM-Instruct development while adding domain-specific data.
- Data and infrastructure: Llama and Mistral models train on one 8×H100 node, while SmolLM trains on a single GPU.The implementations use Hugging Face’s alignment handbook and TRL trainers for SFT, DPO, and ORPO.
- Continued pretraining: Continued pretraining feeds raw text to the model and uses sample packing to improve training efficiency.A start token is added at the beginning of each chunk.
- Supervised fine-tuning: Supervised fine-tuning uses each model’s chat template to format question-answer examples and also applies sample packing.
- Preference optimization: DPO optimizes responses using chosen and rejected examples, directly maximizing the likelihood objective associated with the preference data.The training uses a chat template to incorporate conversational feedback and scientific preferences.
- Preference optimization: ORPO adds a log-odds-ratio term to the negative log-likelihood loss, eliminating the need for a reference model or separate preference-alignment phase.Its data contain a prompt, chosen response, and rejected response; rejected answers receive an additional penalty while chosen answers receive an adaptation signal.
4.7 Model merging
Model merging uses SLERP to interpolate fine-tuned parameter sets along a spherical path after normalization, then restores the original parameter magnitude. The paper links this nonlinear combination to potentially synergistic capabilities, especially when models are diverse and sufficiently large.
- SLERP method: SLERP normalizes two parameter vectors to the unit hypersphere before interpolating between them.The angle between the vectors is computed from their dot product.
- SLERP method: The SLERP interpolation parameter t ranges from 0 to 1, returning the first normalized model at t = 0 and the second at t = 1.
- SLERP method: The final interpolated vector is rescaled to the original magnitude after spherical interpolation.This produces the merged parameter vector used by the model.
- SLERP rationale: SLERP is presented as a geometry-aware alternative to linear interpolation that can avoid high-loss regions in parameter space.The paper describes this as useful for combining strengths from different pretrained models.
- Emergent capabilities: Nonlinear parameter combinations may activate features that neither parent model exhibits individually, enabling new capabilities when complementary models are merged.The paper connects this behavior to overparameterization and ensemble diversity.
- Emergent capabilities: The effectiveness of merged-model capability discovery is speculated to scale with model diversity and parameter count.
4.8 Clustering analysis and dendrograms
The clustering analysis standardizes merged-model performance relative to parent-model expectations and examines relationships among SLERP strategies. K-Means and hierarchical clustering are used to identify performance groupings and strategy-related subclusters.
- Standardized scores: Merged-model performance is standardized using the expected and actual scores of each merged model.The expected score is based on the average performance of its two parent models, while the actual score is its observed performance.
- Standardized scores: The standardized expected and actual scores use the corresponding means and standard deviations across all merged models.
- Analysis parameters: Table 6 defines the parameters used in the clustering and dendrogram analysis.
4.9 Summary of best-performing model releases on Hugging Face
The section identifies the best-performing models and provides their corresponding Hugging Face hub IDs in Table 7.
- Table 7 summarizes the models identified as best-performing.
- Each listed model is paired with a corresponding Hugging Face hub ID.
- The release overview is presented as Table 7.
4.10 Multi-turn human-AI conversations
Multi-turn conversations use consistent human input and system prompts across experiments, with fixed generation settings that allow long, detailed outputs.
- Human input and system prompts are defined consistently for all experiments.
- Generation uses topk=512, topp = 0.9, and repetition_penalty = 1.1.
- The maximum output length is 1024 generated tokens to support long, detailed multi-turn conversations.
4.11 Image synthesis
Image synthesis uses leaf-trained FLUX models and biologically inspired prompting to steer outputs toward organic, abstract forms, with specified generation settings.
- Image synthesis uses lamm-mit/leaf-FLUX.1-dev, fine-tuned from FLUX.1-dev on leaf images with the keyword ‘<leaf microstructure>’.
- The primary model was trained for N = 2, 000 steps with adamw8bit, learning rate 0.0001, and rank r = 16 LoRA adapters on all linear layers.
- Biologically inspired cues are intended to steer generated solutions toward organic, abstract, biological forms.
- Image generation typically uses 25 denoising steps and guidance scale=3.5.
- An alternative model, lamm-mit/leaf-L-FLUX.1-dev, was trained for N = 4, 000 steps at a learning rate of 0.0001.
Code, data and model weight availability
The paper releases its training scripts, model weights, training-data listings, and source-article references through public repositories and supplementary files.
- Training scripts are available in the lamm-mit/LLM-finetuning GitHub repository.
- Model weights are available through the lamm-mit Hugging Face account.
- Training data is listed in Table 5, with downloaded-paper references and DOI numbers provided in three supplementary CSV files.
Conversation with: lamm-mit/SmolLM-Base-1.7B-CPT-SFT-DPO
The conversation develops a Leafy-Coraline composite by combining collagen, cellulose nanocrystals, and alginate with leaf-inspired hierarchical organization. It also connects collagen and leaves through shared structural-support and multiscale stress-distribution principles, yielding a tough, adaptable material concept.
- Collagen–leaf relationship: Collagen and leaves are linked by structural support and multiscale architectures that distribute stress while maintaining functionality.The conversation compares leaf veins and collagen fibrils as nested structures that support their respective biological systems.
- Material design: The design combines collagen fibrils, cellulose nanocrystals, and alginate in a leaf-inspired composite matrix.Collagen fibrils are wet-spun and crosslinked, while cellulose nanocrystals are incorporated through solvent casting.
- Applications: The proposed material is positioned for aerospace, biomedical-device, and sustainable-infrastructure applications.These applications follow the stated combination of mechanical robustness, adaptability, and self-healing.
- Bio-inspired principles: Hierarchical organization, gradient-like structure, and interpenetrating networks are proposed to improve load distribution, crack resistance, and toughness.The concepts draw on dragline silk nanofibrils, leaf tissue transitions, collagen fibers, and multiscale natural composites.