Source-linked AI summary
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, Yongqiang Ma
TL;DR
Adapting LLMs with limited resources is difficult because efficient fine-tuning methods require substantial implementation effort across different models. LlamaFactory unifies these methods for customizable, coding-free fine-tuning of over 100 LLMs, and the framework is empirically validated on language modeling and text generation tasks.
Problem
Fine-tuning extremely large parameter counts with limited resources is challenging, and existing efficient fine-tuning methods lack a systematic framework spanning different LLMs with user customization.
Method
LlamaFactory uses modular components to unify efficient fine-tuning methods and training approaches across models, with a web interface for coding-free customization.
Results
The framework is empirically validated for efficiency and effectiveness on language modeling and text generation tasks.
Takeaways & Limitations
LlamaFactory provides an integrated way to fine-tune over 100 LLMs with diverse efficient techniques and customize training and evaluation without coding.
Abstract
from arXiv · showhide
Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at https://github.com/hiyouga/LLaMA-Factory and received over 25,000 stars and 3,000 forks.
1 Introduction
LLAMAFACTORY addresses the challenge of adapting LLMs with limited resources by unifying efficient fine-tuning methods across hundreds of models and providing coding-free customization. Its modular framework and interfaces support diverse training approaches, and the open-source project has seen substantial adoption.
- Limited resources make fine-tuning the extremely large parameter counts of LLMs difficult, while existing methods lack a unified, customizable framework across models.
- LLAMAFACTORY unifies efficient fine-tuning methods through scalable modules, enabling fine-tuning of hundreds of LLMs with minimal resources and high throughput.
- The framework streamlines generative pre-training, supervised fine-tuning, RLHF, and DPO, with command-line and web interfaces requiring minimal or no coding.
- Its Model Loader, Data Worker, and Trainer minimize dependencies on specific models and datasets, supporting reusable components across training approaches.
- LLAMABOARD provides a Gradio-based interface for customizing LLM fine-tuning without coding.
- Over 25,000 GitHub stars and 3,000 forks accompanied the framework’s release, alongside hundreds of open-source models built upon it.
2 Related Work
The related-work landscape includes frameworks targeting specific models, tasks, training methods, distributed computation, or consumer-device deployment. LLAMAFACTORY is positioned as supporting a broader range of efficient fine-tuning techniques and training approaches.
- Existing frameworks address specialized goals including Llama adaptation, chat completion, generative modeling, instruction training, distributed training, domain specialization, and consumer-device deployment.
- LLAMAFACTORY supports a broader range of efficient fine-tuning techniques and training approaches than the representative competitive frameworks discussed.
- The paper compares features across representative fine-tuning frameworks and lists compatibility among LLAMAFACTORY’s techniques.
3 Efficient Fine-Tuning Techniques
LLAMAFACTORY combines optimization and computation techniques to reduce the cost of fine-tuning LLMs. Its integrated design reduces memory use substantially while supporting different efficiency mechanisms.
- Efficient fine-tuning techniques target either parameter optimization costs or the time and space required for computation.
- Freeze-tuning updates a small subset of decoder-layer parameters, while GaLore projects gradients into a lower-dimensional space for memory-efficient full-parameter learning.
- Mixed precision, activation checkpointing, flash attention, and S2 attention reduce computation or memory demands, including for long-context fine-tuning.
- 0.6 bytes per parameter is the reported memory footprint after combining these techniques, versus 18 bytes in mixed precision or 8 bytes in half precision training.
4 LLAMAFACTORY Framework
LLAMAFACTORY organizes model loading, data processing, efficient training, and a no-code interface into a unified fine-tuning framework. Its modules support diverse models, datasets, training approaches, precision settings, and RLHF workflows.
- Architecture: LLAMAFACTORY consists of Model Loader, Data Worker, and Trainer modules supporting LLMs, VLMs, varied task data, and multiple training approaches.LLAMABOARD provides visual access to these modules for configuring and launching fine-tuning without code.
- Data Worker: The Data Worker standardizes datasets through loading, alignment, merging, and preprocessing into formats compatible with different tasks.A data description specification aligns dataset columns, while streaming mode alternates reads across datasets to preserve shuffling.
- Data Worker: Dataset preprocessing applies model-specific chat templates, computes loss on completions by default, and optionally uses sequence packing to reduce training time.Sequence packing is automatically enabled for generative pre-training.
- Trainer: The Trainer integrates efficient fine-tuning methods with pretraining, supervised fine-tuning, and preference-optimization workflows through replaceable components.It includes methods such as LoRA+, GaLore, BAdam, KTO, and ORPO, and supports distributed training with DeepSpeed.
- Trainer: Model-sharing RLHF uses adapters and value heads switched dynamically so one pretrained model can serve as policy, value, reference, and reward models.The approach is designed to enable entire RLHF training with no more than one pretrained model and supports consumer devices.
5 Empirical Study
The empirical study evaluates LLAMAFACTORY’s training efficiency and downstream adaptation across several models, methods, and text-generation tasks. QLoRA minimizes memory, LoRA often maximizes throughput and task performance, while GaLore can achieve lower perplexity on larger models.
- Evaluation Design: The evaluation measures training efficiency through memory usage, throughput, and perplexity, and adaptation effectiveness on downstream tasks.The study covers language modeling and text generation settings.
- Training Efficiency: The language-modeling experiment fine-tunes Gemma-2B, Llama2-7B, and Llama2-13B on roughly 400K PubMed tokens using full-tuning, freeze-tuning, GaLore, LoRA, and QLoRA.Perplexity on the training corpus is used to compare efficiency-related outcomes.
- Training Efficiency: QLoRA consistently has the lowest memory footprint because pretrained weights use lower precision, while LoRA achieves higher throughput with Unsloth optimization.Full-tuning Llama2-13B overflows memory and is not recorded.
- Training Efficiency: GaLore achieves lower perplexity on large models, whereas LoRA performs better on smaller models.The comparison uses perplexity on the training corpus.
- Downstream Tasks: The downstream study uses 2,000 training examples and 1,000 test examples from CNN/DM, XSum, and AdGen, reporting averaged ROUGE-1, ROUGE-2, and ROUGE-L scores.Several instruction-tuned models are evaluated with efficient fine-tuning methods.
- Downstream Tasks: LoRA and QLoRA achieve the best performance in most downstream cases, with exceptions for ChatGLM3-6B and Llama2-7B on CNN/DM and AdGen.Llama3-8B performs best among the evaluated models, while Yi-6B and Mistral-7B are competitive at similar sizes.
6 Conclusion and Future Work
LLAMAFACTORY is a unified, modular framework for efficient fine-tuning across over 100 LLMs, with LLAMABOARD enabling customization and evaluation without coding. The authors validate its efficiency and effectiveness and outline extensions to modalities, parallelism, and conversational fine-tuning.
- Conclusion: LLAMAFACTORY integrates diverse efficient fine-tuning techniques into a modular framework supporting over 100 LLMs.Its design minimizes dependencies among models, datasets, and training methods.
- Conclusion: LLAMABOARD enables customized fine-tuning and evaluation of LLMs without coding.
- Conclusion: The framework’s efficiency and effectiveness are empirically validated on language modeling and text generation tasks.
- Future Work: Future work will extend fine-tuning to audio and video modalities, integrate sequence and tensor parallelism, and explore self-play for conversational models.
7 Broader Impact and Responsible Use
LLAMAFACTORY has attracted researchers and practitioners, supported open-source model development, and contributed to the growth of open-source LLM communities. Its use requires adherence to model licenses.
- LLAMAFACTORY has attracted LLM researchers, supported models built on Hugging Face, and contributed to open-source community growth.
- Practitioners are encouraged to build LLMs upon the framework for societal benefit, while adhering to model licenses.