Source-linked AI summary
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Hong Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, Yingda Chen
TL;DR
Large-model training and fine-tuning are challenging because of substantial memory demands and fragmented support across models, datasets, and techniques. SWIFT addresses this by providing a unified infrastructure for training, fine-tuning, inference, evaluation, quantization, and related workflows. In agent training, it reports improved average metrics, including 8.25% for Qwen2 and 17% for LLaMA3, while its conclusion identifies incomplete Megatron coverage and limited depth in multimodal research.
Problem
Large models create substantial memory demands, while existing solutions lack broad, unified support for models, datasets, training techniques, and post-training processes.
Method
SWIFT provides a one-stop framework integrating lightweight training, alignment, quantization, inference, evaluation, model templates, and trainers for LLMs and MLLMs.
Results
SWIFT agent training improved average metrics by 8.25% for Qwen2 and 17% for LLaMA3.
Takeaways & Limitations
SWIFT offers a standardized solution for large-model training and development workflows, combining diverse model, dataset, technique, and post-training support.
Takeaways & Limitations
SWIFT’s Megatron support does not fully cover mainstream LLMs and MLLMs, and its multimodal research remains less in-depth.
Abstract
from arXiv · showhide
Recent development in Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) have leverage Attention-based Transformer architectures and achieved superior performance and generalization capabilities. They have since covered extensive areas of traditional learning tasks. For instance, text-based tasks such as text-classification and sequence-labeling, as well as multi-modal tasks like Visual Question Answering (VQA) and Optical Character Recognition (OCR), which were previously addressed using different models, can now be tackled based on one foundation model. Consequently, the training and lightweight fine-tuning of LLMs and MLLMs, especially those based on Transformer architecture, has become particularly important. In recognition of these overwhelming needs, we develop SWIFT, a customizable one-stop infrastructure for large models. With support of over $300+$ LLMs and $50+$ MLLMs, SWIFT stands as the open-source framework that provide the most comprehensive support for fine-tuning large models. In particular, it is the first training framework that provides systematic support for MLLMs. In addition to the core functionalities of fine-tuning, SWIFT also integrates post-training processes such as inference, evaluation, and model quantization, to facilitate fast adoptions of large models in various application scenarios. With a systematic integration of various training techniques, SWIFT offers helpful utilities such as benchmark comparisons among different training techniques for large models. For fine-tuning models specialized in agent framework, we show that notable improvements on the ToolBench leader-board can be achieved by training with customized dataset on SWIFT, with an increase of 5.2%-21.8% in the Act.EM metric over various baseline models, a reduction in hallucination by 1.6%-14.1%, and an average performance improvement of 8%-17%.
1 INTRODUCTION
Transformer-based large models now address diverse text and multimodal tasks, making efficient training and fine-tuning important. SWIFT unifies lightweight training, model support, and post-training operations in one framework.
- Transformer architectures now support text tasks such as classification and sequence labeling alongside multimodal tasks including VQA and OCR.
- Large models’ parameter counts and memory consumption make out-of-box training a significant bottleneck for developers.Earlier resource-efficient methods include Prefix Tuning, Prompt Tuning, P-Tuning, and LoRA.
- Quantization reduces memory by representing tensors with lower precision, including 8-bit, 4-bit, 1-bit, or 1.5-bit formats.BitsAndBytes, AutoGPTQ, and AWQ implement different quantization strategies.
- SWIFT unifies training and post-training workflows, supporting lightweight training, inference, evaluation, and model quantization.The framework streamlines technical components to reduce learning overhead and support efficient large-model development pipelines.
- SWIFT is introduced as a Transformers-compatible framework integrating PEFT and Optimum for pre-training, fine-tuning, human alignment, reinforced fine-tuning, and QLoRA.The framework also supports more than 550 LLMs, 200 MLLMs, and 150 text and multimodal datasets.
2 RELATED WORKS
Existing open-source frameworks combine training, fine-tuning, inference, evaluation, and deployment capabilities with varying model and technique coverage. The related systems provide complementary functionality across text, multimodal, and lightweight-training workflows.
- LLaMA-Factory supports pre-training, fine-tuning, and human alignment for over 100 text LLMs, plus training for several multimodal models.It also provides a Gradio WEB-UI, dataset evaluation, inference, and deployment workflows.
- FastChat combines training, inference, deployment, and evaluation using Transformers and PEFT, with LoRA, QLoRA, and XFormers support.Its deployment stack includes vLLM, SGLang, and LightLLM, while training support is relatively limited.
- Axolotl supports LoRA and QLoRA training across LLaMA, Mistral, Qwen, and Phi model series, along with inference and LoRA merging.It uses TRL, PEFT, and Transformers and encapsulates the mambassm library.
- LMFlow provides pipeline-style SFT and RLHF training, custom optimizers and tuners, evaluation, inference, and acceleration for text and multimodal models.It also develops lightweight fine-tuning techniques such as LISA.
- The frameworks’ auxiliary training capabilities are summarized in Table 1 for comparison.
3 IMPLEMENTATIONS AND FRAMEWORKS
SWIFT unifies lightweight training, multimodal model support, data and template processing, and downstream evaluation within one framework. Its architecture combines extensible tuners and training components with compatibility utilities and support for parallel, reinforced, and multimodal workflows.
- SWIFT organizes lightweight training around parameter reduction, quantization, gradient-memory reduction, freezing, and sharding or mixed precision.Examples include LISA, six quantization methods, GaLore, LoRA, AdaLoRA, DeepSpeed, FSDP, and mixed precision training.
- Its tuner layer extends PEFT with LoRA, AdaLoRA, IA3, BOFT, Vera, SCEdit, ResTuning, LLaMA-Pro, LongLoRA, and LISA, while allowing tuner combinations and customized tuners.PEFT and customized tuners are integrated through prepare_model and from_pretrained methods.
- SWIFT standardizes datasets from ModelScope, Hugging Face, or local CSV and JSONL files, then converts them into model-specific inputs through templates.Templates produce fields such as input_ids, attention_masks, pixel_values, and labels, and convert bounding-box coordinates for multimodal grounding.
- The framework includes SFT/PT and human-alignment trainers, supports Megatron checkpoint conversion and sequence parallelism, and accommodates reinforced fine-tuning workflows.Its human-alignment trainer supports DPO, ORPO, and KTO, while reinforced fine-tuning uses rollout generation and reward-based filtering.
- SWIFT supports multimodal tasks including Visual Question Answering, Optical Character Recognition, Grounded Captioning, and Referring Grounding.
- SWIFT integrates evaluation through EvalScope, combining OpenCompass and VLMEvalKit to support over 100 pure-text and multimodal evaluation sets.The framework also supports custom evaluation datasets and their evaluation processes.
4 EXPERIMENTS
SWIFT is used to validate lightweight tuning methods and agent-training techniques through controlled experiments on large language models. The experiments compare tuner profiles and assess agent performance, loss-scale ablations, and results for LLaMA3 and Qwen2.
- 4.1 Lightweight Tuning Benchmark: The benchmark uses qwen-7b-chat on a single A100-80G GPU to compare memory usage and loss across lightweight tuning algorithms.The experiment also compares other metrics, with hyperparameters reported separately.
- 4.1 Lightweight Tuning Benchmark: LISA achieves the lowest memory consumption and fastest speed, while LoRA+ records the lowest evaluation loss among additional-structure tuners.“Full” denotes the full-parameter training control group.
- 4.1 Lightweight Tuning Benchmark: Q-GaLore exhibits the lowest memory consumption among gradient-reduction methods, and these tuning methods are not included in PEFT.The comparison distinguishes gradient-reduction methods from additional-structure tuners.
- 4.2 Agent Training: Agent-training experiments mix ToolBench and AgentFlan data and compare trained versus untrained LLaMA3-8b-instruct and Qwen2-7b-instruct models.The experiments use the hyperparameter settings reported for the agent-training study.
- 4.2 Agent Training: Loss-scale significantly improves all evaluation metrics in the LLaMA3-8b-instruct ablation study.The ablation compares LoRA training with and without loss-scale for in-domain and out-of-domain settings.
- 4.2 Agent Training: LLaMA3’s average metrics improve by 17% with LoRA training, while Qwen2’s average metrics improve by 8.25% and hallucinations fall to single digits.The reported Qwen2 comparison is against the official Qwen2 model.
5 CONCLUSION
SWIFT is presented as a lightweight, one-stop framework intended to standardize large-model training across models, datasets, and techniques. The conclusion identifies planned improvements in parallel training, multimodal research, and RAG support.
- 5 CONCLUSION: SWIFT supports over 300 LLMs and 50 MLLMs while providing a standardized closed-loop solution for large-model development.The framework also includes a command-line-based WEB-UI intended to reduce usage costs.
- 5 CONCLUSION: Megatron support does not yet fully cover mainstream LLMs and MLLMs, motivating improved large-scale parallel-training support.The stated goal is greater pre-training convenience for foundational-model developers.
- 5 CONCLUSION: SWIFT supports training for most mainstream multimodal models but lacks deeper work on multimodal datasets, models, agents, chain-of-thought, and alignment training.The authors specifically mention high-quality datasets to help prevent knowledge forgetting and new multimodal models trained on ModelScope datasets.
- 5 CONCLUSION: Planned RAG support includes more robust training technology and enhancement training intended to improve recall rates and answer accuracy.The conclusion describes connecting SWIFT to various AI systems, including RAG systems.
A SUPPORTED MODELS AND DATASETS
This appendix provides tables listing portions of the models and datasets supported by SWIFT.
- A SUPPORTED MODELS AND DATASETS: Table 12 lists part of the models supported by SWIFT.The supplied passage provides the table title but no individual model entries.
- A SUPPORTED MODELS AND DATASETS: Table 13 lists part of the datasets supported by SWIFT.The supplied passage provides the table title but no individual dataset entries.
B LOSS-SCALE SETTINGS
This appendix presents a table describing content weights used in Agent training for loss-scale testing.
- B LOSS-SCALE SETTINGS: Table 14 reports content weights used in Agent training for loss-scale testing.The supplied passage provides the table title but no weight values or further experimental details.
C SWIFT COMMANDS
This appendix collects SWIFT implementation examples, including tuner code, prompt formats, and command-line workflows. The supplied passages identify the listings but provide only partial code content.
- C SWIFT COMMANDS: The appendix includes a code listing for training and inference with tuners.The listing references model preparation, checkpoint loading, and tuner configurations such as LoRA and LLaMA-Pro.
- C SWIFT COMMANDS: The appendix includes standard SWIFT prompts for question answering with history and tools.The example format contains query, response, system, history, and tool fields.