Source-linked AI summary
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, Alham Fikri Aji
TL;DR
Large language models are resource-intensive, motivating knowledge distillation into smaller models. LaMini-LM builds a large instruction dataset and diverse smaller models, achieving comparable performance to Alpaca while being significantly smaller.
Problem
Large language models and many distilled models remain computationally demanding, limiting deployment in resource-constrained settings.
Method
The paper creates a 2.58M-example instruction dataset, generates responses with gpt-3.5-turbo, and fine-tunes encoder-decoder and decoder-only models ranging from 61M to 7B parameters.
Results
LaMini-LM achieves comparable performance to Alpaca while being significantly smaller, with some models matching or surpassing LLaMA-7B and Alpaca-7B on downstream tasks.
Takeaways & Limitations
The results indicate that instruction-based knowledge distillation can produce smaller models with competitive performance across downstream and human evaluations.
Takeaways & Limitations
The study evaluates a limited selection of base models and does not optimize its models for multi-turn dialogues.
Abstract
from arXiv · showhide
Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.
1 Introduction
LaMini-LM addresses the resource demands and deployment barriers of instruction-tuned LLMs by distilling their knowledge into smaller, diverse models using a large instruction dataset. The work evaluates these models broadly, including downstream performance, human judgments, hallucination, and toxicity.
- Instruction-tuned LLMs offer strong capabilities but require substantial computational resources, creating energy, environmental, and access challenges.
- LaMini-LM targets deployment constraints by producing smaller models than most existing instruction-tuned and distilled models.Previous distilled models typically range from 7B to 13B parameters, limiting deployment in resource-constrained settings.
- The dataset combines diverse existing resources with newly generated instructions produced using gpt-3.5-turbo.
- LaMini-LM spans multiple model families and sizes, while evaluations cover downstream NLP tasks, human quality judgments, hallucination, and toxicity.The models range from 61M to 7B parameters and include encoder-decoder and decoder-only architectures.
- The LaMini instruction dataset contains over 2.58M examples and is presented as the largest instruction dataset of its kind.
2 Related Work
Prior work establishes instruction tuning and model-generated instructions as effective ways to improve language-model responsiveness, while knowledge distillation transfers capabilities from larger teachers to smaller students. Sequence-level distillation provides an efficient teacher-output training strategy for sequence-to-sequence models.
- Instruction tuning with natural-language instructions improves vanilla language models’ ability to follow general instructions.
- Model-generated instructions have been used for instruction tuning and can improve vanilla language models’ responsiveness.
- Knowledge distillation trains a smaller student model using knowledge from a larger teacher model.Common targets include teacher logits, output probabilities, or intermediate activations.
- Sequence-level distillation trains sequence-to-sequence students on synthetic outputs generated by the teacher and requires running the teacher only once.
3 Dataset Generation
The dataset combines existing prompts with example-guided and topic-guided generation, producing instruction-response pairs for sequence distillation. Analyses examine its statistics, semantic and lexical diversity, and human-rated quality, while identifying limitations in generated context and answers.
- Dataset construction: 2.58M instruction-response pairs are generated from self-instruct, Alpaca, P3, and FLAN resources using ChatGPT.The complete dataset is formed by uniting generated and collected instruction-response subsets.
- Instruction generation: Example-guided generation uses sampled seed tasks and produces batches of instructions under format constraints.The process samples three seed tasks, generating 20 instructions for self-instruct and 10 for P3 and FLAN.
- Instruction generation: Topic-guided generation filters 2.2M Wikipedia categories to 3.5K common topics to improve instruction diversity.The strategy addresses concerns that gpt-3.5-turbo may generate insufficiently diverse text without explicit topic guidance.
- Diversity analysis: Generated instructions bX SI are more semantically diverse than bX A, while bX P3 is slightly more diverse than X P3.The comparisons are based on t-SNE visualizations of sentence embeddings.
- Diversity analysis: Topic-guided subsets show higher lexical diversity than corresponding self-instruct subsets, while DALL has the highest lexical diversity overall.MATTR is used with a window size of 50, and model-generated instructions generally remain less lexically diverse than human-written instructions.
- Human evaluation: Human experts rate generated text highly overall, but topic-guided and P3/FLAN-based instructions can elicit erroneous or incorrect responses.The evaluation samples 400 examples from each DALL subset and uses eight external human experts.
4 Experiments
The experiments train and evaluate LaMini-LM models spanning architectures and sizes, using zero-shot downstream tasks and human assessment on user-oriented instructions. The evaluation framework also covers model-output quality beyond task accuracy.
- Training LaMini-LM: LaMini-LM includes encoder-decoder and decoder-only models ranging from 61M to 7B parameters.The collection is initialized from seven model sources, including T5, Flan-T5, Cerebras-GPT, GPT variants, and LLaMA.
- Automatic evaluation: The models are evaluated on 15 diverse NLP tasks using zero-shot evaluation with the language model evaluation harness.Tasks cover question answering, sentiment analysis, paraphrase identification, natural language inference, coreference, word sense disambiguation, and sentence completion.
- Human evaluation: Human evaluation measures response quality on 252 user-oriented instructions spanning 71 common application use-cases.Responses are rated from A, best, to D, worst, because many user-oriented questions admit multiple correct answers.
5 Results and Discussions
Across downstream NLP tasks and human evaluation, larger models generally perform better, but selected smaller LaMini models match or exceed strong 7B baselines. Results also show that architecture and instruction-subset choice materially affect performance across evaluation settings.
- Automatic Evaluation: LaMini models generally improve with size, while some match or surpass LLaMA-7B and Alpaca-7B on downstream NLP tasks.LaMini-LLaMA-7B significantly outperforms both baselines in average performance.
- Human Evaluation: Human evaluation follows the size trend, while small encoder-decoder T5 models show exceptional performance.The evaluation also identifies a substantial gap between LaMini models and gpt-3.5-turbo.
- Foundation Model Choice: Encoder-decoder LaMini models outperform decoder-only models below 500M parameters, and LaMini-Flan-T5-248M performs on par with LLaMA-7B.LaMini-GPT also outperforms similarly sized LaMini-Cerebras models in downstream and human evaluation.
- Foundation Model Choice: Further fine-tuning with suitable instructions can mitigate Flan-T5 models’ weakness on general user-oriented instructions.The finding underscores the role of instruction-dataset design in model behavior.
- Utility of Subsets: P3- and FLAN-related subsets improve downstream NLP performance more than self-instruct-related subsets, which have greater impact on human evaluation.This discrepancy indicates that subset utility depends on the evaluation type.
6 Hallucination and Toxicity
The paper evaluates hallucination as rejection of unanswerable or inappropriate questions and measures toxicity on non-toxic and toxic prompts. The results expose persistent hallucination challenges and architecture-dependent toxicity changes after instruction tuning.
- Hallucination: The LaMini-Hallucination test reframes hallucination as binary question rejection for unanswerable or inappropriate questions.The test contains 40 questions across four categories, with human evaluation of generated responses.
- Hallucination: Current LLMs still face significant challenges on the 40-question hallucination test.The conclusion reports this challenge after evaluating recommended models on the curated test.
- Toxicity: Toxicity is evaluated using 1K non-toxic and 1K toxic prompts from RealToxicityPrompts, with lower toxic-output counts preferred.Outputs are generated with the prefix “Complete the sentence:”.
- Toxicity: Before instruction tuning, encoder-decoder LaMini-Flan-T5 models generally produce less toxic text than decoder-only LaMini-GPT and LaMini-LLaMA-7B models.Instruction tuning increases toxicity tendency for encoder-decoder models but decreases it for decoder-only models.
7 Conclusion
The paper presents a 2.58M-example instruction dataset and a family of smaller LaMini-LM models distilled from LLM outputs. Comprehensive evaluations show comparable performance to Alpaca while using substantially smaller models, alongside persistent hallucination challenges.
- Conclusion: The LaMini instruction dataset contains over 2.58M examples derived from gpt-3.5-turbo.The paper identifies it as the largest dataset of its kind at the time of the study.
- Conclusion: LaMini-LM comprises 6 encoder-decoder and 11 decoder-only models ranging from 61M to 7B parameters.The family is designed to distill LLM knowledge into smaller, more efficient architectures.
- Conclusion: Comprehensive downstream and human evaluations show comparable performance to Alpaca while the LaMini models are significantly smaller.The evaluation covers general usage, hallucination, and toxicity.
- Conclusion: The curated hallucination evaluation finds that current LLMs still face significant challenges in this area.The study uses 40 questions to examine hallucination behavior.
8 Limitations
The paper identifies limitations in model coverage and dialogue scope, as well as risks inherited from the gpt-3.5-turbo-derived training data. These boundaries constrain conclusions about broader model families, multi-turn interaction, and safety behavior.
- Model Variations: The model selection is limited to T5, GPT-2, Cerebras-GPT, GPT-Neo, and LLaMA base models.The authors recommend expanding the model set to support broader performance-trend analysis and comparisons with prior research.
- Single Turn Dialog: LaMini-LM is not currently optimized for multi-turn dialogues.Training data and user-oriented evaluation primarily focus on dialog-like single-turn instructions.
- Error Propagation: Because training uses condensed knowledge from gpt-3.5-turbo, LaMini-LM inherits potential hallucination and toxicity risks.The authors also report unsatisfactory human-feedback performance in coding and mathematical tasks.
9 Ethical Consideration
LaMini-LM improves small-model performance but remains imperfectly aligned with human objectives, with risks including discrimination, bias, and false information. The authors advocate responsible use and future risk reduction.
- Instruction-tuned LaMini-LM models may generate discriminatory responses and propagate biases from their teacher model.
- The models often generate false information, which may have unintended consequences.
- The authors intend to minimize risks associated with using these models in future research.
- The authors advocate responsible use of LaMini-LM to prevent harm.
C Human Evaluation Protocol
The paper describes its human-evaluation protocol and illustrates instruction-generation and response-generation procedures through example figures and implementation details.
- The human-evaluation protocol includes examples for each rating level.
- External evaluators with at least a master’s degree from an English-speaking country conducted the human evaluations.
- Example-Guided Instruction Generation uses three random examples from self-instruct.
- Topic-Guided Instruction Generation combines three random examples with three random topics from a common topic list.
- The response-generation code uses the OpenAI API and instructs gpt-3.5-turbo to produce concise responses.
D Training Hyperparameters
The paper reports training, evaluation, and qualitative-analysis procedures for LaMini models across 15 NLP tasks, human assessments, and hallucination probes. Results are organized by model family and include comparisons with baselines.
- Training Hyperparameters: Most models were trained for 5 epochs with batch size 1024, while LaMini-GPT-J-6B and LaMini-LLaMA-7B used 6K steps, equivalent to 2.5 epochs.
- Automatic Evaluation: The study evaluates LaMini models on 15 downstream NLP tasks using task-specific metrics and reports results by model family.
- Human Evaluation: Human evaluation compares LaMini-LM responses with Alpaca-7B responses using examples that mark high-quality contents and errors.
- Qualitative Analysis: LaMini-LM responses are shorter than Alpaca-7B responses because response generation was constrained to prioritize concision.
- Hallucination Evaluation: The study probes hallucination with 40 hallucination-inducing questions.
- Automatic Evaluation: The automatic-evaluation tables report micro-average scores across the individual task results.